partition 100% full No space left on device. looks like xfs is corrupted or a bug

Lista Unx
2016-07-29T09:01:49Z

[-- Attachment #1.1: Type: text/plain, Size: 7643 bytes --]

Hello xfs experts,

I am crawling in the dark from few days and I have no idea how to fix the following problem. On a centos 7 system:

# uname -a
Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

df is reporting 100% full of / and du is reporting only 1.7G usage from 50GB available (less than 4%). I want to mention that / is xfs. See below:

# df -a|grep ^/
/dev/mapper/centos-root 52403200 52400396 2804 100% /
^^^^^^^^^^ ^^^^^^^^^^
/dev/sda1 503040 131876 371164 27% /boot
/dev/mapper/centos-home 210529792 35204 210494588 1% /home

du is estimating just 1.7G usage of /
# du -sch /* --exclude=home --exclude=boot
0 /bin
0 /dev
25M /etc
0 /lib
0 /lib64
744K /luarocks-2.3.0
0 /media
0 /mnt
125M /openresty-1.9.7.4
0 /opt
420K /root
49M /run
0 /sbin
0 /srv
0 /sys
0 /tmp
1.3G /usr
227M /var
1.7G total
[root@localhost ~]#

df is also reporting 80% of inode usage:

# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 78160 66218 11942 85% /
^^^^^^^^
devtmpfs 8218272 519 8217753 1% /dev
tmpfs 8221010 1 8221009 1% /dev/shm
tmpfs 8221010 648 8220362 1% /run
tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
/dev/sda1 509952 330 509622 1% /boot
/dev/mapper/centos-home 210632704 99 210632605 1% /home
tmpfs 8221010 1 8221009 1% /run/user/0
#

/ partition is created on top of a LVM having also 50GB size.

# lvdisplay /dev/centos/root
--- Logical volume ---
LV Path /dev/centos/root
LV Name root
VG Name centos

LV Status available
# open 1
LV Size 50.00 GiB
Current LE 12800
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

I've already checked against rootkit without finding anything wrong!

I have another system, identical with this one which is healthy. The only difference I found between those systems is regarding max number of inodes available on / (which has the same size, 50GB on booth servers). On the second one (healthy), max number of inodes are ~52 milions and not only just ~85.000 as are reported on "seek" server.

# df -i|grep ^/
/dev/mapper/centos-root 52424704 66137 52358567 1% /
^^^^^^^^^^^^^
/dev/sda1 509952 330 509622 1% /boot
/dev/mapper/centos-home 210632704 26 210632678 1% /home
[root@localhost ~]#

Suspected also large number of files on /. Counted total number of files and or booth servers are the same: ~180K. So no difference here.

Look to find also files larger than 100M and on booth servers and found just 1 (104M size):

find / -type f -size +100000k -exec ls -lh {} \;
#
/usr/lib/locale/locale-archive
#

Looking to find files larger than 10M, I found just ~20 on booth servers.

# find / -type f -size +10000k -exec ls -lh {} \; |wc -l
16
#

So for sure, there are NO files exhausting free space.

On booth servers, number of used inodes are identical: ~66K. Also xfs_info report is identical for booth. What is different is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy node)!!! How is possible that!!! Booth servers has the same size (50GB) for /!

#lsof -nP |grep -i delete|wc -l
0
#find /proc/*/fd -ls | grep -i dele|wc -l
0

so lsof and find does not report anything wrong (any file deleted and still open)!

reboot does not fix the problem, / remain 100% full

After reboot, on 25th July:

# df -ah|grep centos-root
/dev/mapper/centos-root 50G 50G 4.0M 100% /
#

Also max number of inodes = 67k:
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 66960 66165 795 99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs 8218272 519 8217753 1% /dev
tmpfs 8221010 1 8221009 1% /dev/shm
tmpfs 8221010 630 8220380 1% /run
tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
/dev/sda1 509952 330 509622 1% /boot
/dev/mapper/centos-home 210632704 28 210632676 1% /home
tmpfs 8221010 1 8221009 1% /run/user/0
#

Lets try to run intentionally xfs_grow (which normally should not produce any change)

# xfs_growfs /dev/mapper/centos-root
meta-data=/dev/mapper/centos-root isize=256 agcount=16, agsize=819136 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=13106176, imaxpct=25
= sunit=64 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=6400, version=2
= sectsz=512 sunit=64 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 13106176 to 13107200
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#

Partition remain the same, 50GB size:
[root@nl-hvs-ov001a ~]# df -ah|grep centos-root
/dev/mapper/centos-root 50G 50G 4.0M 100% /

But number of inodes INCREASED with more tha 20%!!!
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 83200 66165 17035 80% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs 8218272 519 8217753 1% /dev
tmpfs 8221010 1 8221009 1% /dev/shm
tmpfs 8221010 630 8220380 1% /run
tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
/dev/sda1 509952 330 509622 1% /boot
/dev/mapper/centos-home 210632704 28 210632676 1% /home
tmpfs 8221010 1 8221009 1% /run/user/0
#

On 27July without changing anything there, max number inodes available for / decreased to ~67k (the same size like 2 days ago, before xfs_grow)!

# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 67024 66225 799 99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs 8218272 519 8217753 1% /dev
tmpfs 8221010 1 8221009 1% /dev/shm
tmpfs 8221010 632 8220378 1% /run
tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
/dev/mapper/centos-home 210632704 99 210632605 1% /home
/dev/sda1 509952 330 509622 1% /boot
tmpfs 8221010 1 8221009 1% /run/user/0
#

Please note that all that time, number of files remain unchanged ~180K, the same for inodes used, the number remain constant ~66K. Just max number of inodes available decreased which is an abnormal behavior.

How can be fixed? Looks like xfs is crrupted or like a bug.

Thanks in advance for help.
Alex

[-- Attachment #1.2: Type: text/html, Size: 18138 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]