"No space left on device" from hell

"No space left on device" from hell

[Arkadiusz Miśkiewicz]

Hello.

3.10.40 kernel and creating files causes "No space left on device". After 
deleting some files I can create new files again until the problem happens 
again.

Any ideas what is going on and, what is more important, how to fix the 
problem?

# df -h
/dev/mapper/vgsys-home  600G  368G  232G  62% /home

# df -i
/dev/mapper/vgsys-home 629145600 7982895 621162705    2% /home

# xfs_info /home
meta-data=/dev/mapper/vgsys-home isize=256    agcount=8, agsize=19660800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=157286400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=38400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=

imaxpct=25% but current inodes take only about 2GB, much less that 25% of 
600G, so that shouldn't be problem

/dev/mapper/vgsys-home /home xfs 
rw,nosuid,nodev,relatime,attr2,inode64,usrquota,prjquota 0 0

so inode64 is not a problem

# xfs_quota -c  "quota -p 0" /home
Disk quotas for Project #0 (0)
Filesystem              Blocks      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                      27848804          0          0   00 [--------] /home
# xfs_quota -c  "quota -u 0" /home
Disk quotas for User root (0)
Filesystem              Blocks      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                        103484          0          0   00 [--------] /home
# xfs_quota -c  "quota -i 0" /home
Disk quotas for User root (0)
Filesystem               Files      Quota      Limit  Warn/Time      Mounted 
on
/dev/mapper/vgsys-home
                          8336          0          0   00 [--------] /home

so quota is not a problem, too.

# xfs_db -r "-c freesp -s"  /dev/mapper/vgsys-home
   from      to extents  blocks    pct
      1       1    3426    3426   0.01
      2       3    3319    8026   0.01
      4       7    3694   19027   0.03
      8      15   23225  330378   0.54
     16      31     396    8180   0.01
     32      63      75    3256   0.01
     64     127      44    3443   0.01
    128     255      26    4892   0.01
    256     511      32   10737   0.02
    512    1023       5    3867   0.01
   1024    2047       8   13576   0.02
   2048    4095       5   13893   0.02
   4096    8191       3   16896   0.03
   8192   16383       1   15532   0.03
  16384   32767       1   17731   0.03
1048576 2097151       1 1414952   2.33
16777216 19660800       3 58913965  96.90
total free extents 34264
total free blocks 60801777
average free extent size 1774.51

unfortunately no idea how to interpret this. Man page isn't helpful.


http://sprunge.us/IVjE
(trace-cmd of touch /home/x ... but looks to be partial only)

-- 
Arkadiusz Miśkiewicz, arekm / maven.pl

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: "No space left on device" from hell

[Fran Tsao Santín]

[-- Attachment #1.1: Type: text/plain, Size: 378 bytes --]

2014-09-24 9:03 GMT+02:00 Arkadiusz Miśkiewicz <arekm@maven.pl>:

>
> Hello.
>
> 3.10.40 kernel and creating files causes "No space left on device". After
> deleting some files I can create new files again until the problem happens
> again.


Are you sure the problem is the filesystem device? Maybe you are running a
process that fills up another device i.e. shm.

[-- Attachment #1.2: Type: text/html, Size: 720 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: "No space left on device" from hell

[Arkadiusz Miśkiewicz]

On Wednesday 24 of September 2014, Fran Tsao Santín wrote:
> 2014-09-24 9:03 GMT+02:00 Arkadiusz Miśkiewicz <arekm@maven.pl>:
> > Hello.
> > 
> > 3.10.40 kernel and creating files causes "No space left on device". After
> > deleting some files I can create new files again until the problem
> > happens again.
> 
> Are you sure the problem is the filesystem device? Maybe you are running a
> process that fills up another device i.e. shm.

Dave Chinner figured out the problem already (on irc).

Basically this fs was grown from 300GB to 600GB but due to bug (fixed by 
commit 9de67c3 ("xfs: allow inode allocations in post-growfs disk space")) 
that new space was never used for new inodes.

So ended up with full AGs 0-3 (xfs_db -r -c "freesp -s -a XX"  /dev/xyz shows 
that for each AG XX).

Fs was never unmounted after grows but fortunately mount -o remount,inode32 
... followed by mount -o remount,inode64 cured it without a need to reboot, so 
now new inodes are distributed to all AGs.

I still need to free some space in AGs 0-3 though (by moving data to new dirs 
allocated in new AGs; xfs_db -r -c "convert inode DIR_INODE_NR agno" /dev/xyz 
shows in which AG new dir got allocated).

There are more bugs hiding (like if ENOSPC on some AG happens it should try 
next AG instead of failing). Dave knows all that magic.

Big thanks to Dave!

-- 
Arkadiusz Miśkiewicz, arekm / maven.pl

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs