Re: Slightly Urgent: XFS No Space Left On Device

Dave Chinner
2015-04-02T00:12:43Z

On Wed, Apr 01, 2015 at 03:53:28PM -0400, Dave Hall wrote:
> Please pardon the 'top-post', but here is the additional information
> requested:
>
> This is a Dell R720xd dual 8-core Xeon system with 128GB RAM. The
> RAID controller is Dell PERC H710 Mini with 12 2TB disks in RAID6.
>
> The OS is Debian 6 with kernel 3.2.0-0.bpo.4-amd64 #1 SMP Debian
> 3.2.65-1+deb7u2~bpo60+1 x86_64.

So defaults to inode32 allocation....

> From /proc/mounts:
>
> /dev/sdb1 /data xfs
> rw,noexec,noatime,attr2,delaylog,allocsize=64k,logbsize=64k,sunit=128,swidth=1280,usrquota,prjquota
> 0 0

... and inode64 is not in the mount options.....

> The output from xfs_info was previously included, but is repeated here:
>
> # xfs_info /data
> meta-data=/dev/sdb1 isize=256 agcount=19,agsize=268435440 blks

Inode allocation requires contiguous free space of 16k aligned to 8k
boundaries to allocate new inode chunks. Also, 1TB AGs, so with
inode32, inodes can only be allocated in AG 0.

> Here are the more extensive freesp outputs for each of the 19 AGs:
>
> # xfs_db -r /dev/sdb1 -c 'freesp -s -a0'
> from to extents blocks pct
> 1 1 747 747 19.68
> 2 3 1045 2496 65.77
> 4 7 138 552 14.55
> total free extents 1930
> total free blocks 3795
> average free extent size 1.96632

And that says you have no correctly aligned free 16k extents that
can be allocated in AG 0. i.e. no more inodes can be allocated, and
that's where the ENOSPC is coming from.

Unmount, add the inode64 mount option, and you'll be able to
allocate inodes again as they will be allowed to be allocated in
any AG, not just AG 0.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com