Re: xfs wrongly report ENOSPACE

["peng.hse" <peng.hse@xtaotech.com>]

On 2017年02月06日 21:14, Brian Foster wrote:
> Please use the XFS list, updated cc.
>
> On Mon, Feb 06, 2017 at 08:37:26PM +0800, peng.hse wrote:
>> Hi DJ and xfs developers,
>>
>> i am ceph developer, build the OSD on top of the xfs mount point. the kernel
>> xfs module i used was build from the 4.9.8 kernel release from
>> kernel.org, that includes your recent changes in Dec, 2016.
>>
>> recently, i found our cluster will get faulted due to the xfs mount wrongly
>> report there is no space to create the new file as
>>   " No space left on device ", but apparently the xfs mount point only be
>> around 50% full.
>>
>> i used the systemtap to narrow it down the the function xfs_ialloc_ag_select
>> which reports no free ag selected, the problem seems
>> still exist, not sure your recent patches will fix it or not.
>>
>> i am very glad to provide more info if your like to assist you to identify
>> the problem.
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>
> In this case, please at least provide the output from 'df -i <mnt>,'
> 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
>
> Chances are that this is caused by free space fragmentation. This means
> that while there might be plenty of free space, there isn't a contiguous
> free extent large enough to allocate an inode chunk. We've seen this
> kind of thing before with Ceph and it can be exacerbated by the use of
> large inodes. Potential workarounds may be to use the default inode size
> and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
>
> Another possible cause is that you've simply hit the maximum inode
> allocation limit (see the xfs_growfs manpage and the '-m' option)...
>
> Brian
>

the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/

[root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No 
space left on device

[root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
Filesystem     Inodes    IUsed   IFree       IUse%  Mounted on
/dev/sdc2      982848  47296  935552    5% 
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e



[root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
meta-data=/dev/sdc2              isize=2048    agcount=4, agsize=491455 blks
                  =                             sectsz=512   attr=2, 
projid32bit=1
                  =                             crc=0 finobt=0
data          =                             bsize=4096 blocks=1965819, 
imaxpct=25
                  =                             sunit=0 swidth=0 blks
naming     =version 2               bsize=4096   ascii-ci=0 ftype=0
log            =internal                  bsize=4096   blocks=2560, 
version=2
                  =                              sectsz=512   sunit=0 
blks, lazy-count=1
realtime    =none                      extsz=4096   blocks=0, rtextents=0



[root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
    from      to extents  blocks    pct
       1       1      80      80   0.01
       2       3      61     159   0.02
       4       7      88     483   0.06
       8      15     146    1680   0.21
      16      31   34312  788594  99.70
total free extents 34687
total free blocks 790996
average free extent size 22.8038

the total number of files underneath the mnt is
[root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l
47294