Re: xfs wrongly report ENOSPACE

Re: xfs wrongly report ENOSPACE

[Brian Foster]

Please use the XFS list, updated cc.

On Mon, Feb 06, 2017 at 08:37:26PM +0800, peng.hse wrote:
> Hi DJ and xfs developers,
> 
> i am ceph developer, build the OSD on top of the xfs mount point. the kernel
> xfs module i used was build from the 4.9.8 kernel release from
> kernel.org, that includes your recent changes in Dec, 2016.
> 
> recently, i found our cluster will get faulted due to the xfs mount wrongly
> report there is no space to create the new file as
>  " No space left on device ", but apparently the xfs mount point only be
> around 50% full.
> 
> i used the systemtap to narrow it down the the function xfs_ialloc_ag_select
> which reports no free ag selected, the problem seems
> still exist, not sure your recent patches will fix it or not.
> 
> i am very glad to provide more info if your like to assist you to identify
> the problem.

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

In this case, please at least provide the output from 'df -i <mnt>,'
'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'

Chances are that this is caused by free space fragmentation. This means
that while there might be plenty of free space, there isn't a contiguous
free extent large enough to allocate an inode chunk. We've seen this
kind of thing before with Ceph and it can be exacerbated by the use of
large inodes. Potential workarounds may be to use the default inode size
and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').

Another possible cause is that you've simply hit the maximum inode
allocation limit (see the xfs_growfs manpage and the '-m' option)...

Brian

Re: xfs wrongly report ENOSPACE

[peng.hse]

On 2017年02月06日 21:14, Brian Foster wrote:
> Please use the XFS list, updated cc.
>
> On Mon, Feb 06, 2017 at 08:37:26PM +0800, peng.hse wrote:
>> Hi DJ and xfs developers,
>>
>> i am ceph developer, build the OSD on top of the xfs mount point. the kernel
>> xfs module i used was build from the 4.9.8 kernel release from
>> kernel.org, that includes your recent changes in Dec, 2016.
>>
>> recently, i found our cluster will get faulted due to the xfs mount wrongly
>> report there is no space to create the new file as
>>   " No space left on device ", but apparently the xfs mount point only be
>> around 50% full.
>>
>> i used the systemtap to narrow it down the the function xfs_ialloc_ag_select
>> which reports no free ag selected, the problem seems
>> still exist, not sure your recent patches will fix it or not.
>>
>> i am very glad to provide more info if your like to assist you to identify
>> the problem.
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>
> In this case, please at least provide the output from 'df -i <mnt>,'
> 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
>
> Chances are that this is caused by free space fragmentation. This means
> that while there might be plenty of free space, there isn't a contiguous
> free extent large enough to allocate an inode chunk. We've seen this
> kind of thing before with Ceph and it can be exacerbated by the use of
> large inodes. Potential workarounds may be to use the default inode size
> and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
>
> Another possible cause is that you've simply hit the maximum inode
> allocation limit (see the xfs_growfs manpage and the '-m' option)...
>
> Brian
>

the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/

[root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No 
space left on device

[root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
Filesystem     Inodes    IUsed   IFree       IUse%  Mounted on
/dev/sdc2      982848  47296  935552    5% 
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e



[root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
meta-data=/dev/sdc2              isize=2048    agcount=4, agsize=491455 blks
                  =                             sectsz=512   attr=2, 
projid32bit=1
                  =                             crc=0 finobt=0
data          =                             bsize=4096 blocks=1965819, 
imaxpct=25
                  =                             sunit=0 swidth=0 blks
naming     =version 2               bsize=4096   ascii-ci=0 ftype=0
log            =internal                  bsize=4096   blocks=2560, 
version=2
                  =                              sectsz=512   sunit=0 
blks, lazy-count=1
realtime    =none                      extsz=4096   blocks=0, rtextents=0



[root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
    from      to extents  blocks    pct
       1       1      80      80   0.01
       2       3      61     159   0.02
       4       7      88     483   0.06
       8      15     146    1680   0.21
      16      31   34312  788594  99.70
total free extents 34687
total free blocks 790996
average free extent size 22.8038

the total number of files underneath the mnt is
[root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l
47294

Re: xfs wrongly report ENOSPACE

[Carlos Maiolino]

> > In this case, please at least provide the output from 'df -i <mnt>,'
> > 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
> > 
> > Chances are that this is caused by free space fragmentation. This means
> > that while there might be plenty of free space, there isn't a contiguous
> > free extent large enough to allocate an inode chunk. We've seen this
> > kind of thing before with Ceph and it can be exacerbated by the use of
> > large inodes. Potential workarounds may be to use the default inode size
> > and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
> > 
> > Another possible cause is that you've simply hit the maximum inode
> > allocation limit (see the xfs_growfs manpage and the '-m' option)...
> > 
> > Brian
> > 
> 
> the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
> 
> [root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
> touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No space
> left on device
> 
> [root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
> Filesystem     Inodes    IUsed   IFree       IUse%  Mounted on
> /dev/sdc2      982848  47296  935552    5%
> /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
> 
> 
> 
> [root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
> meta-data=/dev/sdc2              isize=2048    agcount=4, agsize=491455 blks
>                  =                             sectsz=512   attr=2,

2048 bytes inode, I'd bet your filesystem is fragmented enough to not have any
more room to create inodes into it.

> projid32bit=1
>                  =                             crc=0 finobt=0
> data          =                             bsize=4096 blocks=1965819,
> imaxpct=25
>                  =                             sunit=0 swidth=0 blks
> naming     =version 2               bsize=4096   ascii-ci=0 ftype=0
> log            =internal                  bsize=4096   blocks=2560,
> version=2
>                  =                              sectsz=512   sunit=0 blks,
> lazy-count=1
> realtime    =none                      extsz=4096   blocks=0, rtextents=0
> 
> 
> 
> [root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
>    from      to extents  blocks    pct
>       1       1      80      80   0.01
>       2       3      61     159   0.02
>       4       7      88     483   0.06
>       8      15     146    1680   0.21
>      16      31   34312  788594  99.70

for creating new inodes you will need a contiguous space of 131072 bytes, which
you don't have anymore, so you won't be able to create any more files into this
filesystem, unless you free up some space to allow room for the creation of new
inodes.

XFS create inodes in chunks of 64 inodes, if you don't have enough space for
that, then you get a -ENOSPC as you are getting.


Cheers.


> total free extents 34687
> total free blocks 790996
> average free extent size 22.8038
> 
> the total number of files underneath the mnt is
> [root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l
> 47294
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Carlos