* xfs wrongly report ENOSPACE
@ 2017-02-06 12:37 peng.hse
2017-02-06 13:14 ` Brian Foster
0 siblings, 1 reply; 4+ messages in thread
From: peng.hse @ 2017-02-06 12:37 UTC (permalink / raw)
To: djwong, linux-fsdevel, darrick.wong
Hi DJ and xfs developers,
i am ceph developer, build the OSD on top of the xfs mount point. the
kernel xfs module i used was build from the 4.9.8 kernel release from
kernel.org, that includes your recent changes in Dec, 2016.
recently, i found our cluster will get faulted due to the xfs mount
wrongly report there is no space to create the new file as
" No space left on device ", but apparently the xfs mount point only
be around 50% full.
i used the systemtap to narrow it down the the function
xfs_ialloc_ag_select which reports no free ag selected, the problem seems
still exist, not sure your recent patches will fix it or not.
i am very glad to provide more info if your like to assist you to
identify the problem.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: xfs wrongly report ENOSPACE
2017-02-06 12:37 xfs wrongly report ENOSPACE peng.hse
@ 2017-02-06 13:14 ` Brian Foster
2017-02-07 12:39 ` peng.hse
0 siblings, 1 reply; 4+ messages in thread
From: Brian Foster @ 2017-02-06 13:14 UTC (permalink / raw)
To: peng.hse; +Cc: djwong, linux-xfs, darrick.wong
Please use the XFS list, updated cc.
On Mon, Feb 06, 2017 at 08:37:26PM +0800, peng.hse wrote:
> Hi DJ and xfs developers,
>
> i am ceph developer, build the OSD on top of the xfs mount point. the kernel
> xfs module i used was build from the 4.9.8 kernel release from
> kernel.org, that includes your recent changes in Dec, 2016.
>
> recently, i found our cluster will get faulted due to the xfs mount wrongly
> report there is no space to create the new file as
> " No space left on device ", but apparently the xfs mount point only be
> around 50% full.
>
> i used the systemtap to narrow it down the the function xfs_ialloc_ag_select
> which reports no free ag selected, the problem seems
> still exist, not sure your recent patches will fix it or not.
>
> i am very glad to provide more info if your like to assist you to identify
> the problem.
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
In this case, please at least provide the output from 'df -i <mnt>,'
'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
Chances are that this is caused by free space fragmentation. This means
that while there might be plenty of free space, there isn't a contiguous
free extent large enough to allocate an inode chunk. We've seen this
kind of thing before with Ceph and it can be exacerbated by the use of
large inodes. Potential workarounds may be to use the default inode size
and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
Another possible cause is that you've simply hit the maximum inode
allocation limit (see the xfs_growfs manpage and the '-m' option)...
Brian
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: xfs wrongly report ENOSPACE
2017-02-06 13:14 ` Brian Foster
@ 2017-02-07 12:39 ` peng.hse
2017-02-07 13:01 ` Carlos Maiolino
0 siblings, 1 reply; 4+ messages in thread
From: peng.hse @ 2017-02-07 12:39 UTC (permalink / raw)
To: Brian Foster; +Cc: linux-xfs
On 2017年02月06日 21:14, Brian Foster wrote:
> Please use the XFS list, updated cc.
>
> On Mon, Feb 06, 2017 at 08:37:26PM +0800, peng.hse wrote:
>> Hi DJ and xfs developers,
>>
>> i am ceph developer, build the OSD on top of the xfs mount point. the kernel
>> xfs module i used was build from the 4.9.8 kernel release from
>> kernel.org, that includes your recent changes in Dec, 2016.
>>
>> recently, i found our cluster will get faulted due to the xfs mount wrongly
>> report there is no space to create the new file as
>> " No space left on device ", but apparently the xfs mount point only be
>> around 50% full.
>>
>> i used the systemtap to narrow it down the the function xfs_ialloc_ag_select
>> which reports no free ag selected, the problem seems
>> still exist, not sure your recent patches will fix it or not.
>>
>> i am very glad to provide more info if your like to assist you to identify
>> the problem.
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
>
> In this case, please at least provide the output from 'df -i <mnt>,'
> 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
>
> Chances are that this is caused by free space fragmentation. This means
> that while there might be plenty of free space, there isn't a contiguous
> free extent large enough to allocate an inode chunk. We've seen this
> kind of thing before with Ceph and it can be exacerbated by the use of
> large inodes. Potential workarounds may be to use the default inode size
> and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
>
> Another possible cause is that you've simply hit the maximum inode
> allocation limit (see the xfs_growfs manpage and the '-m' option)...
>
> Brian
>
the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
[root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No
space left on device
[root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdc2 982848 47296 935552 5%
/mnt/819f7954-bf88-430d-a637-a08bad56cf6e
[root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
meta-data=/dev/sdc2 isize=2048 agcount=4, agsize=491455 blks
= sectsz=512 attr=2,
projid32bit=1
= crc=0 finobt=0
data = bsize=4096 blocks=1965819,
imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=2560,
version=2
= sectsz=512 sunit=0
blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
from to extents blocks pct
1 1 80 80 0.01
2 3 61 159 0.02
4 7 88 483 0.06
8 15 146 1680 0.21
16 31 34312 788594 99.70
total free extents 34687
total free blocks 790996
average free extent size 22.8038
the total number of files underneath the mnt is
[root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l
47294
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: xfs wrongly report ENOSPACE
2017-02-07 12:39 ` peng.hse
@ 2017-02-07 13:01 ` Carlos Maiolino
0 siblings, 0 replies; 4+ messages in thread
From: Carlos Maiolino @ 2017-02-07 13:01 UTC (permalink / raw)
To: peng.hse; +Cc: Brian Foster, linux-xfs
> > In this case, please at least provide the output from 'df -i <mnt>,'
> > 'xfs_info <mnt>' and 'xfs_db -c "freesp -s" <dev>.'
> >
> > Chances are that this is caused by free space fragmentation. This means
> > that while there might be plenty of free space, there isn't a contiguous
> > free extent large enough to allocate an inode chunk. We've seen this
> > kind of thing before with Ceph and it can be exacerbated by the use of
> > large inodes. Potential workarounds may be to use the default inode size
> > and/or enable the use of sparse inodes ('mkfs.xfs -i sparse <dev>').
> >
> > Another possible cause is that you've simply hit the maximum inode
> > allocation limit (see the xfs_growfs manpage and the '-m' option)...
> >
> > Brian
> >
>
> the problematic mnt point is /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
>
> [root@xt1 ~]# touch /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx
> touch: cannot touch ‘/mnt/819f7954-bf88-430d-a637-a08bad56cf6e/xx’: No space
> left on device
>
> [root@xt1 ~]# df -i /mnt/819f7954-bf88-430d-a637-a08bad56cf6e/
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sdc2 982848 47296 935552 5%
> /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
>
>
>
> [root@xt1 ~]# xfs_info /mnt/819f7954-bf88-430d-a637-a08bad56cf6e
> meta-data=/dev/sdc2 isize=2048 agcount=4, agsize=491455 blks
> = sectsz=512 attr=2,
2048 bytes inode, I'd bet your filesystem is fragmented enough to not have any
more room to create inodes into it.
> projid32bit=1
> = crc=0 finobt=0
> data = bsize=4096 blocks=1965819,
> imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=0
> log =internal bsize=4096 blocks=2560,
> version=2
> = sectsz=512 sunit=0 blks,
> lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
>
>
> [root@xt1 ~]# xfs_db -r '-c freesp -s' /dev/sdc2
> from to extents blocks pct
> 1 1 80 80 0.01
> 2 3 61 159 0.02
> 4 7 88 483 0.06
> 8 15 146 1680 0.21
> 16 31 34312 788594 99.70
for creating new inodes you will need a contiguous space of 131072 bytes, which
you don't have anymore, so you won't be able to create any more files into this
filesystem, unless you free up some space to allow room for the creation of new
inodes.
XFS create inodes in chunks of 64 inodes, if you don't have enough space for
that, then you get a -ENOSPC as you are getting.
Cheers.
> total free extents 34687
> total free blocks 790996
> average free extent size 22.8038
>
> the total number of files underneath the mnt is
> [root@xt1 819f7954-bf88-430d-a637-a08bad56cf6e]# find . |wc -l
> 47294
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Carlos
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-02-07 13:01 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-02-06 12:37 xfs wrongly report ENOSPACE peng.hse
2017-02-06 13:14 ` Brian Foster
2017-02-07 12:39 ` peng.hse
2017-02-07 13:01 ` Carlos Maiolino
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.