* Question about continous blocks for inode due to 'no space left on device' problem
@ 2016-02-05 13:13 Michel Verbraak
2016-02-05 22:30 ` Dave Chinner
0 siblings, 1 reply; 5+ messages in thread
From: Michel Verbraak @ 2016-02-05 13:13 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 3680 bytes --]
All,
Recently we ran into a problem where our filesystem (300GB in size)
reported 'no space left on device' (ENOSPC) but when we looked at disk
space usage and inode usage it was around 52% for disk space and 11% for
inode. (sorry do not have a save of the output of df command).
Second we looked at the xfs_info and xfs_db -c freesp for each AG. Sadly we
do not have a copy anymore of the xfs_info from that time but I do have
that of the freesp (at end of this mail).
As we did not fully understand the output of the 'xfs_db -c freesp' we
simply added another 100GB to the filesystem and our problem was solved for
now. The AG count went from 4 to 6.
For now we are looking into how can we detect (monitor) this filesystem and
see the same problem creeping up to us before it happens.
After searching online, reading a lot of mail threads and info pages it
looks like we hit the problem of not having any continuous blocks of at
least 16k to hold a new inode (
http://oss.sgi.com/pipermail/xfs/2014-September/038301.html).
Currently the xfs_info (after adding 100GB) looks like this:
meta-data=/dev/sdb isize=256 agcount=6, agsize=19660800
blks
= sectsz=512 attr=2
data = bsize=4096 blocks=104857600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=38400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
The output of 'xfs_db -c freesp' shows a historgram. We first thought the
"from" column shows the minimal number of continous blocks. And as each
block is 4K (4096) the row with 'from 4 to 7' contains at least 4
continuous blocks with a minimal size of 4x4k=16k. So our filesystem has
enough blocks and extents which are 16k for new inodes, atleast what we
think.
- So why did we still get the ENOSPC error?
- Where can I find a better explanation of the 'xfs_db -c freesp' output?
- What is the best way to monitor/detect this problem before it happens.
Some extra info about the system and files on the system:
- Ubuntu 12.04.5 LTS
- Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17
18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- 32 million files of which about 75% is smaller than 1k. Files are
separated over different folders to keep the number of files per folder low.
- mount options: defaults,noatime,inode64,nobarrier
Regards,
Michel Verbraak.
Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
from to extents blocks pct
1 1 3282633 3282633 9.03
2 3 3416223 8372325 23.03
4 7 6175009 24700036 67.94
Store 03 (root@server):~# xfs_db -c "freesp -a 0" /dev/sdb
from to extents blocks pct
1 1 813491 813491 8.95
2 3 806828 1953270 21.50
4 7 1579585 6318340 69.55
Store 03 (root@server):~# xfs_db -c "freesp -a 1" /dev/sdb
from to extents blocks pct
1 1 835174 835174 9.20
2 3 953626 2381033 26.22
4 7 1465787 5863148 64.58
Store 03 (root@server):~# xfs_db -c "freesp -a 2" /dev/sdb
from to extents blocks pct
1 1 819098 819098 9.05
2 3 846158 2074235 22.93
4 7 1538593 6154372 68.02
Store 03 (root@server):~# xfs_db -c "freesp -a 3" /dev/sdb
from to extents blocks pct
1 1 814870 814870 8.91
2 3 809611 1963787 21.48
4 7 1591044 6364176 69.61
[-- Attachment #1.2: Type: text/html, Size: 4721 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Question about continous blocks for inode due to 'no space left on device' problem
2016-02-05 13:13 Question about continous blocks for inode due to 'no space left on device' problem Michel Verbraak
@ 2016-02-05 22:30 ` Dave Chinner
2016-02-08 13:09 ` Michel Verbraak
0 siblings, 1 reply; 5+ messages in thread
From: Dave Chinner @ 2016-02-05 22:30 UTC (permalink / raw)
To: Michel Verbraak; +Cc: xfs
On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> All,
>
> Recently we ran into a problem where our filesystem (300GB in size)
> reported 'no space left on device' (ENOSPC) but when we looked at disk
> space usage and inode usage it was around 52% for disk space and 11% for
> inode. (sorry do not have a save of the output of df command).
....
> meta-data=/dev/sdb isize=256 agcount=6, agsize=19660800
Inode allocation requires aligned 16k extents.
......
> Some extra info about the system and files on the system:
> - Ubuntu 12.04.5 LTS
> - Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17
> 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> - 32 million files of which about 75% is smaller than 1k. Files are
> separated over different folders to keep the number of files per folder low.
You have lots of small single block files which will cause free
space to tend towards small extents as the filesystem fills up...
> - mount options: defaults,noatime,inode64,nobarrier
>
> Regards,
>
> Michel Verbraak.
>
> Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> from to extents blocks pct
> 1 1 3282633 3282633 9.03
> 2 3 3416223 8372325 23.03
> 4 7 6175009 24700036 67.94
Yup, no more aligned 4 block extents in the filesystem.
This is what the experimental sparse inode feature in current
kernels addresses - it allows unaligned block allocation for inodes
and so allows you to abuse the filesystem all the way up to 100%
full with workloads like this.
But without moving to a current 4.4 kernel and reformatting all your
filesystems, there's absolutely nothing you can do about it except
grow the filesystem to be larger every time you drive the filesystem
into this condition.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Question about continous blocks for inode due to 'no space left on device' problem
2016-02-05 22:30 ` Dave Chinner
@ 2016-02-08 13:09 ` Michel Verbraak
2016-02-08 14:35 ` Brian Foster
0 siblings, 1 reply; 5+ messages in thread
From: Michel Verbraak @ 2016-02-08 13:09 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
[-- Attachment #1.1: Type: text/plain, Size: 1973 bytes --]
2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > All,
> >
> > Recently we ran into a problem where our filesystem (300GB in size)
> > reported 'no space left on device' (ENOSPC) but when we looked at disk
> > space usage and inode usage it was around 52% for disk space and 11% for
> .......
>
> > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> > from to extents blocks pct
> > 1 1 3282633 3282633 9.03
> > 2 3 3416223 8372325 23.03
> > 4 7 6175009 24700036 67.94
>
> Yup, no more aligned 4 block extents in the filesystem.
>
> Dave Thank you for your answer but are you able to explain a bit more on
how you determine there are no aligned 4 block extends available?
It apparently is not the 'from 4 to 7' line telling there are at least
continuous extents of 4 blocks big. Or is it because 24700036 divided
by 6175009
is exactly 4 and not a bit more?
When I calculate the ratio between the individual AG freesp they were all
four exactly 4.
As said we would like to detect this problem from happening again. And we
would like this detection to be done by our monitoring service.
One of my colleagues mentions the following calculation to find out if
there are enough aligned 4 block extents:
Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
<extents-column>) > 0. When not there is not enough free space for new
inodes.
On another system we have which does not see these problems yet we have:
from to extents blocks pct
1 1 3483987 3483987 7.83
2 3 3560607 8709838 19.58
4 7 6402893 26486634 59.55
8 15 418450 5799206 13.04
26486634 / 6402893 = 4.136666 and
26486634 - (4 * 6402893) = 875062 > 0 and
5799205 - (4 * 418450) = 4125405 > 0
> into this condition.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
Michel.
[-- Attachment #1.2: Type: text/html, Size: 3436 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Question about continous blocks for inode due to 'no space left on device' problem
2016-02-08 13:09 ` Michel Verbraak
@ 2016-02-08 14:35 ` Brian Foster
2016-02-08 15:13 ` Michel Verbraak
0 siblings, 1 reply; 5+ messages in thread
From: Brian Foster @ 2016-02-08 14:35 UTC (permalink / raw)
To: Michel Verbraak; +Cc: xfs
On Mon, Feb 08, 2016 at 02:09:15PM +0100, Michel Verbraak wrote:
> 2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:
>
> > On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > > All,
> > >
> > > Recently we ran into a problem where our filesystem (300GB in size)
> > > reported 'no space left on device' (ENOSPC) but when we looked at disk
> > > space usage and inode usage it was around 52% for disk space and 11% for
> > .......
>
>
> >
> > > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> > > from to extents blocks pct
> > > 1 1 3282633 3282633 9.03
> > > 2 3 3416223 8372325 23.03
> > > 4 7 6175009 24700036 67.94
> >
> > Yup, no more aligned 4 block extents in the filesystem.
> >
> > Dave Thank you for your answer but are you able to explain a bit more on
> how you determine there are no aligned 4 block extends available?
> It apparently is not the 'from 4 to 7' line telling there are at least
> continuous extents of 4 blocks big. Or is it because 24700036 divided
> by 6175009
> is exactly 4 and not a bit more?
> When I calculate the ratio between the individual AG freesp they were all
> four exactly 4.
>
I'm not sure there is a way to determine whether these extents are
aligned or not short of walking the free space btree structures with
xfs_db checking directly.
IIRC from doing some of the sparse inode work, the slight caveat to this
whole thing is that it's technically not enough to have an aligned
extent available to satisfy an inode chunk allocation. The allocation
group has to have an extent large enough for the inode chunk plus some
set of blocks to _guarantee_ the allocation can be aligned for the
caller.
If you look at the code in xfs_alloc_space_available(), you can see that
in practice the allocator will start to fail when the longest free
extent in the AG is less than something along the lines of <extsize> +
<alignment>. For a 4k fsb fs w/ 256b inodes (and thus 16k inode chunks),
a quick test shows that the alignment is 8k (for a 24k total required
extent length).
I do wonder if we could do something basic here like search the AG for
an actual aligned extent, but I expect performance would be terrible so
this would be a last resort (and may or may not be useful). Another
option might be to scan or somehow or another track the number of inode
chunk aligned extents in each AG based on the inode alignment of the fs,
but I'd have to investigate how easy that is to see whether it's worth
it.
BTW, a metadump of your fs while it's in this ENOSPC state would be
useful towards testing something like that, if you'd be willing to
provide one. Note that xfs_metadump has obfuscation that hides filenames
and whatnot for privacy purposes.
Brian
> As said we would like to detect this problem from happening again. And we
> would like this detection to be done by our monitoring service.
> One of my colleagues mentions the following calculation to find out if
> there are enough aligned 4 block extents:
>
> Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
> <extents-column>) > 0. When not there is not enough free space for new
> inodes.
>
> On another system we have which does not see these problems yet we have:
>
> from to extents blocks pct
> 1 1 3483987 3483987 7.83
> 2 3 3560607 8709838 19.58
> 4 7 6402893 26486634 59.55
> 8 15 418450 5799206 13.04
>
> 26486634 / 6402893 = 4.136666 and
> 26486634 - (4 * 6402893) = 875062 > 0 and
> 5799205 - (4 * 418450) = 4125405 > 0
>
>
> > into this condition.
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
> >
>
> Michel.
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Question about continous blocks for inode due to 'no space left on device' problem
2016-02-08 14:35 ` Brian Foster
@ 2016-02-08 15:13 ` Michel Verbraak
0 siblings, 0 replies; 5+ messages in thread
From: Michel Verbraak @ 2016-02-08 15:13 UTC (permalink / raw)
To: Brian Foster; +Cc: xfs
[-- Attachment #1.1: Type: text/plain, Size: 3107 bytes --]
2016-02-08 15:35 GMT+01:00 Brian Foster <bfoster@redhat.com>:
> On Mon, Feb 08, 2016 at 02:09:15PM +0100, Michel Verbraak wrote:
> > 2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> >
> > > On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > > > All,
> > > >
> > > > Recently we ran into a problem where our filesystem (300GB in size)
> > > > reported 'no space left on device' (ENOSPC) but when we looked at
> disk
> > > > space usage and inode usage it was around 52% for disk space and 11%
> for
> > > .......
> >
> >
> > >
> > > > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> > > > from to extents blocks pct
> > > > 1 1 3282633 3282633 9.03
> > > > 2 3 3416223 8372325 23.03
> > > > 4 7 6175009 24700036 67.94
> > >
> > > Yup, no more aligned 4 block extents in the filesystem.
> > >
> > > Dave Thank you for your answer but are you able to explain a bit more
> on
> > how you determine there are no aligned 4 block extends available?
> > It apparently is not the 'from 4 to 7' line telling there are at least
> > continuous extents of 4 blocks big. Or is it because 24700036 divided
> > by 6175009
> > is exactly 4 and not a bit more?
> > When I calculate the ratio between the individual AG freesp they were all
> > four exactly 4.
> >
>
> I'm not sure there is a way to determine whether these extents are
> aligned or not short of walking the free space btree structures with
> ..<snip>
> BTW, a metadump of your fs while it's in this ENOSPC state would be
> useful towards testing something like that, if you'd be willing to
> provide one. Note that xfs_metadump has obfuscation that hides filenames
> and whatnot for privacy purposes.
>
> If we do run into problem again, we do hope not as it is on a production
system, we will create the xfs_metadump and provide it to this list.
Thank you for the extra explanation.
Cheers Michel.
> Brian
>
> > As said we would like to detect this problem from happening again. And we
> > would like this detection to be done by our monitoring service.
> > One of my colleagues mentions the following calculation to find out if
> > there are enough aligned 4 block extents:
> >
> > Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
> > <extents-column>) > 0. When not there is not enough free space for new
> > inodes.
> >
> > On another system we have which does not see these problems yet we have:
> >
> > from to extents blocks pct
> > 1 1 3483987 3483987 7.83
> > 2 3 3560607 8709838 19.58
> > 4 7 6402893 26486634 59.55
> > 8 15 418450 5799206 13.04
> >
> > 26486634 / 6402893 = 4.136666 and
> > 26486634 - (4 * 6402893) = 875062 > 0 and
> > 5799205 - (4 * 418450) = 4125405 > 0
> >
> >
> > > into this condition.
> > >
> > > Cheers,
> > >
> > > Dave.
> > > --
> > > Dave Chinner
> > > david@fromorbit.com
> > >
> >
> > Michel.
>
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
>
[-- Attachment #1.2: Type: text/html, Size: 4729 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-02-08 15:13 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-02-05 13:13 Question about continous blocks for inode due to 'no space left on device' problem Michel Verbraak
2016-02-05 22:30 ` Dave Chinner
2016-02-08 13:09 ` Michel Verbraak
2016-02-08 14:35 ` Brian Foster
2016-02-08 15:13 ` Michel Verbraak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox