Question about continous blocks for inode due to 'no space left on device' problem

Question about continous blocks for inode due to 'no space left on device' problem

[Michel Verbraak]

[-- Attachment #1.1: Type: text/plain, Size: 3680 bytes --]

All,

Recently we ran into a problem where our filesystem (300GB in size)
reported 'no space left on device' (ENOSPC) but when we looked at disk
space usage and inode usage it was around 52% for disk space and 11% for
inode. (sorry do not have a save of the output of df command).

Second we looked at the xfs_info and xfs_db -c freesp for each AG. Sadly we
do not have a copy anymore of the xfs_info from that time but I do have
that of the freesp (at end of this mail).

As we did not fully understand the output of the 'xfs_db -c freesp' we
simply added another 100GB to the filesystem and our problem was solved for
now. The AG count went from 4 to 6.

For now we are looking into how can we detect (monitor) this filesystem and
see the same problem creeping up to us before it happens.
After searching online, reading a lot of mail threads and info pages it
looks like we hit the problem of not having any continuous blocks of at
least 16k to hold a new inode (
http://oss.sgi.com/pipermail/xfs/2014-September/038301.html).

Currently the xfs_info (after adding 100GB) looks like this:

meta-data=/dev/sdb               isize=256    agcount=6, agsize=19660800
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096 blocks=104857600, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=38400, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

The output of 'xfs_db -c freesp' shows a historgram. We first thought the
"from" column shows the minimal number of continous blocks. And as each
block is 4K (4096) the row with 'from 4 to 7' contains at least 4
continuous blocks with a minimal size of 4x4k=16k. So our filesystem has
enough blocks and extents which are 16k for new inodes, atleast what we
think.

- So why did we still get the ENOSPC error?
- Where can I find a better explanation of the 'xfs_db -c freesp' output?
- What is the best way to monitor/detect this problem before it happens.

Some extra info about the system and files on the system:
- Ubuntu 12.04.5 LTS
- Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17
18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- 32 million files of which about 75% is smaller than 1k. Files are
separated over different folders to keep the number of files per folder low.
- mount options: defaults,noatime,inode64,nobarrier

Regards,

Michel Verbraak.

Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
   from      to extents  blocks    pct
      1       1 3282633 3282633   9.03
      2       3 3416223 8372325  23.03
      4       7 6175009 24700036  67.94

Store 03 (root@server):~# xfs_db -c "freesp -a 0" /dev/sdb
   from      to extents  blocks    pct
      1       1  813491  813491   8.95
      2       3  806828 1953270  21.50
      4       7 1579585 6318340  69.55

Store 03 (root@server):~# xfs_db -c "freesp -a 1" /dev/sdb
   from      to extents  blocks    pct
      1       1  835174  835174   9.20
      2       3  953626 2381033  26.22
      4       7 1465787 5863148  64.58

Store 03 (root@server):~# xfs_db -c "freesp -a 2" /dev/sdb
   from      to extents  blocks    pct
      1       1  819098  819098   9.05
      2       3  846158 2074235  22.93
      4       7 1538593 6154372  68.02

Store 03 (root@server):~# xfs_db -c "freesp -a 3" /dev/sdb
   from      to extents  blocks    pct
      1       1  814870  814870   8.91
      2       3  809611 1963787  21.48
      4       7 1591044 6364176  69.61

[-- Attachment #1.2: Type: text/html, Size: 4721 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Question about continous blocks for inode due to 'no space left on device' problem

[Dave Chinner]

On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> All,
> 
> Recently we ran into a problem where our filesystem (300GB in size)
> reported 'no space left on device' (ENOSPC) but when we looked at disk
> space usage and inode usage it was around 52% for disk space and 11% for
> inode. (sorry do not have a save of the output of df command).
....
> meta-data=/dev/sdb               isize=256    agcount=6, agsize=19660800

Inode allocation requires aligned 16k extents.
......
> Some extra info about the system and files on the system:
> - Ubuntu 12.04.5 LTS
> - Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17
> 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> - 32 million files of which about 75% is smaller than 1k. Files are
> separated over different folders to keep the number of files per folder low.

You have lots of small single block files which will cause free
space to tend towards small extents as the filesystem fills up...

> - mount options: defaults,noatime,inode64,nobarrier
> 
> Regards,
> 
> Michel Verbraak.
> 
> Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
>    from      to extents  blocks    pct
>       1       1 3282633 3282633   9.03
>       2       3 3416223 8372325  23.03
>       4       7 6175009 24700036  67.94

Yup, no more aligned 4 block extents in the filesystem.

This is what the experimental sparse inode feature in current
kernels addresses - it allows unaligned block allocation for inodes
and so allows you to abuse the filesystem all the way up to 100%
full with workloads like this.

But without moving to a current 4.4 kernel and reformatting all your
filesystems, there's absolutely nothing you can do about it except
grow the filesystem to be larger every time you drive the filesystem
into this condition.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Question about continous blocks for inode due to 'no space left on device' problem

[Michel Verbraak]

[-- Attachment #1.1: Type: text/plain, Size: 1973 bytes --]

2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:

> On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > All,
> >
> > Recently we ran into a problem where our filesystem (300GB in size)
> > reported 'no space left on device' (ENOSPC) but when we looked at disk
> > space usage and inode usage it was around 52% for disk space and 11% for
> .......


>
> > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> >    from      to extents  blocks    pct
> >       1       1 3282633 3282633   9.03
> >       2       3 3416223 8372325  23.03
> >       4       7 6175009 24700036  67.94
>
> Yup, no more aligned 4 block extents in the filesystem.
>
> Dave Thank you for your answer but are you able to explain a bit more on
how you determine there are no aligned 4 block extends available?
It apparently is not the 'from 4 to 7' line telling there are at least
continuous extents of 4 blocks big. Or is it because 24700036 divided
by 6175009
is exactly 4 and not a bit more?
When I calculate the ratio between the individual AG freesp they were all
four exactly 4.

As said we would like to detect this problem from happening again. And we
would like this detection to be done by our monitoring service.
One of my colleagues mentions the following calculation to find out if
there are enough aligned 4 block extents:

Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
<extents-column>) > 0. When not there is not enough free space for new
inodes.

On another system we have which does not see these problems yet we have:

   from      to extents  blocks    pct
      1       1 3483987 3483987   7.83
      2       3 3560607 8709838  19.58
      4       7 6402893 26486634  59.55
      8      15  418450 5799206  13.04

26486634 / 6402893 = 4.136666 and
26486634 - (4 * 6402893) = 875062 > 0 and
5799205 - (4 * 418450) = 4125405 > 0


> into this condition.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

Michel.

[-- Attachment #1.2: Type: text/html, Size: 3436 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Question about continous blocks for inode due to 'no space left on device' problem

[Brian Foster]

On Mon, Feb 08, 2016 at 02:09:15PM +0100, Michel Verbraak wrote:
> 2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> 
> > On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > > All,
> > >
> > > Recently we ran into a problem where our filesystem (300GB in size)
> > > reported 'no space left on device' (ENOSPC) but when we looked at disk
> > > space usage and inode usage it was around 52% for disk space and 11% for
> > .......
> 
> 
> >
> > > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> > >    from      to extents  blocks    pct
> > >       1       1 3282633 3282633   9.03
> > >       2       3 3416223 8372325  23.03
> > >       4       7 6175009 24700036  67.94
> >
> > Yup, no more aligned 4 block extents in the filesystem.
> >
> > Dave Thank you for your answer but are you able to explain a bit more on
> how you determine there are no aligned 4 block extends available?
> It apparently is not the 'from 4 to 7' line telling there are at least
> continuous extents of 4 blocks big. Or is it because 24700036 divided
> by 6175009
> is exactly 4 and not a bit more?
> When I calculate the ratio between the individual AG freesp they were all
> four exactly 4.
> 

I'm not sure there is a way to determine whether these extents are
aligned or not short of walking the free space btree structures with
xfs_db checking directly.

IIRC from doing some of the sparse inode work, the slight caveat to this
whole thing is that it's technically not enough to have an aligned
extent available to satisfy an inode chunk allocation. The allocation
group has to have an extent large enough for the inode chunk plus some
set of blocks to _guarantee_ the allocation can be aligned for the
caller.

If you look at the code in xfs_alloc_space_available(), you can see that
in practice the allocator will start to fail when the longest free
extent in the AG is less than something along the lines of <extsize> +
<alignment>. For a 4k fsb fs w/ 256b inodes (and thus 16k inode chunks),
a quick test shows that the alignment is 8k (for a 24k total required
extent length).

I do wonder if we could do something basic here like search the AG for
an actual aligned extent, but I expect performance would be terrible so
this would be a last resort (and may or may not be useful). Another
option might be to scan or somehow or another track the number of inode
chunk aligned extents in each AG based on the inode alignment of the fs,
but I'd have to investigate how easy that is to see whether it's worth
it.

BTW, a metadump of your fs while it's in this ENOSPC state would be
useful towards testing something like that, if you'd be willing to
provide one. Note that xfs_metadump has obfuscation that hides filenames
and whatnot for privacy purposes. 

Brian

> As said we would like to detect this problem from happening again. And we
> would like this detection to be done by our monitoring service.
> One of my colleagues mentions the following calculation to find out if
> there are enough aligned 4 block extents:
> 
> Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
> <extents-column>) > 0. When not there is not enough free space for new
> inodes.
> 
> On another system we have which does not see these problems yet we have:
> 
>    from      to extents  blocks    pct
>       1       1 3483987 3483987   7.83
>       2       3 3560607 8709838  19.58
>       4       7 6402893 26486634  59.55
>       8      15  418450 5799206  13.04
> 
> 26486634 / 6402893 = 4.136666 and
> 26486634 - (4 * 6402893) = 875062 > 0 and
> 5799205 - (4 * 418450) = 4125405 > 0
> 
> 
> > into this condition.
> >
> > Cheers,
> >
> > Dave.
> > --
> > Dave Chinner
> > david@fromorbit.com
> >
> 
> Michel.

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Question about continous blocks for inode due to 'no space left on device' problem

[Michel Verbraak]

[-- Attachment #1.1: Type: text/plain, Size: 3107 bytes --]

2016-02-08 15:35 GMT+01:00 Brian Foster <bfoster@redhat.com>:

> On Mon, Feb 08, 2016 at 02:09:15PM +0100, Michel Verbraak wrote:
> > 2016-02-05 23:30 GMT+01:00 Dave Chinner <david@fromorbit.com>:
> >
> > > On Fri, Feb 05, 2016 at 02:13:00PM +0100, Michel Verbraak wrote:
> > > > All,
> > > >
> > > > Recently we ran into a problem where our filesystem (300GB in size)
> > > > reported 'no space left on device' (ENOSPC) but when we looked at
> disk
> > > > space usage and inode usage it was around 52% for disk space and 11%
> for
> > > .......
> >
> >
> > >
> > > > Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
> > > >    from      to extents  blocks    pct
> > > >       1       1 3282633 3282633   9.03
> > > >       2       3 3416223 8372325  23.03
> > > >       4       7 6175009 24700036  67.94
> > >
> > > Yup, no more aligned 4 block extents in the filesystem.
> > >
> > > Dave Thank you for your answer but are you able to explain a bit more
> on
> > how you determine there are no aligned 4 block extends available?
> > It apparently is not the 'from 4 to 7' line telling there are at least
> > continuous extents of 4 blocks big. Or is it because 24700036 divided
> > by 6175009
> > is exactly 4 and not a bit more?
> > When I calculate the ratio between the individual AG freesp they were all
> > four exactly 4.
> >
>
> I'm not sure there is a way to determine whether these extents are
> aligned or not short of walking the free space btree structures with
> ..<snip>



> BTW, a metadump of your fs while it's in this ENOSPC state would be
> useful towards testing something like that, if you'd be willing to
> provide one. Note that xfs_metadump has obfuscation that hides filenames
> and whatnot for privacy purposes.
>
> If we do run into problem again, we do hope not as it is on a production
system, we will create the xfs_metadump and provide it to this list.

Thank you for the extra explanation.

Cheers Michel.


> Brian
>
> > As said we would like to detect this problem from happening again. And we
> > would like this detection to be done by our monitoring service.
> > One of my colleagues mentions the following calculation to find out if
> > there are enough aligned 4 block extents:
> >
> > Starting from line "from 4 to 7" calculate: <blocks-column> - (4 *
> > <extents-column>) > 0. When not there is not enough free space for new
> > inodes.
> >
> > On another system we have which does not see these problems yet we have:
> >
> >    from      to extents  blocks    pct
> >       1       1 3483987 3483987   7.83
> >       2       3 3560607 8709838  19.58
> >       4       7 6402893 26486634  59.55
> >       8      15  418450 5799206  13.04
> >
> > 26486634 / 6402893 = 4.136666 and
> > 26486634 - (4 * 6402893) = 875062 > 0 and
> > 5799205 - (4 * 418450) = 4125405 > 0
> >
> >
> > > into this condition.
> > >
> > > Cheers,
> > >
> > > Dave.
> > > --
> > > Dave Chinner
> > > david@fromorbit.com
> > >
> >
> > Michel.
>
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
>

[-- Attachment #1.2: Type: text/html, Size: 4729 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs