ENOSPC at 90% with plenty of inodes

ENOSPC at 90% with plenty of inodes

[James Braid]

Hi,

On a fairly busy mail server hosting about 1000 cyrus mailboxes (each
message is stored as a separate file), we are observing ENOSPC errors
from the cyrus pop/imap daemons when attempting to create (tiny)
files. e.g.

Oct  8 12:18:49 xxx cyrus/pop3[14119]: IOERROR: creating
/user/imap/proc/14119: No space left on device
Oct  8 12:18:49 xxx cyrus/lmtpunix[14361]: IOERROR: creating quota
file /user/imap/quota/m/user.xx.NEW: No space left on device
Oct  8 12:18:49 xxx cyrus/imap[12804]: IOERROR: creating quota file
/user/imap/quota/d/user.xxx.NEW: No space left on device

The filesystem appears to have plenty of space available:

# df -i /user
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            102294688 10201313 92093375   10% /user
# df /user
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdb1            213014148 190023408  22990740  90% /user

Any ideas for how can we go about debugging where ENOSPC is coming
from? Unfortunately it's running a vintage 2.6.24 kernel (Ubuntu 8.04)
- we're trying to get it upgraded to 10.04 (2.6.32)

Thanks, James

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Richard Scobie]

James Braid wrote:

 > Any ideas for how can we go about debugging where ENOSPC is coming
 > from?

Hi James,

You're not hitting this issue are you?

<http://xfs.org/index.php/XFS_FAQ#Q:_What_is_the_inode64_mount_option_for.3F>

Regards,

Richard

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Emmanuel Florac]

Le Fri, 8 Oct 2010 18:17:52 +0100 vous écriviez:

> On a fairly busy mail server hosting about 1000 cyrus mailboxes (each
> message is stored as a separate file), we are observing ENOSPC errors
> from the cyrus pop/imap daemons when attempting to create (tiny)
> files. e.g.

You must be running out of inodes, this probably should be a FAQ...
Check with xfs_info how much space is dedicated to inodes, and expand
it with xfs_growfs -m.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Emmanuel Florac]

Le Fri, 8 Oct 2010 18:17:52 +0100 vous écriviez:

> The filesystem appears to have plenty of space available:
> 

I forgot a point : 10% is NOT plenty of space. At 90%, a filesystem
must be considered full, particularly with lots of inodes, even if the
available space is several terabytes. System administration 101...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Dave Chinner]

On Fri, Oct 08, 2010 at 06:17:52PM +0100, James Braid wrote:
> Hi,
> 
> On a fairly busy mail server hosting about 1000 cyrus mailboxes (each
> message is stored as a separate file), we are observing ENOSPC errors
> from the cyrus pop/imap daemons when attempting to create (tiny)
> files. e.g.
> 
> Oct  8 12:18:49 xxx cyrus/pop3[14119]: IOERROR: creating
> /user/imap/proc/14119: No space left on device
> Oct  8 12:18:49 xxx cyrus/lmtpunix[14361]: IOERROR: creating quota
> file /user/imap/quota/m/user.xx.NEW: No space left on device
> Oct  8 12:18:49 xxx cyrus/imap[12804]: IOERROR: creating quota file
> /user/imap/quota/d/user.xxx.NEW: No space left on device
> 
> The filesystem appears to have plenty of space available:
> 
> # df -i /user
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sdb1            102294688 10201313 92093375   10% /user
> # df /user
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdb1            213014148 190023408  22990740  90% /user
> 
> Any ideas for how can we go about debugging where ENOSPC is coming
> from? Unfortunately it's running a vintage 2.6.24 kernel (Ubuntu 8.04)
> - we're trying to get it upgraded to 10.04 (2.6.32)

Sounds like fragmented free space. What is the output of:

# xfs_db -r -c "freesp -s" <device>

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[James Braid]

On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
> Sounds like fragmented free space. What is the output of:
>
> # xfs_db -r -c "freesp -s" <device>

# xfs_db -r -c "freesp -s" /dev/sdb
   from      to extents  blocks    pct
      1       1 2298052 2298052  40.52
      2       3 1568338 3337017  58.84
      4       7    8432   35716   0.63
      8      15      50     423   0.01
total free extents 3874872
total free blocks 5671208
average free extent size 1.46359

Which seems to say there are a few tiny pieces of free space
available? The files that were failing to be written were a few
hundred bytes in size.

We haven't seen any errors so far today, but xfs_fsr ran over the
weekend, so perhaps I guess it's reorganized the filesystem.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Dave Chinner]

On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
> > Sounds like fragmented free space. What is the output of:
> >
> > # xfs_db -r -c "freesp -s" <device>
> 
> # xfs_db -r -c "freesp -s" /dev/sdb
>    from      to extents  blocks    pct
>       1       1 2298052 2298052  40.52
>       2       3 1568338 3337017  58.84
>       4       7    8432   35716   0.63
>       8      15      50     423   0.01
> total free extents 3874872
> total free blocks 5671208
> average free extent size 1.46359
> 
> Which seems to say there are a few tiny pieces of free space
> available? The files that were failing to be written were a few
> hundred bytes in size.

The error has nothing to do with the size of the files, but
everything to do with being able to allocate more inodes. Inode
allocation requires 4 contiguous blocks (for 256 byte inodes, more
for larger inodes) with alignment constraints. That means when you
run out of 8 block or larger free extents, inode allocation will
start failing and you'll get ENOSPC being reported.

> We haven't seen any errors so far today, but xfs_fsr ran over the
> weekend, so perhaps I guess it's reorganized the filesystem.

Only a little. xfs_fsr will not improve fragmented free space
conditions (indeed, it normally fragments free space more). The only
way to reduce the fragmentation of free space is to remove a
significant amount of data and inodes from the filesystem...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Stan Hoeppner]

Dave Chinner put forth on 10/11/2010 5:35 PM:
> On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
>> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
>>> Sounds like fragmented free space. What is the output of:
>>>
>>> # xfs_db -r -c "freesp -s" <device>
>>
>> # xfs_db -r -c "freesp -s" /dev/sdb
>>    from      to extents  blocks    pct
>>       1       1 2298052 2298052  40.52
>>       2       3 1568338 3337017  58.84
>>       4       7    8432   35716   0.63
>>       8      15      50     423   0.01
>> total free extents 3874872
>> total free blocks 5671208
>> average free extent size 1.46359
>>
>> Which seems to say there are a few tiny pieces of free space
>> available? The files that were failing to be written were a few
>> hundred bytes in size.
> 
> The error has nothing to do with the size of the files, but
> everything to do with being able to allocate more inodes. Inode
> allocation requires 4 contiguous blocks (for 256 byte inodes, more
> for larger inodes) with alignment constraints. That means when you
> run out of 8 block or larger free extents, inode allocation will
> start failing and you'll get ENOSPC being reported.
> 
>> We haven't seen any errors so far today, but xfs_fsr ran over the
>> weekend, so perhaps I guess it's reorganized the filesystem.
> 
> Only a little. xfs_fsr will not improve fragmented free space
> conditions (indeed, it normally fragments free space more). The only
> way to reduce the fragmentation of free space is to remove a
> significant amount of data and inodes from the filesystem...

Hay Dave, would a "backup/reformat/restore" help with free space
fragmentation in this case?  If so, could/should the OP specify anything
during the mkfs.xfs reformat that may help alleviate or mitigate his
problem in the future?

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Dave Chinner]

On Mon, Oct 11, 2010 at 08:27:00PM -0500, Stan Hoeppner wrote:
> Dave Chinner put forth on 10/11/2010 5:35 PM:
> > On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
> >> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
> >>> Sounds like fragmented free space. What is the output of:
> >>>
> >>> # xfs_db -r -c "freesp -s" <device>
> >>
> >> # xfs_db -r -c "freesp -s" /dev/sdb
> >>    from      to extents  blocks    pct
> >>       1       1 2298052 2298052  40.52
> >>       2       3 1568338 3337017  58.84
> >>       4       7    8432   35716   0.63
> >>       8      15      50     423   0.01
> >> total free extents 3874872
> >> total free blocks 5671208
> >> average free extent size 1.46359
> >>
> >> Which seems to say there are a few tiny pieces of free space
> >> available? The files that were failing to be written were a few
> >> hundred bytes in size.
> > 
> > The error has nothing to do with the size of the files, but
> > everything to do with being able to allocate more inodes. Inode
> > allocation requires 4 contiguous blocks (for 256 byte inodes, more
> > for larger inodes) with alignment constraints. That means when you
> > run out of 8 block or larger free extents, inode allocation will
> > start failing and you'll get ENOSPC being reported.
> > 
> >> We haven't seen any errors so far today, but xfs_fsr ran over the
> >> weekend, so perhaps I guess it's reorganized the filesystem.
> > 
> > Only a little. xfs_fsr will not improve fragmented free space
> > conditions (indeed, it normally fragments free space more). The only
> > way to reduce the fragmentation of free space is to remove a
> > significant amount of data and inodes from the filesystem...
> 
> Hay Dave, would a "backup/reformat/restore" help with free space
> fragmentation in this case?

Of course. But that's the last resort....

> If so, could/should the OP specify anything
> during the mkfs.xfs reformat that may help alleviate or mitigate his
> problem in the future?

No. These problems usually appear in filesystems that have run at
greater than 85-90% full for extended periods of time without being
emptied at all. Once you start to free up space, it naturally
defragments itself, but if you never free up any significant amount
of space in the filesytesm, this cannot occur and so fragmentation
just keeps getting worse....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Jan Derfinak]

Dave Chinner wrote:

>> If so, could/should the OP specify anything
>> during the mkfs.xfs reformat that may help alleviate or mitigate his
>> problem in the future?
> 
> No. These problems usually appear in filesystems that have run at
> greater than 85-90% full for extended periods of time without being
> emptied at all. Once you start to free up space, it naturally
> defragments itself, but if you never free up any significant amount
> of space in the filesytesm, this cannot occur and so fragmentation
> just keeps getting worse....

I'm curious if using noikeep mount option has any effect on free space
fragmentation?

jan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Stan Hoeppner]

Dave Chinner put forth on 10/12/2010 5:29 AM:
> On Mon, Oct 11, 2010 at 08:27:00PM -0500, Stan Hoeppner wrote:
>> Dave Chinner put forth on 10/11/2010 5:35 PM:
>>> On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
>>>> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
>>>>> Sounds like fragmented free space. What is the output of:
>>>>>
>>>>> # xfs_db -r -c "freesp -s" <device>
>>>>
>>>> # xfs_db -r -c "freesp -s" /dev/sdb
>>>>    from      to extents  blocks    pct
>>>>       1       1 2298052 2298052  40.52
>>>>       2       3 1568338 3337017  58.84
>>>>       4       7    8432   35716   0.63
>>>>       8      15      50     423   0.01
>>>> total free extents 3874872
>>>> total free blocks 5671208
>>>> average free extent size 1.46359
>>>>
>>>> Which seems to say there are a few tiny pieces of free space
>>>> available? The files that were failing to be written were a few
>>>> hundred bytes in size.
>>>
>>> The error has nothing to do with the size of the files, but
>>> everything to do with being able to allocate more inodes. Inode
>>> allocation requires 4 contiguous blocks (for 256 byte inodes, more
>>> for larger inodes) with alignment constraints. That means when you
>>> run out of 8 block or larger free extents, inode allocation will
>>> start failing and you'll get ENOSPC being reported.
>>>
>>>> We haven't seen any errors so far today, but xfs_fsr ran over the
>>>> weekend, so perhaps I guess it's reorganized the filesystem.
>>>
>>> Only a little. xfs_fsr will not improve fragmented free space
>>> conditions (indeed, it normally fragments free space more). The only
>>> way to reduce the fragmentation of free space is to remove a
>>> significant amount of data and inodes from the filesystem...
>>
>> Hay Dave, would a "backup/reformat/restore" help with free space
>> fragmentation in this case?
> 
> Of course. But that's the last resort....
> 
>> If so, could/should the OP specify anything
>> during the mkfs.xfs reformat that may help alleviate or mitigate his
>> problem in the future?
> 
> No. These problems usually appear in filesystems that have run at
> greater than 85-90% full for extended periods of time without being
> emptied at all. Once you start to free up space, it naturally
> defragments itself, but if you never free up any significant amount
> of space in the filesytesm, this cannot occur and so fragmentation
> just keeps getting worse....

So, given that this problem is on a production IMAP server, and the OP
likely can't just willy nilly start deleting user files, would adding
more disk (and assuming he's using LVM or somesuch) and growing the
filesystem alleviate this inode issue?

Or would he be better off adding more disk, creating a new filesystem,
and moving half or so of his mailboxen over to the new filesystem at the
Cyrus (application) level?  I've never used Cyrus, though IIRC Dovecot
can do this "split mail store" setup.

-- 
Stan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[Dave Chinner]

On Tue, Oct 12, 2010 at 06:21:42PM -0500, Stan Hoeppner wrote:
> Dave Chinner put forth on 10/12/2010 5:29 AM:
> > On Mon, Oct 11, 2010 at 08:27:00PM -0500, Stan Hoeppner wrote:
> >> Dave Chinner put forth on 10/11/2010 5:35 PM:
> >>> On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
> >>>> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
> >>>>> Sounds like fragmented free space. What is the output of:
> >>>>>
> >>>>> # xfs_db -r -c "freesp -s" <device>
> >>>>
> >>>> # xfs_db -r -c "freesp -s" /dev/sdb
> >>>>    from      to extents  blocks    pct
> >>>>       1       1 2298052 2298052  40.52
> >>>>       2       3 1568338 3337017  58.84
> >>>>       4       7    8432   35716   0.63
> >>>>       8      15      50     423   0.01
> >>>> total free extents 3874872
> >>>> total free blocks 5671208
> >>>> average free extent size 1.46359
> >>>>
> >>>> Which seems to say there are a few tiny pieces of free space
> >>>> available? The files that were failing to be written were a few
> >>>> hundred bytes in size.
> >>>
> >>> The error has nothing to do with the size of the files, but
> >>> everything to do with being able to allocate more inodes. Inode
> >>> allocation requires 4 contiguous blocks (for 256 byte inodes, more
> >>> for larger inodes) with alignment constraints. That means when you
> >>> run out of 8 block or larger free extents, inode allocation will
> >>> start failing and you'll get ENOSPC being reported.
> >>>
> >>>> We haven't seen any errors so far today, but xfs_fsr ran over the
> >>>> weekend, so perhaps I guess it's reorganized the filesystem.
> >>>
> >>> Only a little. xfs_fsr will not improve fragmented free space
> >>> conditions (indeed, it normally fragments free space more). The only
> >>> way to reduce the fragmentation of free space is to remove a
> >>> significant amount of data and inodes from the filesystem...
> >>
> >> Hay Dave, would a "backup/reformat/restore" help with free space
> >> fragmentation in this case?
> > 
> > Of course. But that's the last resort....
> > 
> >> If so, could/should the OP specify anything
> >> during the mkfs.xfs reformat that may help alleviate or mitigate his
> >> problem in the future?
> > 
> > No. These problems usually appear in filesystems that have run at
> > greater than 85-90% full for extended periods of time without being
> > emptied at all. Once you start to free up space, it naturally
> > defragments itself, but if you never free up any significant amount
> > of space in the filesytesm, this cannot occur and so fragmentation
> > just keeps getting worse....
> 
> So, given that this problem is on a production IMAP server, and the OP
> likely can't just willy nilly start deleting user files, would adding
> more disk (and assuming he's using LVM or somesuch) and growing the
> filesystem alleviate this inode issue?

As long as you are unsing inode64 then growing the filesystem will
alow more inodes to be allocated.

> Or would he be better off adding more disk, creating a new filesystem,
> and moving half or so of his mailboxen over to the new filesystem at the
> Cyrus (application) level?  I've never used Cyrus, though IIRC Dovecot
> can do this "split mail store" setup.

Sure, that'd work, too.

Fundamentally, moving data and inodes around after a grow (or new
filesystem is added) is the only way to reduce existing free space
fragmentation. Achieving this data movement is left as an exercise
for the reader.  ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC at 90% with plenty of inodes

[XIE Zhengmao]

James Braid <jamesb <at> loreland.org> writes:

> 
> Hi,
> 

"no space left" when your still have space and inodes...
Please fsck the file system, too many fs errors cause this problem. 



_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs