Re: ENOSPC at 90% with plenty of inodes

[Dave Chinner <david@fromorbit.com>]

On Tue, Oct 12, 2010 at 06:21:42PM -0500, Stan Hoeppner wrote:
> Dave Chinner put forth on 10/12/2010 5:29 AM:
> > On Mon, Oct 11, 2010 at 08:27:00PM -0500, Stan Hoeppner wrote:
> >> Dave Chinner put forth on 10/11/2010 5:35 PM:
> >>> On Mon, Oct 11, 2010 at 03:03:28PM +0100, James Braid wrote:
> >>>> On Fri, Oct 8, 2010 at 23:51, Dave Chinner <david@fromorbit.com> wrote:
> >>>>> Sounds like fragmented free space. What is the output of:
> >>>>>
> >>>>> # xfs_db -r -c "freesp -s" <device>
> >>>>
> >>>> # xfs_db -r -c "freesp -s" /dev/sdb
> >>>>    from      to extents  blocks    pct
> >>>>       1       1 2298052 2298052  40.52
> >>>>       2       3 1568338 3337017  58.84
> >>>>       4       7    8432   35716   0.63
> >>>>       8      15      50     423   0.01
> >>>> total free extents 3874872
> >>>> total free blocks 5671208
> >>>> average free extent size 1.46359
> >>>>
> >>>> Which seems to say there are a few tiny pieces of free space
> >>>> available? The files that were failing to be written were a few
> >>>> hundred bytes in size.
> >>>
> >>> The error has nothing to do with the size of the files, but
> >>> everything to do with being able to allocate more inodes. Inode
> >>> allocation requires 4 contiguous blocks (for 256 byte inodes, more
> >>> for larger inodes) with alignment constraints. That means when you
> >>> run out of 8 block or larger free extents, inode allocation will
> >>> start failing and you'll get ENOSPC being reported.
> >>>
> >>>> We haven't seen any errors so far today, but xfs_fsr ran over the
> >>>> weekend, so perhaps I guess it's reorganized the filesystem.
> >>>
> >>> Only a little. xfs_fsr will not improve fragmented free space
> >>> conditions (indeed, it normally fragments free space more). The only
> >>> way to reduce the fragmentation of free space is to remove a
> >>> significant amount of data and inodes from the filesystem...
> >>
> >> Hay Dave, would a "backup/reformat/restore" help with free space
> >> fragmentation in this case?
> > 
> > Of course. But that's the last resort....
> > 
> >> If so, could/should the OP specify anything
> >> during the mkfs.xfs reformat that may help alleviate or mitigate his
> >> problem in the future?
> > 
> > No. These problems usually appear in filesystems that have run at
> > greater than 85-90% full for extended periods of time without being
> > emptied at all. Once you start to free up space, it naturally
> > defragments itself, but if you never free up any significant amount
> > of space in the filesytesm, this cannot occur and so fragmentation
> > just keeps getting worse....
> 
> So, given that this problem is on a production IMAP server, and the OP
> likely can't just willy nilly start deleting user files, would adding
> more disk (and assuming he's using LVM or somesuch) and growing the
> filesystem alleviate this inode issue?

As long as you are unsing inode64 then growing the filesystem will
alow more inodes to be allocated.

> Or would he be better off adding more disk, creating a new filesystem,
> and moving half or so of his mailboxen over to the new filesystem at the
> Cyrus (application) level?  I've never used Cyrus, though IIRC Dovecot
> can do this "split mail store" setup.

Sure, that'd work, too.

Fundamentally, moving data and inodes around after a grow (or new
filesystem is added) is the only way to reduce existing free space
fragmentation. Achieving this data movement is left as an exercise
for the reader.  ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs