No space left on device on xfs filesystem with 1.6TB free

No space left on device on xfs filesystem with 1.6TB free

[Owen Dunn]

I'm running SuSE Linux Enterprise Server 10 SP2, on a 32-bit i386 Linux 
machine with kernel version 2.6.16.60-0.33-smp.  I appear to have version 
2.9.4 of xfsprogs.

I have a 6TB xfs filesystem which has been grown twice from 2TB, and which 
appears to have 1.6TB and plenty of inodes free.  However, attempts to 
create a new file fail with ENOSPC:

sf3:/EXPORT/space # touch splat
touch: cannot touch `splat': No space left on device
sf3:/EXPORT/space # df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/bigvg-lv1
                       6.0T  4.5T  1.6T  75% /EXPORT/space
sf3:/EXPORT/space # df -ih .
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/bigvg-lv1
                         4.0G    887K    4.0G    1% /EXPORT/space

Any idea what's going on and how to fix it?  xfs_check stops straight away 
with `out of memory' and xfs_repair stops in Phase 6, presumably when it 
finds itself unable to make a new lost+found directory.

Owen

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Bryon Roche]

On Wed, 21 Jan 2009 10:33:10 +0000, Owen Dunn wrote:

> I'm running SuSE Linux Enterprise Server 10 SP2, on a 32-bit i386 Linux
> machine with kernel version 2.6.16.60-0.33-smp.  I appear to have
> version 2.9.4 of xfsprogs.
> 
> I have a 6TB xfs filesystem which has been grown twice from 2TB, and
> which appears to have 1.6TB and plenty of inodes free.  However,
> attempts to create a new file fail with ENOSPC:
You probably need to mount with inode64 then, to allow xfs to make inodes 
in the later space in the FS.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Patrick Schreurs]

Bryon Roche wrote:
> On Wed, 21 Jan 2009 10:33:10 +0000, Owen Dunn wrote:
> 
>> I'm running SuSE Linux Enterprise Server 10 SP2, on a 32-bit i386 Linux
>> machine with kernel version 2.6.16.60-0.33-smp.  I appear to have
>> version 2.9.4 of xfsprogs.
>>
>> I have a 6TB xfs filesystem which has been grown twice from 2TB, and
>> which appears to have 1.6TB and plenty of inodes free.  However,
>> attempts to create a new file fail with ENOSPC:
> You probably need to mount with inode64 then, to allow xfs to make inodes 
> in the later space in the FS.

We've seen this on 64-bit 2.6.23.x and 2.6.27.x kernels with inode64 
mounted partitions. xfsprogs 2.8.11 (Debian Etch).

Unfortunately we've never been able to track this down.

Patrick Schreurs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Dave Chinner]

On Wed, Jan 21, 2009 at 10:33:10AM +0000, Owen Dunn wrote:
> I'm running SuSE Linux Enterprise Server 10 SP2, on a 32-bit i386 Linux 
> machine with kernel version 2.6.16.60-0.33-smp.  I appear to have version 
> 2.9.4 of xfsprogs.
> 
> I have a 6TB xfs filesystem which has been grown twice from 2TB, and which 
> appears to have 1.6TB and plenty of inodes free.  However, attempts to 
> create a new file fail with ENOSPC:

You've run out of inode space below 1TB or free space is
sufficiently fragmented below 1TB that new inode chunks cannot
be allocated.

> sf3:/EXPORT/space # touch splat
> touch: cannot touch `splat': No space left on device
> sf3:/EXPORT/space # df -h .
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/bigvg-lv1
>                        6.0T  4.5T  1.6T  75% /EXPORT/space
> sf3:/EXPORT/space # df -ih .
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/mapper/bigvg-lv1
>                          4.0G    887K    4.0G    1% /EXPORT/space

df -i really doesn't tell you anything useful on filesystems that
dynamically allocate inodes like XFS.

> Any idea what's going on and how to fix it? 

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Patrick Schreurs]

Dave Chinner wrote:
> You've run out of inode space below 1TB or free space is
> sufficiently fragmented below 1TB that new inode chunks cannot
> be allocated.

Is this related to fs growth or can this happen with any xfs partition?

Thanks,

Patrick Schreurs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Eric Sandeen]

Patrick Schreurs wrote:
> Dave Chinner wrote:
>> You've run out of inode space below 1TB or free space is
>> sufficiently fragmented below 1TB that new inode chunks cannot
>> be allocated.
> 
> Is this related to fs growth or can this happen with any xfs partition?

a grown fs may be more susceptible to it...

xfs dynamically allocates inodes, and inode numbers reflect their
location on disk.

For large-offset disk locations, the inode numbers grow large, and can
be > 32 bits, which causes problems for some applications.

By default xfs keeps inode numbers below 32 bits, and hence restricts
them to the lower part of the filesystem, with data going out towards
the end.

If your fs started out small, more of that low-offset space will be
occupied by data, and be unavailable for inodes even as the filesystem
grows; in short when the fs was small, you used up the space that the
larger fs would have liked to use for inodes.

-Eric

> Thanks,
> 
> Patrick Schreurs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Christian Kujau]

On Thu, 22 Jan 2009, Dave Chinner wrote:
> The only way to fix this is to move data around to free up space
> below 1TB. Find your oldest data (i.e. that was around before even
> the first grow) and move it off the filesystem (move, not copy).
> Then if you copy it back on, the data blocks will end up above 1TB
> and that should leave you with plenty of space for inodes below 1TB.

With volumes bigger and bigger, should this procedure be mentioned in 
the xfs_growfs manpage or in the XFS FAQ?

Christian.
-- 
BOFH excuse #53:

Little hamster in running wheel had coronary; waiting for replacement to be Fedexed from Wyoming

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Dave Chinner]

On Thu, Jan 22, 2009 at 03:15:59PM +0100, Christian Kujau wrote:
> On Thu, 22 Jan 2009, Dave Chinner wrote:
>> The only way to fix this is to move data around to free up space
>> below 1TB. Find your oldest data (i.e. that was around before even
>> the first grow) and move it off the filesystem (move, not copy).
>> Then if you copy it back on, the data blocks will end up above 1TB
>> and that should leave you with plenty of space for inodes below 1TB.
>
> With volumes bigger and bigger, should this procedure be mentioned in  
> the xfs_growfs manpage or in the XFS FAQ?

Sure: the FAQ is a wiki now so you can add it yourself. ;)

http://xfs.org/index.php/XFS_FAQ

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device on xfs filesystem with 1.6TB free

[Christian Kujau]

On Fri, 23 Jan 2009, Dave Chinner wrote:
> Sure: the FAQ is a wiki now so you can add it yourself. ;)
> http://xfs.org/index.php/XFS_FAQ

Ah, the FAQ has moved - it's been a while since I visited 
http://oss.sgi.com/projects/xfs/faq.html :)

Thanks,
Christian.
-- 
BOFH excuse #109:

The electricity substation in the car park blew up.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs