XFS filesystem reports as full though it isn't

XFS filesystem reports as full though it isn't

[Emmanuel Florac]

I have a 64 bits (x86_64) machine running Linux 2.6.22.19 with a 24TB
XFS filesystem. There are some 15TB of data on it. All is well, no
error except that I can't create a single file (touch foo : no space
left on device). I don't understand what can be going wrong... 

History : this filesystem was extended (xfs_growfs) from 16TB to 24.

I've tought about all XFS problems I can remember of; first, inode
exhaustion :

cluster1:/proc/sys/fs# df -i
Sys. de fich.         Inodes   IUtil.  ILib. %IUti. Mont?sur
/dev/sda1                  0       0       0    -  /
tmpfs                2058624       3 2058621    1% /lib/init/rw
udev                 2058624    1445 2057179    1% /dev
tmpfs                2058624       1 2058623    1% /dev/shm
/dev/mapper/vg0-lv0  25379856384    4287 25379852097    1% /mnt/raid

OK so there's still plenty of inodes available here, that should be OK.
Then maybe we have something wrong elsewhere :

cluster1:/proc/sys/fs# xfs_db -r -c 'freesp -s -a 0' /dev/vg0/lv0
   from      to extents  blocks    pct
      1       1       5       5  71.43
      2       3       1       2  28.57
total free extents 6
total free blocks 7
average free extent size 1.16667

7 blocks free? But there are 9 TB of free space! 

Here is the output from xfs_info /dev/vg0/lv0

meta-data=/dev/vg0/lv0           isize=256    agcount=47,
agsize=137245616 blks =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=6344964096,
imaxpct=25 =                       sunit=16     swidth=32 blks,
unwritten=1 naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=131072 blocks=0, rtextents=0

I fail to see nothing special there however.

The only significant thing I see is that the FS is really close to 16
TB of allocated data (15.7TB). I tried mounting it with "inode64"
option with no more loving.

Any help would be tremendously welcome. Users are starving for more
space and they can't even create a single file! 

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Fri, 16 May 2008 22:27:55 +0200 vous écriviez:

> History : this filesystem was extended (xfs_growfs) from 16TB to 24.
> 

One important thing I forgot to mention : some time before I
have extended the FS from 16 to 24 TB, the FS got filled up commpletely
(186MB free!!!). I've asked the user to make some room to avoid
problems, so they freed 1.6 TB in a couple of days before growing up
the FS. Is it possible that something got wrong with the filesystem
being too filled?

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS filesystem reports as full though it isn't

[Christoph Hellwig]

On Fri, May 16, 2008 at 10:27:55PM +0200, Emmanuel Florac wrote:
> cluster1:/proc/sys/fs# xfs_db -r -c 'freesp -s -a 0' /dev/vg0/lv0
>    from      to extents  blocks    pct
>       1       1       5       5  71.43
>       2       3       1       2  28.57
> total free extents 6
> total free blocks 7
> average free extent size 1.16667
> 
> 7 blocks free? But there are 9 TB of free space! 

The -a 0 means you're only looking at the first AG, which given the
history of the filesystem has a fair chance to be full.  To get the
summary for all AG do a

	xfs_db -r -c 'freesp -s' /dev/vg0/lv0

No idea yet on what could cause your problem, sorry.

> The only significant thing I see is that the FS is really close to 16
> TB of allocated data (15.7TB). I tried mounting it with "inode64"
> option with no more loving.

Did you remount it with that option as in mount -o remount or did you
completely unmount it and mount it again?  Unfortunately none of the
xfs-specific mount options can be reset with remount yet, and it doesn't
return an error either (both are added to my todo list now)

Re: XFS filesystem reports as full though it isn't

[Christoph Hellwig]

On Sun, May 18, 2008 at 09:18:55AM -0400, Christoph Hellwig wrote:
> > The only significant thing I see is that the FS is really close to 16
> > TB of allocated data (15.7TB). I tried mounting it with "inode64"
> > option with no more loving.
> 
> Did you remount it with that option as in mount -o remount or did you
> completely unmount it and mount it again?  Unfortunately none of the
> xfs-specific mount options can be reset with remount yet, and it doesn't
> return an error either (both are added to my todo list now)

Oh, and please post the output of /proc/self/mounts to see if all
expected mount options are actualy seen by the filesystem.

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Sun, 18 May 2008 09:21:27 -0400 vous écriviez:

> > Did you remount it with that option as in mount -o remount or did
> > you completely unmount it and mount it again?  Unfortunately none
> > of the xfs-specific mount options can be reset with remount yet,
> > and it doesn't return an error either (both are added to my todo
> > list now)  
> 

IIRC I tried both, however I'll double check monday...

> Oh, and please post the output of /proc/self/mounts to see if all
> expected mount options are actualy seen by the filesystem.

OK, thanks.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS filesystem reports as full though it isn't

[Christian Røsnes]

On Fri, May 16, 2008 at 10:27 PM, Emmanuel Florac
<eflorac@intellique.com> wrote:
>
> I have a 64 bits (x86_64) machine running Linux 2.6.22.19 with a 24TB
> XFS filesystem. There are some 15TB of data on it. All is well, no
> error except that I can't create a single file (touch foo : no space
> left on device). I don't understand what can be going wrong...
>
> History : this filesystem was extended (xfs_growfs) from 16TB to 24.
>
>
> Here is the output from xfs_info /dev/vg0/lv0
>
> meta-data=/dev/vg0/lv0           isize=256    agcount=47,
> agsize=137245616 blks =                       sectsz=512   attr=0
> data     =                       bsize=4096   blocks=6344964096,
> imaxpct=25 =                       sunit=16     swidth=32 blks,
> unwritten=1 naming   =version 2              bsize=4096
> log      =internal               bsize=4096   blocks=32768, version=1
>         =                       sectsz=512   sunit=0 blks
> realtime =none                   extsz=131072 blocks=0, rtextents=0
>
> I fail to see nothing special there however.
>
> The only significant thing I see is that the FS is really close to 16
> TB of allocated data (15.7TB). I tried mounting it with "inode64"
> option with no more loving.
>

On my system I get "no space left on device"  when I reach 99% full
with about 20GB free space left on 2TB partitions.
I also use sunit and swidth for the data section of the xfs
filesystem, and it could be that the XFS system cannot allocate
space according to these parameters ? You seem to be around the
original 16TB limit, and maybe it tries to allocate
from this original disk layout ? (I'm no XFS expert, so please take my
"theories" with a grain of salt)

I recently had two "identical" partitions: A and B, where A was the
master and B was the rsync copy. These had been
written to for several years, and during that period they both have
had the role as master. I suppose the disk usage layout
was different. All of a sudden partition B reported "no space left on
device", even though partition A contained the
same data without any "no space left on device". What I did to get the
B to copy the missing data from partition A was to:
Temporarily move some data away from partition B, then run xfs_fsr on
partition B, then move the data back.
It brought the fragmentation down on partition B, and I could copy the
missing data from partition B.

Christian

Re: XFS filesystem reports as full though it isn't

[Christian Røsnes]

On Mon, May 19, 2008 at 1:31 PM, Christian Røsnes
<christian.rosnes@gmail.com> wrote:
> On Fri, May 16, 2008 at 10:27 PM, Emmanuel Florac
> <eflorac@intellique.com> wrote:
>>
>> I have a 64 bits (x86_64) machine running Linux 2.6.22.19 with a 24TB
>> XFS filesystem. There are some 15TB of data on it. All is well, no
>> error except that I can't create a single file (touch foo : no space
>> left on device). I don't understand what can be going wrong...
>>
>> History : this filesystem was extended (xfs_growfs) from 16TB to 24.
>>
>>
>> Here is the output from xfs_info /dev/vg0/lv0
>>
>> meta-data=/dev/vg0/lv0           isize=256    agcount=47,
>> agsize=137245616 blks =                       sectsz=512   attr=0
>> data     =                       bsize=4096   blocks=6344964096,
>> imaxpct=25 =                       sunit=16     swidth=32 blks,
>> unwritten=1 naming   =version 2              bsize=4096
>> log      =internal               bsize=4096   blocks=32768, version=1
>>         =                       sectsz=512   sunit=0 blks
>> realtime =none                   extsz=131072 blocks=0, rtextents=0
>>
>> I fail to see nothing special there however.
>>
>> The only significant thing I see is that the FS is really close to 16
>> TB of allocated data (15.7TB). I tried mounting it with "inode64"
>> option with no more loving.
>>
>
> On my system I get "no space left on device"  when I reach 99% full
> with about 20GB free space left on 2TB partitions.
> I also use sunit and swidth for the data section of the xfs
> filesystem, and it could be that the XFS system cannot allocate
> space according to these parameters ? You seem to be around the
> original 16TB limit, and maybe it tries to allocate
> from this original disk layout ? (I'm no XFS expert, so please take my
> "theories" with a grain of salt)
>
> I recently had two "identical" partitions: A and B, where A was the
> master and B was the rsync copy.

Both these 2TB partitions (A and B) were 99% full.

> These had been
> written to for several years, and during that period they both have
> had the role as master. I suppose the disk usage layout
> was different. All of a sudden partition B reported "no space left on
> device", even though partition A contained the
> same data without any "no space left on device". What I did to get the
> B to copy the missing data from partition A was to:
> Temporarily move some data away from partition B, then run xfs_fsr on
> partition B, then move the data back.
> It brought the fragmentation down on partition B, and I could copy the
> missing data from partition B.
>
Correction to last sentence: ... and I could copy the missing data
_to_ partition B.

Christian

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Sun, 18 May 2008 15:42:08 +0200 vous écriviez:

> IIRC I tried both, however I'll double check monday...

Well obviously I didn't try both, because it worked as expected when
using "inode64". However there'ssomething slightly weird in the data
layout:

/dev/dm-0:
AG     Inodes    IUsed    IFree    Use%
  0      4288     4277       11     99%
  1         0        0        0      0%
  2         0        0        0      0%
  3         0        0        0      0%
  4         0        0        0      0%
  5         0        0        0      0%
  6         0        0        0      0%
  7         0        0        0      0%
  8         0        0        0      0%
  9         0        0        0      0%
 10         0        0        0      0%
 11         0        0        0      0%
 12         0        0        0      0%
 13         0        0        0      0%
 14         0        0        0      0%
 15         0        0        0      0%
 16         0        0        0      0%
 17         0        0        0      0%
 18         0        0        0      0%
 19         0        0        0      0%
 20         0        0        0      0%
 21         0        0        0      0%
 22         0        0        0      0%
 23         0        0        0      0%
 24         0        0        0      0%
 25         0        0        0      0%
 26         0        0        0      0%
 27         0        0        0      0%
 28         0        0        0      0%
 29         0        0        0      0%
 30         0        0        0      0%
 31         0        0        0      0%
 32         0        0        0      0%
 33         0        0        0      0%
 34         0        0        0      0%
 35         0        0        0      0%
 36         0        0        0      0%
 37         0        0        0      0%
 38         0        0        0      0%
 39         0        0        0      0%
 40         0        0        0      0%
 41         0        0        0      0%
 42         0        0        0      0%
 43         0        0        0      0%
 44         0        0        0      0%
 45         0        0        0      0%
 46         0        0        0      0%
ALL      4288     4277       11     99%

See how all inodes are in the same vg? is it OK?

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Mon, 19 May 2008 13:48:01 +0200 vous écriviez:

> Correction to last sentence: ... and I could copy the missing data
> _to_ partition B.

Another reminder that we should always keep our disks less than 95ù
full :)

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS filesystem reports as full though it isn't

[David Chinner]

On Mon, May 19, 2008 at 10:10:57PM +0200, Emmanuel Florac wrote:
> Le Sun, 18 May 2008 15:42:08 +0200 vous écriviez:
> 
> > IIRC I tried both, however I'll double check monday...
> 
> Well obviously I didn't try both, because it worked as expected when
> using "inode64". However there'ssomething slightly weird in the data
> layout:
> 
> /dev/dm-0:
> AG     Inodes    IUsed    IFree    Use%
>   0      4288     4277       11     99%
>   1         0        0        0      0%
>   2         0        0        0      0%
.....

If there's no free space in AGs > 0 when you enable inode64, then
you will still get enospc...

> See how all inodes are in the same vg? is it OK?
                                     ^^ AG

This is typical of inode32 allocation...

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Tue, 20 May 2008 16:10:47 +1000
David Chinner <dgc@sgi.com> écrivait:

> If there's no free space in AGs > 0 when you enable inode64, then
> you will still get enospc...
> 
> > See how all inodes are in the same vg? is it OK?  
>                                      ^^ AG
> 
> This is typical of inode32 allocation...

OK. I've mounted it "inode64" and apparently it worked fine. I
started copying 6 more TB of data, I'll check how it goes this
afternoon.

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

Re: XFS filesystem reports as full though it isn't

[Christoph Hellwig]

On Mon, May 19, 2008 at 10:10:57PM +0200, Emmanuel Florac wrote:
> Le Sun, 18 May 2008 15:42:08 +0200 vous ?criviez:
> 
> > IIRC I tried both, however I'll double check monday...
> 
> Well obviously I didn't try both, because it worked as expected when
> using "inode64".

I've submitted a patch that should give back an error when remounting
with an option not supported in remount, that should fix this little
oversight in the future.  Thanks for the report!

Re: XFS filesystem reports as full though it isn't

[Emmanuel Florac]

Le Tue, 20 May 2008 06:19:18 -0400
Christoph Hellwig <hch@infradead.org> écrivait:

> I've submitted a patch that should give back an error when remounting
> with an option not supported in remount, that should fix this little
> oversight in the future.  Thanks for the report!

Thanks for the patch :)

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------