XFS "no space left" problem

XFS "no space left" problem

[wildcat]

Hi dear list,


I recently migrated a 1.3Tbyte partition from reiserfs to XFS.
As the 1.3Tb partition was full, my idea was to shrink reiserfs
partition of 100Gig at a time, and then grow the XFS one.

This has been working since 1.1To for XFS partition. Now I'm in the
following situation:


/dev/mapper/crypt-data
                      1.2T  1.1T  134G  89% /jail2

dunno jail2 # touch a
dunno jail2 # touch b 
touch: cannot touch `b': No space left on device
dunno jail2 # 


>From there if I delete 'a', I could create a file of any size. but then
will be running
again into the "no space left".

>From here, some people told me of an inodes problem, that I checked
first:
/dev/mapper/crypt-data
                     561850144  158335 561691809    1% /jail2

is outputing from df -i.

umount/remount of the partition doesn't change anything. 


here are some information about the filesystem details:


dunno jail2 # xfs_info /jail2
meta-data=/dev/crypt/data        isize=256    agcount=188,
agsize=1638400 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=306708480,
imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=12800, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

dunno / # xfs_db /dev/crypt/data 
xfs_db> frag
actual 140099, ideal 134926, fragmentation factor 3.69%
xfs_db> freesp
   from      to extents  blocks    pct
      1       1     801     801   0.00
      2       3       2       5   0.00
     32      63       2      89   0.00
    128     255       3     517   0.00
    256     511       4    1676   0.00
    512    1023       5    3776   0.01
   1024    2047       7   10765   0.03
 262144  524287      23 9200216  26.21
1048576 1638400      16 25888649  73.74
xfs_db> 


I can't think of a Fix solution from here and I'm stuck.

As this box is in production and remotely managed; Go to the
place, take the hard drives and backup/restore is not a solution.

Any help/comment/fix/idea would be very appreciated.
Any additionnal information you need could be asked :)


TIA,
Best regards,



wildcat

Re: XFS "no space left" problem

[David Chinner]

On Sat, Feb 03, 2007 at 07:37:23PM +0100, wildcat wrote:
> Hi dear list,
> 
> 
> I recently migrated a 1.3Tbyte partition from reiserfs to XFS.
> As the 1.3Tb partition was full, my idea was to shrink reiserfs
> partition of 100Gig at a time, and then grow the XFS one.

So you've filled up the lower terabyte of the partition
before you added the space above 1TB. The problem
is almost certainly going to be that there's no room left
to allocate inodes below the 1TB mark (32 bit inode filesystem)
because all that space is taken up by data....

> dunno jail2 # xfs_info /jail2
> meta-data=/dev/crypt/data        isize=256    agcount=188,
> agsize=1638400 blks
>          =                       sectsz=512   attr=0
> data     =                       bsize=4096   blocks=306708480,
> imaxpct=25
>          =                       sunit=0      swidth=0 blks, unwritten=1
> naming   =version 2              bsize=4096  
> log      =internal               bsize=4096   blocks=12800, version=1
>          =                       sectsz=512   sunit=0 blks
> realtime =none                   extsz=65536  blocks=0, rtextents=0
> 
> dunno / # xfs_db /dev/crypt/data 
> xfs_db> frag
> actual 140099, ideal 134926, fragmentation factor 3.69%
> xfs_db> freesp
>    from      to extents  blocks    pct
>       1       1     801     801   0.00
>       2       3       2       5   0.00
>      32      63       2      89   0.00
>     128     255       3     517   0.00
>     256     511       4    1676   0.00
>     512    1023       5    3776   0.01
>    1024    2047       7   10765   0.03
>  262144  524287      23 9200216  26.21
> 1048576 1638400      16 25888649  73.74
> xfs_db> 

Mostly large blocks. I bet they are all in AGs above the 1TB mark.
One thing you can do is look at the freespace per AG:

# for ag in `seq 0 1 188`; do
> echo freespace in AG $ag
> xfs_db ­r -c "freesp -s -a $ag" /dev/crypt/data | grep "total free"
> done

And see how many of the AGs below 1TB (~6.4GB per AG, so those below about
about AG 160) have no free space. To allocate new inodes you need a contiguous
extent of 16k plus another 4 free blocks, so you need at _least_ 8 free
blocks in an AG to allocate a new inode chunk on disk.

If this is the case, then you need to delete some files to get space
free below 1TB for more inodes and then copy a large file or two so
they are moved above the 1TB mark, then remove the original copy
to make space for inodes below 1TB.

Or get a 64bit machine and use inode64. ;)

HTH.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS "no space left" problem

[Bernd Schubert]

David Chinner wrote:

> Or get a 64bit machine and use inode64. ;)

David, can you tell more in detail what you mean with "use inode64"?

We also just migrated our server to xfs, the largest partition has a size of
4TB. Lets say in the future a problem occurs and we could solve this by a
64-bit system? Migrating to 64-bit wouldn't be difficult, since the server
systems are already opterons. Only since we never bothered to migrate the
installation to x86_64 its not 64-bit yet. Would we need to tell xfs
somehow that it update its internal values to 64-bit integers or will it do
that automatically? 
Btw, if you need 64-bit integers, why don't you use long long, or even
better with C99 int64_t?

Thanks,
Bernd

Re: XFS "no space left" problem

[Eric Sandeen]

Bernd Schubert wrote:
> David Chinner wrote:
> 
>> Or get a 64bit machine and use inode64. ;)
> 
> David, can you tell more in detail what you mean with "use inode64"?
> 
> We also just migrated our server to xfs, the largest partition has a size of
> 4TB. Lets say in the future a problem occurs and we could solve this by a
> 64-bit system? Migrating to 64-bit wouldn't be difficult, since the server
> systems are already opterons. Only since we never bothered to migrate the
> installation to x86_64 its not 64-bit yet. Would we need to tell xfs
> somehow that it update its internal values to 64-bit integers or will it do
> that automatically? 

xfs is generally fully 64 bit, even on a 32 bit machine.  see more below...

> Btw, if you need 64-bit integers, why don't you use long long, or even
> better with C99 int64_t?

32-bit machines can only do 32-bit inodes (vfs limitation) (at least 
'til recently), so by default xfs doesn't let inode numbers go above 32 
bits - even though -internally- it -does-  use a 64-bit container.

The xfs inode number encodes its location on disk, so this restriction 
also restricts which blocks can be allocated as inodes.

xfs tries to reserve enough of these low blocks that you won't run out 
of potential inode blocks, but i've seen that get sideways.  The 
original poster probably ran into trouble because he started with a 100G 
filesystem & grew it 10 or so times.

if you mount with inode64 you'll allow inodes to be allocated with more 
  than 32 bits set.  But this can only work on a 64 bit machine.

and then you have to be sure your userspace can cope w/ 64 bit inode 
numbers too.

-Eric

Re: XFS "no space left" problem

[Bernd Schubert]

On Monday 05 February 2007 19:28, you wrote:
> Bernd Schubert wrote:
> > David Chinner wrote:
> xfs is generally fully 64 bit, even on a 32 bit machine.  see more below...
>
> > Btw, if you need 64-bit integers, why don't you use long long, or even
> > better with C99 int64_t?
>
> 32-bit machines can only do 32-bit inodes (vfs limitation) (at least
> 'til recently), so by default xfs doesn't let inode numbers go above 32
> bits - even though -internally- it -does-  use a 64-bit container.

Ah, I see, linux vfs limitation.

>
> The xfs inode number encodes its location on disk, so this restriction
> also restricts which blocks can be allocated as inodes.
>
> xfs tries to reserve enough of these low blocks that you won't run out
> of potential inode blocks, but i've seen that get sideways.  The
> original poster probably ran into trouble because he started with a 100G
> filesystem & grew it 10 or so times.

Well, we just also switched to a new raid device and before we bought it, we 
also made sure its possible to do an online expansion. So a migration from 6 
TB to 12 TB will come... I guess in two or three years. 
 
>
> if you mount with inode64 you'll allow inodes to be allocated with more
>   than 32 bits set.  But this can only work on a 64 bit machine.

Ah, its a mount option! And even undocumented in "man 8 mount". Thanks will 
add this to our wiki and write a debian bugreport.

>
> and then you have to be sure your userspace can cope w/ 64 bit inode
> numbers too.

Hmm, sizeof(ino_t) is 4 on x86_32, that will be a problem. We do have quite a 
lot of programs, not written very well and not compilable with recent 
compilers. On the other side, the linux nfs client converts inode numbers to 
32 bit, if it would cause an overflow. However, it doesn't do that on x86_64 
for 32bit programs. I know that quite well, due to another problem.


Thanks a lot for your insights,
Bernd


-- 
Bernd Schubert
PCI / Theoretische Chemie
Universität Heidelberg
INF 229
69120 Heidelberg