Issue with "no space left on device"

Issue with "no space left on device"

[Sander van Schie / True]

Hello,

I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.

Mount options:

# mount | grep vdc1
/dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)

Used disk space:

# df /dev/vdc1
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vdc1       36681200 16066576  20614624  44% /mnt

Used inodes:

# df -i /dev/vdc1
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/vdc1      917440 199616 717824   22% /mnt

xfs info:

# xfs_info /dev/vdc1
meta-data=/dev/vdc1              isize=2048   agcount=4, agsize=2293695 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1 spinodes=0
data     =                       bsize=4096   blocks=9174779, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=4479, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


I've already tried repairing and rebooting, which unfortunately didn't help.

Any idea what's wrong?

Thanks!

- Sander

Re: Issue with "no space left on device"

[Brian Foster]

On Thu, Aug 17, 2017 at 02:54:48PM +0000, Sander van Schie / True wrote:
> Hello,
> 
> I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.
> 
> Mount options:
> 
> # mount | grep vdc1
> /dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
> 
> Used disk space:
> 
> # df /dev/vdc1
> Filesystem     1K-blocks     Used Available Use% Mounted on
> /dev/vdc1       36681200 16066576  20614624  44% /mnt
> 
> Used inodes:
> 
> # df -i /dev/vdc1
> Filesystem     Inodes  IUsed  IFree IUse% Mounted on
> /dev/vdc1      917440 199616 717824   22% /mnt
> 
> xfs info:
> 
> # xfs_info /dev/vdc1
> meta-data=/dev/vdc1              isize=2048   agcount=4, agsize=2293695 blks

With 2k sized inodes, the most likely cause is free space fragmentation.
What does 'xfs_db -c "freesp -s" <dev>' print for this fs?

Brian

>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=1        finobt=1 spinodes=0
> data     =                       bsize=4096   blocks=9174779, imaxpct=5
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
> log      =internal               bsize=4096   blocks=4479, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> 
> I've already tried repairing and rebooting, which unfortunately didn't help.
> 
> Any idea what's wrong?
> 
> Thanks!
> 
> - Sander--
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issue with "no space left on device"

[Sander van Schie]

On 17-08-17 18:47, Brian Foster wrote:
> On Thu, Aug 17, 2017 at 02:54:48PM +0000, Sander van Schie / True wrote:
>> Hello,
>>
>> I have a 35 GB XFS partition with both plenty of free space and inodes, but I'm still getting the error "No space left on device" when trying to create new files.
>>
>> Mount options:
>>
>> # mount | grep vdc1
>> /dev/vdc1 on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
>>
>> Used disk space:
>>
>> # df /dev/vdc1
>> Filesystem     1K-blocks     Used Available Use% Mounted on
>> /dev/vdc1       36681200 16066576  20614624  44% /mnt
>>
>> Used inodes:
>>
>> # df -i /dev/vdc1
>> Filesystem     Inodes  IUsed  IFree IUse% Mounted on
>> /dev/vdc1      917440 199616 717824   22% /mnt
>>
>> xfs info:
>>
>> # xfs_info /dev/vdc1
>> meta-data=/dev/vdc1              isize=2048   agcount=4, agsize=2293695 blks
> 
> With 2k sized inodes, the most likely cause is free space fragmentation.
> What does 'xfs_db -c "freesp -s" <dev>' print for this fs?
> 
> Brian
> 

The output of the command is as follows:

# xfs_db -c "freesp -s" /dev/vdc1
    from      to extents  blocks    pct
       1       1      28      28   0,00
       2       3      50     149   0,00
       4       7     907    3799   0,07
       8      15    1369   12990   0,25
      16      31  183206 5126962  99,34
      32      63     492   17197   0,33
total free extents 186052
total free blocks 5161125
average free extent size 27,7402

(not sure what I'm looking at myself)


>>           =                       sectsz=512   attr=2, projid32bit=1
>>           =                       crc=1        finobt=1 spinodes=0
>> data     =                       bsize=4096   blocks=9174779, imaxpct=5
>>           =                       sunit=0      swidth=0 blks
>> naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
>> log      =internal               bsize=4096   blocks=4479, version=2
>>           =                       sectsz=512   sunit=0 blks, lazy-count=1
>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>
>>
>> I've already tried repairing and rebooting, which unfortunately didn't help.
>>
>> Any idea what's wrong?
>>
>> Thanks!
>>
>> - Sander--
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Issue with "no space left on device"

[Eric Sandeen]

On 8/17/17 12:15 PM, Sander van Schie wrote:
> 
> On 17-08-17 18:47, Brian Foster wrote:

...

>> With 2k sized inodes, the most likely cause is free space fragmentation.
>> What does 'xfs_db -c "freesp -s" <dev>' print for this fs?
>>
>> Brian
>>
> 
> The output of the command is as follows:
> 
> # xfs_db -c "freesp -s" /dev/vdc1
>    from      to extents  blocks    pct
>       1       1      28      28   0,00
>       2       3      50     149   0,00
>       4       7     907    3799   0,07
>       8      15    1369   12990   0,25
>      16      31  183206 5126962  99,34
>      32      63     492   17197   0,33
> total free extents 186052
> total free blocks 5161125
> average free extent size 27,7402
> 
> (not sure what I'm looking at myself)

It shows you where the freespace is in the filesystem, and how large the
various freespace extents are; i.e. you have 28 single blocks free,
and 50 free extents containing two to three free filesystem blocks,
etc.

We allocate inodes in contiguous chunks of 64; with 2k inodes and 4k blocks,
that means we allocate inodes in chunks of 32 blocks.

So you do have 492 extents with 32 free blocks or more, but allocation also
has alignment requirements - these allocated chunks must also be aligned,
in this case aligned to a 32-block boundary.  From the output above, we don't
know the alignment of that freespace

recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters,
i.e.

    xfs_db> freesp -A 32
    
    will show only 32-block aligned free extents.

You may not have that recent of xfsprogs, but you could check out the git
tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build
it, and run xfs_db from within the tree, i.e.

# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1

Can you provide that output?

-Eric

Re: Issue with "no space left on device"

[Sander van Schie]

On 17-08-17 19:45, Eric Sandeen wrote:
> On 8/17/17 12:15 PM, Sander van Schie wrote:
>>
>> On 17-08-17 18:47, Brian Foster wrote:
> 
> ...
> 
>>> With 2k sized inodes, the most likely cause is free space fragmentation.
>>> What does 'xfs_db -c "freesp -s" <dev>' print for this fs?
>>>
>>> Brian
>>>
>>
>> The output of the command is as follows:
>>
>> # xfs_db -c "freesp -s" /dev/vdc1
>>     from      to extents  blocks    pct
>>        1       1      28      28   0,00
>>        2       3      50     149   0,00
>>        4       7     907    3799   0,07
>>        8      15    1369   12990   0,25
>>       16      31  183206 5126962  99,34
>>       32      63     492   17197   0,33
>> total free extents 186052
>> total free blocks 5161125
>> average free extent size 27,7402
>>
>> (not sure what I'm looking at myself)
> 
> It shows you where the freespace is in the filesystem, and how large the
> various freespace extents are; i.e. you have 28 single blocks free,
> and 50 free extents containing two to three free filesystem blocks,
> etc.
> 
> We allocate inodes in contiguous chunks of 64; with 2k inodes and 4k blocks,
> that means we allocate inodes in chunks of 32 blocks.
> 
> So you do have 492 extents with 32 free blocks or more, but allocation also
> has alignment requirements - these allocated chunks must also be aligned,
> in this case aligned to a 32-block boundary.  From the output above, we don't
> know the alignment of that freespace
> 
> recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters,
> i.e.
> 
>      xfs_db> freesp -A 32
>      
>      will show only 32-block aligned free extents.
> 
> You may not have that recent of xfsprogs, but you could check out the git
> tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build
> it, and run xfs_db from within the tree, i.e.
> 
> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
> 
> Can you provide that output?
> 
> -Eric
> 

Thank you for the explanation!

The output of the command is as follows:

# db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
    from      to extents  blocks    pct
       1       1       1       1   0,00
       2       3       7      21   0,01
       4       7     235     970   0,24
       8      15     130    1313   0,33
      16      31   14214  397375  99,42
total free extents 14587
total free blocks 399680
average free extent size 27,3997

For what it's worth, practically all files (194024 out of 196385) on the 
filesystem are 80102 bytes in size.

- Sander

Re: Issue with "no space left on device"

[Eric Sandeen]

On 8/17/17 3:11 PM, Sander van Schie wrote:
> 
> On 17-08-17 19:45, Eric Sandeen wrote:

...

>> recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters,
>> i.e.
>>
>>      xfs_db> freesp -A 32
>>           will show only 32-block aligned free extents.
>>
>> You may not have that recent of xfsprogs, but you could check out the git
>> tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build
>> it, and run xfs_db from within the tree, i.e.
>>
>> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>>
>> Can you provide that output?
>>
>> -Eric
>>
> 
> Thank you for the explanation!
> 
> The output of the command is as follows:
> 
> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>    from      to extents  blocks    pct
>       1       1       1       1   0,00
>       2       3       7      21   0,01
>       4       7     235     970   0,24
>       8      15     130    1313   0,33
>      16      31   14214  397375  99,42

So, I think that's the problem: There are no 32-block aligned
free regions of 32 blocks length or greater.

(now that I think about it, the -A filter filters on free extents
/starting/ on that alignment; I don't know if the inode allocator
can make use of, say, a 64 block free extent which /overlaps/ an
aligned 32-block range... hm)

Out of curiosity, what was the reason for 2k inodes?

-Eric

Re: Issue with "no space left on device"

[Eric Sandeen]

On 8/17/17 4:21 PM, Eric Sandeen wrote:
> On 8/17/17 3:11 PM, Sander van Schie wrote:
>>
>> On 17-08-17 19:45, Eric Sandeen wrote:
> 
> ...
> 
>>> recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters,
>>> i.e.
>>>
>>>      xfs_db> freesp -A 32
>>>           will show only 32-block aligned free extents.
>>>
>>> You may not have that recent of xfsprogs, but you could check out the git
>>> tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build
>>> it, and run xfs_db from within the tree, i.e.
>>>
>>> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>>>
>>> Can you provide that output?
>>>
>>> -Eric
>>>
>>
>> Thank you for the explanation!
>>
>> The output of the command is as follows:
>>
>> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>>    from      to extents  blocks    pct
>>       1       1       1       1   0,00
>>       2       3       7      21   0,01
>>       4       7     235     970   0,24
>>       8      15     130    1313   0,33
>>      16      31   14214  397375  99,42
> 
> So, I think that's the problem: There are no 32-block aligned
> free regions of 32 blocks length or greater.
> 
> (now that I think about it, the -A filter filters on free extents
> /starting/ on that alignment; I don't know if the inode allocator
> can make use of, say, a 64 block free extent which /overlaps/ an
> aligned 32-block range... hm)
> 
> Out of curiosity, what was the reason for 2k inodes?

Also for what it's worth - the sparse inodes feature, which is
default on newer filesystems, alleviates this problem.  When
mounted, what does xfs_info /mount/point say, does it contain
output for "spinodes?"

If you need 2k inodes, you probaboy want to get userspace+kernel
that can support sparse inode allocation.

-Eric

Re: Issue with "no space left on device"

[Sander van Schie]

On 17-08-17 23:28, Eric Sandeen wrote:
> On 8/17/17 4:21 PM, Eric Sandeen wrote:
>> On 8/17/17 3:11 PM, Sander van Schie wrote:
>>>
>>> On 17-08-17 19:45, Eric Sandeen wrote:
>>
>> ...
>>
>>>> recent xfsprogs v4.12 has a new option to freesp, to specify alignment filters,
>>>> i.e.
>>>>
>>>>       xfs_db> freesp -A 32
>>>>            will show only 32-block aligned free extents.
>>>>
>>>> You may not have that recent of xfsprogs, but you could check out the git
>>>> tree from git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git, build
>>>> it, and run xfs_db from within the tree, i.e.
>>>>
>>>> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>>>>
>>>> Can you provide that output?
>>>>
>>>> -Eric
>>>>
>>>
>>> Thank you for the explanation!
>>>
>>> The output of the command is as follows:
>>>
>>> # db/xfs_db -c "freesp -A 32 -s" /dev/vdc1
>>>     from      to extents  blocks    pct
>>>        1       1       1       1   0,00
>>>        2       3       7      21   0,01
>>>        4       7     235     970   0,24
>>>        8      15     130    1313   0,33
>>>       16      31   14214  397375  99,42
>>
>> So, I think that's the problem: There are no 32-block aligned
>> free regions of 32 blocks length or greater.
>>
>> (now that I think about it, the -A filter filters on free extents
>> /starting/ on that alignment; I don't know if the inode allocator
>> can make use of, say, a 64 block free extent which /overlaps/ an
>> aligned 32-block range... hm)
>>
>> Out of curiosity, what was the reason for 2k inodes?

The filesystem was created by a default Ceph deployement, so no 
particular reason. An inode size of 2k seems to be the default of 
Ceph(-deploy) to prevent performance issues due to, as I understand it, 
metadata otherwise possibly not fitting in a single extent.

Is this currently primarily an issue due to the fairly small partition 
size and big inode size? Will this be less of an issue with a partition 
of let's say 1 TB?

I'll do some more research as it's not very clear for me yet (due to my 
lack of experience regarding XFS, or filesystems in general). Your 
information was very insightful though, so thank you for that!

> 
> Also for what it's worth - the sparse inodes feature, which is
> default on newer filesystems, alleviates this problem.  When
> mounted, what does xfs_info /mount/point say, does it contain
> output for "spinodes?"

Currently it's set to 0: spinodes=0

> 
> If you need 2k inodes, you probaboy want to get userspace+kernel
> that can support sparse inode allocati
I will look into this

> 
> -Eric
>