ARC-1260: No space left on device, when there is (or should be) free space left

ARC-1260: No space left on device, when there is (or should be) free space left

[Magnus Naeslund]

It seems I have a problem accessing the whole device, it filled up too quickly.
I thought I was avoiding the 2tb limit when not using any partition tables, shame on me :)

# df -B 1 /dev/sdb
Filesystem           1B-blocks      Used Available Use% Mounted on
/dev/sdb             5999417057280 1908287229952 4091129827328  32% /storage/disk1

(Ok here I have 68% free space on our 5.5TB device)

# echo zzz > /storage/disk1/dummy
/storage/disk1/dummy: No space left on device

(But here it seems to think the device is full)

It's an amd64 based system running 2.6.21.5.
Please tell me if you need more information!

Regards,
Magnus

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Robert P. J. Day]

On Mon, 18 Jun 2007, Magnus Naeslund wrote:

> It seems I have a problem accessing the whole device, it filled up too quickly.
> I thought I was avoiding the 2tb limit when not using any partition tables, shame on me :)
>
> # df -B 1 /dev/sdb
> Filesystem           1B-blocks      Used Available Use% Mounted on
> /dev/sdb             5999417057280 1908287229952 4091129827328  32% /storage/disk1
>
> (Ok here I have 68% free space on our 5.5TB device)

$ df -i

rday
-- 
========================================================================
Robert P. J. Day
Linux Consulting, Training and Annoying Kernel Pedantry
Waterloo, Ontario, CANADA

http://fsdev.net/wiki/index.php?title=Main_Page
========================================================================

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Magnus Naeslund]

Robert P. J. Day wrote:
>>
>> (Ok here I have 68% free space on our 5.5TB device)
> 
> $ df -i
> 

Duh, this was it!
What is the recommended method on how to increase the inode count on the filesystem (there is no need for online resize in this case)?
I have the resize_inode option set when looking at the tune2fs -l output.

Magnus

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Theodore Tso]

On Mon, Jun 18, 2007 at 02:20:15PM +0200, Magnus Naeslund wrote:
> Duh, this was it!
> What is the recommended method on how to increase the inode count on the filesystem (there is no need for online resize in this case)?
> I have the resize_inode option set when looking at the tune2fs -l output.

Unfortunately, at the moment there isn't a way to increase the inode
count without resizing the filesystem (which will add more blocks and
inodes as you add more blockgroups).  That's because the number of
inodes per block group is fixed at mke2fs time.   

In theory it should be possible to whip up a program which would
implement this, but it would be a non-trivial exercise.  So at the
moment, the only solution is to recreate it with a larger number of
inodes, or resize the filesystem.

How are you using the filesystem?  This wouldn't happen to be one of
the backup schemes that use hard links and huge numbers of
directories, would it?  And how did you create the filesystem
originally?  Normally mke2fs is quite generous with the number of
inodes it creates to avoid this problem.  Did you use -T largefile or
-T largefile4 by any chance?  Or did you manually specify a
non-standard inode_ratio size?

						- Ted

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Magnus Naeslund]

Theodore Tso wrote:
> 
> How are you using the filesystem?  This wouldn't happen to be one of
> the backup schemes that use hard links and huge numbers of
> directories, would it?  And how did you create the filesystem
> originally?  Normally mke2fs is quite generous with the number of
> inodes it creates to avoid this problem.  Did you use -T largefile or
> -T largefile4 by any chance?  Or did you manually specify a
> non-standard inode_ratio size?
> 

It's a filesystem containing images varying from 300kb to 2mb of size.
It currently contains 1426394 files in 4125 directories.

Magnus

I believe this has the appropriate information:
tune2fs 1.40-WIP (14-Nov-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed directory hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1430528
Block count:              1464843264
Reserved block count:     0
Free blocks:              998810993
Free inodes:              0
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      674
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32
Inode blocks per group:   1
Filesystem created:       Sun Apr  1 12:46:39 2007
Last mount time:          Mon Jun 18 14:23:26 2007
Last write time:          Mon Jun 18 14:23:26 2007
Mount count:              8
Maximum mount count:      20
Last checked:             Sun Apr  1 12:46:39 2007
Check interval:           15552000 (6 months)
Next check after:         Fri Sep 28 12:46:39 2007
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      f252b473-6db0-499b-8de1-c788e84563dc
Journal backup:           inode blocks

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Jason L Tibbitts III]

>>>>> "MN" == Magnus Naeslund <mag@kite.se> writes:

MN> Inode count: 1430528

That's stunningly small; I created an 8TB partition here and made an
ext3 filesystem in it with the default parameters; I got nearly 1000
times as many inodes as you have:

Inode count:              1072414720
Block count:              2144799744

 - J<

Re: ARC-1260: No space left on device, when there is (or should be) free space left

[Theodore Tso]

On Mon, Jun 18, 2007 at 06:28:08PM +0200, Magnus Naeslund wrote:
> 
> It's a filesystem containing images varying from 300kb to 2mb of size.
> It currently contains 1426394 files in 4125 directories.
> 
> Magnus
> 
> Inode count:              1430528
> Block count:              1464843264
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         32

Well, *thar's* your problem!  :-)

I'm guessing you created the filesystem with mke2fs -T largefile4.
This basically told mke2fs that you anticipated the filesystem to have
an average inode size of 4 megabytes.  Given that your filesystem has
files ranging from 300k and 2mb, that's why you're running into
problems.

						- Ted