ext4 no space left

ext4 no space left

[Victor Pelt]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

Hey,

i tried to switch my root partition to ext4 (from reiserfs) but i ran
into problems when copying over all the data that it says 'no space
left on the device'
i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)

i deleted quite a bit and then tried copying again, this time it still
gave the same error but more files (data wise didn't count if the
files were bigger) were copied. currently df shows

rootfs                 5044188   4310972    476980  91% /  <= file
system that is 'full'
/dev/root              5044188   4310972    476980  91% /
rc-svcdir                 1024       132       892  13% /lib/rc/init.d
udev                     10240        68     10172   1% /dev
cachedir                  4096         4      4092   1% /lib/splash/cache
none                    776940         4    776936   1% /dev/shm
/dev/hda3             24416008  22546040   1869968  93% /opt
/dev/hda6             11560764   9300396   1673112  85% /home
tmpfs                   776940      1728    775212   1% /tmp
tmpfs                  1331200        36   1331164   1% /var/tmp
tmpfs                 11560764   9300396   1673112  85% /usr/portage/distfiles

i like to use that last 470MB if possible

i also inclunded my tune2fs info

fsck didn't help (tried that)

[-- Attachment #2: tune2fs.dump --]
[-- Type: application/octet-stream, Size: 1749 bytes --]

tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          bd8da577-7b58-45bc-9c94-8254f718e55a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              320640
Block count:              1281183
Reserved block count:     64059
Free blocks:              183304
Free inodes:              59590
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      312
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8016
Inode blocks per group:   501
Flex block group size:    16
Filesystem created:       Tue Jan 20 15:29:58 2009
Last mount time:          Wed Jan 21 15:19:41 2009
Last write time:          Wed Jan 21 15:19:41 2009
Mount count:              7
Maximum mount count:      30
Last checked:             Wed Jan 21 00:49:50 2009
Check interval:           15552000 (6 months)
Next check after:         Mon Jul 20 01:49:50 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      04df98fa-5851-4f24-b232-c8b4738aef0f
Journal backup:           inode blocks

Re: ext4 no space left

[Theodore Tso]

On Wed, Jan 21, 2009 at 03:19:28PM +0100, Victor Pelt wrote:
> Hey,
> 
> i tried to switch my root partition to ext4 (from reiserfs) but i ran
> into problems when copying over all the data that it says 'no space
> left on the device'
> i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)
> 
> rootfs                 5044188   4310972    476980  91% /  <= file
> system that is 'full'
>
> i like to use that last 470MB if possible
> 
> fsck didn't help (tried that)

If this is the problem I think it is, it's fixed in patches that were
merged for 2.6.29 (in -rc1), and you can work around it via umounting
and remounting the filesystem (or in case of the root filesystem,
rebooting).

However, when you said "fsck didn't help", what exactly did you mean
by that?  That fsck didn't show any problems, when you ran it while
the filesystem was mounted?  Did you actually reboot and force an
fsck, and then found that you were still getting ENOSPC errors?

I assume you have tried writing as root, and that quotas are disabled?

  	     	  		   	 - Ted

P.S.  If it is the problem I think it is, upgrading 2.6.29-rcX should
fix the problem, as would using the ext4-stable branch in the ext4 git
tree, or applying the patches in the for-stable branch of the ext4 git
tree:

    http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git

Re: ext4 no space left

[Victor Pelt]

i set the number mounted counter to 31, which forced fsck to run on
the next reboot. I rebooted, fsck didn't show any erros, but i still
got the no space left error when i tried to copy files to the
partition.

i'll try the latest git kernel though.

On Wed, Jan 21, 2009 at 3:55 PM, Theodore Tso <tytso@mit.edu> wrote:
> On Wed, Jan 21, 2009 at 03:19:28PM +0100, Victor Pelt wrote:
>> Hey,
>>
>> i tried to switch my root partition to ext4 (from reiserfs) but i ran
>> into problems when copying over all the data that it says 'no space
>> left on the device'
>> i'm running 2.6.28-gentoo-r1 (based on 2.6.28.1 if i'm not mistaken)
>>
>> rootfs                 5044188   4310972    476980  91% /  <= file
>> system that is 'full'
>>
>> i like to use that last 470MB if possible
>>
>> fsck didn't help (tried that)
>
> If this is the problem I think it is, it's fixed in patches that were
> merged for 2.6.29 (in -rc1), and you can work around it via umounting
> and remounting the filesystem (or in case of the root filesystem,
> rebooting).
>
> However, when you said "fsck didn't help", what exactly did you mean
> by that?  That fsck didn't show any problems, when you ran it while
> the filesystem was mounted?  Did you actually reboot and force an
> fsck, and then found that you were still getting ENOSPC errors?
>
> I assume you have tried writing as root, and that quotas are disabled?
>
>                                         - Ted
>
> P.S.  If it is the problem I think it is, upgrading 2.6.29-rcX should
> fix the problem, as would using the ext4-stable branch in the ext4 git
> tree, or applying the patches in the for-stable branch of the ext4 git
> tree:
>
>    http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git
>

Re: ext4 no space left

[Theodore Tso]

On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
> i set the number mounted counter to 31, which forced fsck to run on
> the next reboot. I rebooted, fsck didn't show any erros, but i still
> got the no space left error when i tried to copy files to the
> partition.

OK, if a reboot didn't help, I'll bet I know what happened.  E2fsprogs
changed the default default inode ratio, which means that number of
inodes being created is half what it was previously.  If you do a "df
-i", you'll probably see that you have exhausted the number of inodes
in the filesystem.

The default inode ratio controlled by /etc/mke2fs.conf, and is
currently 16k.  That is, it assumes the average size of files on the
filesystem is at least 16k.  (It previously was 8k.)  For large
filesystems, this is not a problem; I'm guessing that you have a small
root filesystem, and probably are using a hard-coded /dev partition,
so the large number of (zero-length) device files is throwing off the
average.  If you recreate the filesystem with mke2fs -I 8192, it
should allow you to copy over all of your files in the root
filesystem.

Finally, note that we made this change for all ext2/3/4 filesystems,
so this is not unique to ext4; it's just that you reformatted your
root filesystem for the first time since upgrading to a newer
e2fsprogs with the changed default, and you ran into this problem.  If
there are enough people who are using small root filesystems, maybe
we'll need to have some adjusted hueristics.  Right now we have
"floppy" for filesystems less than 3 megs, "small" for filesystems
less than 512 megs, and every thing else is default.  Maybe we need to
have a "medium" for filesystems smaller than 10 gigs, perhaps, and use
a default inode ratio of 8192 for medium-sized filesystems....

						- Ted

Re: ext4 no space left

[Victor Pelt]

[-- Attachment #1: Type: text/plain, Size: 2441 bytes --]

upgrading to 2.6.29-rc2-git1 did allow me to use most of the space but
i still can't quite use all of it. maybe this is to be expected.
(note: i'm just trying testing now, everything i used to have on my
old partition fits. i'm just trying to understand what exactly is
'full'. it seems the file system is 'holding something back'

i included a dumo of df, df -i and tune2fs -l /dev/hda2
(and partition that has ext4 is the / partition or /dev/hda2)

victor


On Wed, Jan 21, 2009 at 6:04 PM, Theodore Tso <tytso@mit.edu> wrote:
> On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
>> i set the number mounted counter to 31, which forced fsck to run on
>> the next reboot. I rebooted, fsck didn't show any erros, but i still
>> got the no space left error when i tried to copy files to the
>> partition.
>
> OK, if a reboot didn't help, I'll bet I know what happened.  E2fsprogs
> changed the default default inode ratio, which means that number of
> inodes being created is half what it was previously.  If you do a "df
> -i", you'll probably see that you have exhausted the number of inodes
> in the filesystem.
>
> The default inode ratio controlled by /etc/mke2fs.conf, and is
> currently 16k.  That is, it assumes the average size of files on the
> filesystem is at least 16k.  (It previously was 8k.)  For large
> filesystems, this is not a problem; I'm guessing that you have a small
> root filesystem, and probably are using a hard-coded /dev partition,
> so the large number of (zero-length) device files is throwing off the
> average.  If you recreate the filesystem with mke2fs -I 8192, it
> should allow you to copy over all of your files in the root
> filesystem.
>
> Finally, note that we made this change for all ext2/3/4 filesystems,
> so this is not unique to ext4; it's just that you reformatted your
> root filesystem for the first time since upgrading to a newer
> e2fsprogs with the changed default, and you ran into this problem.  If
> there are enough people who are using small root filesystems, maybe
> we'll need to have some adjusted hueristics.  Right now we have
> "floppy" for filesystems less than 3 megs, "small" for filesystems
> less than 512 megs, and every thing else is default.  Maybe we need to
> have a "medium" for filesystems smaller than 10 gigs, perhaps, and use
> a default inode ratio of 8192 for medium-sized filesystems....
>
>                                                - Ted
>

[-- Attachment #2: dump --]
[-- Type: application/octet-stream, Size: 3288 bytes --]

df

Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 5044188   4725008     62944  99% /
/dev/root              5044188   4725008     62944  99% /
rc-svcdir                 1024       120       904  12% /lib/rc/init.d
udev                     10240        68     10172   1% /dev
cachedir                  4096         4      4092   1% /lib/splash/cache
none                    776804         0    776804   0% /dev/shm
/dev/hda3             24416008  22546040   1869968  93% /opt
/dev/hda6             11560764   9970208   1003300  91% /home
tmpfs                   776804         0    776804   0% /tmp
tmpfs                  1331200         0   1331200   0% /var/tmp
tmpfs                   776804         0    776804   0% /usr/portage/distfiles

df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
rootfs                320640  261978   58662   82% /
/dev/root             320640  261978   58662   82% /
rc-svcdir             194201      94  194107    1% /lib/rc/init.d
udev                  194201    1427  192774    1% /dev
cachedir              194201       2  194199    1% /lib/splash/cache
none                  194201       1  194200    1% /dev/shm
/dev/hda3            2066576   54161 2012415    3% /opt
/dev/hda6            1468800  559088  909712   39% /home
tmpfs                 194201       4  194197    1% /tmp
tmpfs                 194201       1  194200    1% /var/tmp
tmpfs                 194201       1  194200    1% /usr/portage/distfiles

tune2fs -l /dev/hda2

tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          bd8da577-7b58-45bc-9c94-8254f718e55a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              320640
Block count:              1281183
Reserved block count:     64059
Free blocks:              79795
Free inodes:              58662
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      312
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8016
Inode blocks per group:   501
Flex block group size:    16
Filesystem created:       Tue Jan 20 15:29:58 2009
Last mount time:          Wed Jan 21 19:35:45 2009
Last write time:          Wed Jan 21 19:35:45 2009
Mount count:              10
Maximum mount count:      30
Last checked:             Wed Jan 21 00:49:50 2009
Check interval:           15552000 (6 months)
Next check after:         Mon Jul 20 01:49:50 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      04df98fa-5851-4f24-b232-c8b4738aef0f
Journal backup:           inode blocks

Re: ext4 no space left

[Theodore Tso]

On Wed, Jan 21, 2009 at 07:15:26PM +0100, Victor Pelt wrote:
> upgrading to 2.6.29-rc2-git1 did allow me to use most of the space but
> i still can't quite use all of it. maybe this is to be expected.

No, it's not to be expected.  You should be able to use everything up
to 100% of what is reported by df.  There is the 5% reserved blocks
which only root is allowed to use, but df takes that into account.  So
if you can't use that last 62944k reported by df after a reboot, and
df -i reports enough inodes, that's definitely a bug in ext4.

Would you be willing to send me a compressed raw e2image file?  You
can generate it as follows:

    e2image -r /dev/hda2 - | bzip2 > /tmp/hda2.e2i.bz2

Ideally this should be done while the filesystem is unmounted, or
mounted read-only. 

It will give me a copy of the filesystem metadata, but no data blocks.
That will allow me to try to replicate the problem on my system.  It
will probably be still too large e-mail, so if you can put it up on
some web or ftp server, and send me the URL, that would be best.

Thanks!!

							- Ted

Re: ext4 no space left

[Pavel Machek]

On Wed 2009-01-21 12:04:23, Theodore Tso wrote:
> On Wed, Jan 21, 2009 at 04:18:35PM +0100, Victor Pelt wrote:
> > i set the number mounted counter to 31, which forced fsck to run on
> > the next reboot. I rebooted, fsck didn't show any erros, but i still
> > got the no space left error when i tried to copy files to the
> > partition.
> 
> OK, if a reboot didn't help, I'll bet I know what happened.  E2fsprogs
> changed the default default inode ratio, which means that number of
> inodes being created is half what it was previously.  If you do a "df
> -i", you'll probably see that you have exhausted the number of inodes
> in the filesystem.

Actually it would be cool if df warned about low inode free space by
default. "df -i" is an old trap, but still effective to these days
:-).
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html