Wrong calculation of space remaining on a 32 bit system.

Wrong calculation of space remaining on a 32 bit system.

[François Valenduc]

Hello everybody,

I try to use ext4 but I encounter an annoying problem. It seems that
only one third of the filesystem can be used. I have a LVM volume for
the /usr/src directory which currently is formatted as reiserfs. So I
made a compressed tar file of the content. When I uncompress it after
having formatted this volume as ext4, the decompression stops when
appreoximately 600 Mb of space is used. At this time, it's no more
possible to write anything on this volume even if there are 1.4 Gb of
free space remaining.
What seems even stranger is that I encounter this problem only on a 32
bits system. On another computer having an Intel Core 2 Duo and running
a 64 system, the problem doesn't occur.
I have already filed a bug report (see
http://bugzilla.kernel.org/show_bug.cgi?id=11840) but unfortunately,
nobody seems to care about it.

I tried the 2.6.28-rc3 or 2.6.27 patched with the ext4-2 patchset but it
 gives the same problem.

Can anybody explains what's happening ?

Thanks in advance for your help.

François Valenduc
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[Theodore Tso]

On Sun, Nov 02, 2008 at 03:40:25PM +0100, François Valenduc wrote:
> I have already filed a bug report (see
> http://bugzilla.kernel.org/show_bug.cgi?id=11840) but unfortunately,
> nobody seems to care about it.

Hi,

	Sorry no one responded, but while the Bugzilla system is great
for tracking bugs, it's not a good place to call attention to a bug.
In fact, I'm not sure that fs_ext4@kernel-bugs.osdl.org forwards
anywhere sane (I'm checking up on that, though).  In general it's a
good idea by sending e-mail to either linux-kernel@vger.kernel.org or
(preferred) linux-ext4@vger.kernel.org.

	From your description, the ext4 filesystem thought there was
some kind of on-disk corruption or other problem with your filesystem,
and remounted it read-only.  That's consistent with your report of the
messages in the syslog of:

ext4_da_writepages: jbd2_start: 1024 pages, ino 40898; err -30

There should have been something in your logs *before* all of the err
-30 complaints, however.  Was this your root filesystem, or do you
have this saved in logfile in /var/log?  If so, can you search back
for something that occurs before all of the ext4_da_writepages error
messages?

							- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[news.gmane.org]

Theodore Tso a écrit :
> On Sun, Nov 02, 2008 at 03:40:25PM +0100, François Valenduc wrote:
>> I have already filed a bug report (see
>> http://bugzilla.kernel.org/show_bug.cgi?id=11840) but unfortunately,
>> nobody seems to care about it.
> 
> Hi,
> 
> 	Sorry no one responded, but while the Bugzilla system is great
> for tracking bugs, it's not a good place to call attention to a bug.
> In fact, I'm not sure that fs_ext4@kernel-bugs.osdl.org forwards
> anywhere sane (I'm checking up on that, though).  In general it's a
> good idea by sending e-mail to either linux-kernel@vger.kernel.org or
> (preferred) linux-ext4@vger.kernel.org.
> 
> 	From your description, the ext4 filesystem thought there was
> some kind of on-disk corruption or other problem with your filesystem,
> and remounted it read-only.  That's consistent with your report of the
> messages in the syslog of:
> 
> ext4_da_writepages: jbd2_start: 1024 pages, ino 40898; err -30
> 
> There should have been something in your logs *before* all of the err
> -30 complaints, however.  Was this your root filesystem, or do you
> have this saved in logfile in /var/log?  If so, can you search back
> for something that occurs before all of the ext4_da_writepages error
> messages?
> 
> 							- Ted

Thanks for your reply.
In fact, the errors I mentionned don't occur anymore. But the problem of
  the diskspace remains. I never managed until now to use more than one
third of a partition on a 32 bit system. After that, I always get a
message saying that the partition is full, which can't be true.

François
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[Graham Murray]

"news.gmane.org" <francois.valenduc@tvcablenet.be> writes:

> In fact, the errors I mentionned don't occur anymore. But the problem of
>   the diskspace remains. I never managed until now to use more than one
> third of a partition on a 32 bit system. After that, I always get a
> message saying that the partition is full, which can't be true.

Just an idea, which I am sure you have already considered, but have you
run out of inodes? 

Re: Wrong calculation of space remaining on a 32 bit system.

[François Valenduc]

Graham Murray a écrit :
> "news.gmane.org" <francois.valenduc@tvcablenet.be> writes:
> 
>> In fact, the errors I mentionned don't occur anymore. But the problem of
>>   the diskspace remains. I never managed until now to use more than one
>> third of a partition on a 32 bit system. After that, I always get a
>> message saying that the partition is full, which can't be true.
> 
> Just an idea, which I am sure you have already considered, but have you
> run out of inodes? 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

How can I know if I ran out of inodes ? I already tried 128 and 256
inode sizes but the problem occurs in both cases. As I said in my bug
report, I found a patch dated from november 2007 which seems to adress
the problem (see
http://osdir.com/ml/file-systems.ext4/2007-11/msg00200.htm). Off course,
I can't apply it any more now. But it seems this kind of problem still
exists. I have no problem to use ext4 on a 64 bit systems with logical
volumes having approximately the same size.

François

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[Theodore Tso]

On Sun, Nov 02, 2008 at 11:24:31PM +0100, François Valenduc wrote:
> 
> How can I know if I ran out of inodes ? I already tried 128 and 256
> inode sizes but the problem occurs in both cases. 

By using the command "df -i".  It will list the number of inodes that
are in use.

The tuning parameters for mke2fs is -i (the inode ratio), or -N
(number of inodes).  The number of inodes is normally calculated as a
ratio to the disk size.  The normal inode ratio is 16384, which allows
for an average inode size of 16k.  If you have a huge number of small
files, or small directories, or a huge number of symbolic links or
device nodes in the filesystem, you can run out of inodes.  You can
change this either by specifying a a smaller inode ratio, or by
explicitly specifying the number of inodes you want created using the
-N option.

> As I said in my bug
> report, I found a patch dated from november 2007 which seems to adress
> the problem (see
> http://osdir.com/ml/file-systems.ext4/2007-11/msg00200.htm).  Off course,
> I can't apply it any more now.

You can't apply it because it's already been applied; it's been in
mainline for quite a while.

>  But it seems this kind of problem still
> exists. I have no problem to use ext4 on a 64 bit systems with logical
> volumes having approximately the same size.

That seems very strange.  Maybe you have a different version of
mke2fs, or a different mke2fs.conf file?  (The default inode size can
be controlled by the mke2fs.conf file.)

Can you send the output of "dumpe2fs -h" on a filesystem where you are
having this problem, and on a filesystem from the 64-bit system where
it is working correctly?

					- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[François Valenduc]

[-- Attachment #1: Type: text/plain, Size: 2922 bytes --]

Theodore Tso a écrit :
> On Sun, Nov 02, 2008 at 11:24:31PM +0100, François Valenduc wrote:
>> How can I know if I ran out of inodes ? I already tried 128 and 256
>> inode sizes but the problem occurs in both cases. 
> 
> By using the command "df -i".  It will list the number of inodes that
> are in use.
> 
> The tuning parameters for mke2fs is -i (the inode ratio), or -N
> (number of inodes).  The number of inodes is normally calculated as a
> ratio to the disk size.  The normal inode ratio is 16384, which allows
> for an average inode size of 16k.  If you have a huge number of small
> files, or small directories, or a huge number of symbolic links or
> device nodes in the filesystem, you can run out of inodes.  You can
> change this either by specifying a a smaller inode ratio, or by
> explicitly specifying the number of inodes you want created using the
> -N option.
> 
>> As I said in my bug
>> report, I found a patch dated from november 2007 which seems to adress
>> the problem (see
>> http://osdir.com/ml/file-systems.ext4/2007-11/msg00200.htm).  Off course,
>> I can't apply it any more now.
> 
> You can't apply it because it's already been applied; it's been in
> mainline for quite a while.
> 
>>  But it seems this kind of problem still
>> exists. I have no problem to use ext4 on a 64 bit systems with logical
>> volumes having approximately the same size.
> 
> That seems very strange.  Maybe you have a different version of
> mke2fs, or a different mke2fs.conf file?  (The default inode size can
> be controlled by the mke2fs.conf file.)
> 
> Can you send the output of "dumpe2fs -h" on a filesystem where you are
> having this problem, and on a filesystem from the 64-bit system where
> it is working correctly?
> 
> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

I indeed ran out of inode and this probably has nothing to do with 32 or
64 bit systems. The df -i output looks like this:
/dev/mapper/system-portage      45K     45K       0  100% /usr/portage

As you may have guessed, I created a logical volume for the portage tree
 of gentoo. It indeeds contains a lot of small files with all the
ebuilds. I use XFS for the portage tree on the 64 bits computer. Maybe
ext4 is not appropriate for this case. Maybe I would have the same
problem also on the 64 bits computer.

I am also attaching the dump2efs output. I finally managed to use ext4
on this computer. I created a logical volume for the /usr/src directory
and I managed to copy 2 sources of linux kernel and a clone of    the
git tree on it. The df -i output is the following:
/dev/mapper/system-src   120K     96K     25K   80% /usr/src

I checked the mke2fs.conf files on both computer and the default inode
ratio is set to 16384 on the 2 systems.

François


[-- Attachment #2: dump-prob --]
[-- Type: text/plain, Size: 1737 bytes --]

Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          f0c32b27-a50f-40ea-999f-bea17ac620d9
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              45120
Block count:              180224
Reserved block count:     9011
Free blocks:              173110
Free inodes:              45109
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      43
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7520
Inode blocks per group:   470
Flex block group size:    16
Filesystem created:       Mon Nov  3 00:35:25 2008
Last mount time:          Mon Nov  3 00:35:31 2008
Last write time:          Mon Nov  3 00:35:31 2008
Mount count:              1
Maximum mount count:      27
Last checked:             Mon Nov  3 00:35:25 2008
Check interval:           15552000 (6 months)
Next check after:         Sat May  2 01:35:25 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      c29d076b-7903-4f63-bc67-fcf72f753cf7
Journal backup:           inode blocks
Journal size:             16M

Re: Wrong calculation of space remaining on a 32 bit system.

[Theodore Tso]

On Mon, Nov 03, 2008 at 07:28:34PM +0100, François Valenduc wrote:
> 
> As you may have guessed, I created a logical volume for the portage tree
>  of gentoo. It indeeds contains a lot of small files with all the
> ebuilds. I use XFS for the portage tree on the 64 bits computer. Maybe
> ext4 is not appropriate for this case. Maybe I would have the same
> problem also on the 64 bits computer.

There's nothing wrong with using ext4 for this case; it's just that
you do need to specify the appropriate average inode size.  An average
inode size of 16k works fine for most applications, but not for this
one.  You could calculate an appropriate inode ratio by figuring the
average size of the files used on your system, which I would guess is
somewhere between 4k and 8k, and then this wouldn't be a problem.

I don't know a whole lot about Gentoo's ebuilds, and why it creates so
many small files but if it is common, we could create an entry in
/etc/mke2fs.conf, i.e., like this:

[fs_types]
	portage = {
		inode_ratio = 8192
	}

Now if you run "mke2fs -t portage", it would automatically create a
filesystem with an inode ratio of 8k.  

As another example, suppose a spy agency needed to create filesystem
where they stored the results of "pen registers", which record the
time, date, source and destination of phone calls[1].  And suppose
they decide to store each intercept as a separate file, and each file
on average is less than 1k.  (No kidding, I was once at a "next
generation filesystem" requirements session at Sandia National Labs
some 9 years ago where they asked for something very similar to this
--- the ability to write huge numbers of itsy-bitsy files very
quickly, in contrast to the a-bomb simulation folks, who wanted the
ability to write a few really big files very quickly.)  Then they
might want to do something like this:

[fs_types]
	nsa = {
	    inode_ratio = 1024
	    blocksize = 1024
	}

(Alternatively, they might want to consider storing their intercepts
in something *other* than a filesystem, like say a database instead.  :-)

[1] http://www.capsteps.com/sounds/nsa-glory.mp3

						- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[François Valenduc]

Theodore Tso a écrit :
> On Mon, Nov 03, 2008 at 07:28:34PM +0100, François Valenduc wrote:
>   
>> As you may have guessed, I created a logical volume for the portage tree
>>  of gentoo. It indeeds contains a lot of small files with all the
>> ebuilds. I use XFS for the portage tree on the 64 bits computer. Maybe
>> ext4 is not appropriate for this case. Maybe I would have the same
>> problem also on the 64 bits computer.
>>     
>
> There's nothing wrong with using ext4 for this case; it's just that
> you do need to specify the appropriate average inode size.  An average
> inode size of 16k works fine for most applications, but not for this
> one.  You could calculate an appropriate inode ratio by figuring the
> average size of the files used on your system, which I would guess is
> somewhere between 4k and 8k, and then this wouldn't be a problem.
>
> I don't know a whole lot about Gentoo's ebuilds, and why it creates so
> many small files but if it is common, we could create an entry in
> /etc/mke2fs.conf, i.e., like this:
>
> [fs_types]
> 	portage = {
> 		inode_ratio = 8192
> 	}
>
> Now if you run "mke2fs -t portage", it would automatically create a
> filesystem with an inode ratio of 8k.  
>
> As another example, suppose a spy agency needed to create filesystem
> where they stored the results of "pen registers", which record the
> time, date, source and destination of phone calls[1].  And suppose
> they decide to store each intercept as a separate file, and each file
> on average is less than 1k.  (No kidding, I was once at a "next
> generation filesystem" requirements session at Sandia National Labs
> some 9 years ago where they asked for something very similar to this
> --- the ability to write huge numbers of itsy-bitsy files very
> quickly, in contrast to the a-bomb simulation folks, who wanted the
> ability to write a few really big files very quickly.)  Then they
> might want to do something like this:
>
> [fs_types]
> 	nsa = {
> 	    inode_ratio = 1024
> 	    blocksize = 1024
> 	}
>
> (Alternatively, they might want to consider storing their intercepts
> in something *other* than a filesystem, like say a database instead.  :-)
>
> [1] http://www.capsteps.com/sounds/nsa-glory.mp3
>
> 						- Ted
>
>   
Thanks for all your explanations. I finally found that 4096 seems to be
the more suitable ratio for the gentoo portage tree. Decreasing the
inode ratio to 8192 is not enough to avoid the problem. In fact the tree
consists of all the ebuilds which are all small text files located in a
lot of subdirectories. I have no idea how to compute the exact average
size but the most common size seems to be between 4 and 6 k. Then there
are all the source files which by default are located in the 
/usr/portage/distfiles directory. These can be very large (around 300 Mb
for openoffice).
So, as you suggested I add the following section in mke2fs.conf:

   portage = {
                features =
has_journal,extents,flex_bg,uninit_bg,dir_nlink,extra_isize
                inode_ratio=4096
        }

So running mke2fs -T portage creates indeeed an ext4 filesystem with the
appropriate inode ratio for the gentoo portage tree, without huge_file
support which is certainly not needed for this case.

François
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Wrong calculation of space remaining on a 32 bit system.

[Theodore Tso]

On Mon, Nov 03, 2008 at 10:23:32PM +0100, François Valenduc wrote:
> So, as you suggested I add the following section in mke2fs.conf:
> 
>    portage = {
>                 features =
> has_journal,extents,flex_bg,uninit_bg,dir_nlink,extra_isize
>                 inode_ratio=4096
>         }

You can also just do

    portage = {
    	    inode_ratio=4096
    }

... and then "mke2fs -t ext4 -T portage" or "/sbin/mkfs.ext4 -T
portage" will do the right thing.  Note BTW, that this issue with the
inode_ratio is also true for ext3, so what I've suggested means that
"mke2fs -t ext3 -T portage" or "mke2fs -t ext2 -T portage" will also
do the right thing for an ext3 or ext2 filesystem, respectively.

					- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html