"no space left" but df -k shows plenty of free space

"no space left" but df -k shows plenty of free space

[D_i_F_i]

Hello,

while copying some data into a logical volume with reiserfs as
filesystem, I've got an error message, that there isn't enough
space left on the device. Even just creating a new empty file
isn't possible anymore:

sammy:/opt/kgm/data/log # touch abc
touch: creating `abc': No space left on device

But the df-command has still reported quite a lot of free capacity:

sammy:/opt/kgm/data/log # df -k /opt/kgm/data
Filesystem        1k-blocks     Used Available Use% Mounted on
/dev/kgmvg/datalv  10485436  7987364   2498072  77% /opt/kgm/data

If I had used ext2 or ext3, the number of already used inodes could
be the reason for this problem. But this isn't an issue when using
reiserfs as the following command shows:

sammy:/opt/kgm/data/log # df -i /opt/kgm/data
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/kgmvg/datalv         -       -       -    -  /opt/kgm/data

Even "du -hc /opt/kgm/data/" reports an amount of data, that should
be easily stored into a filesystem with a capacity of 10G:

411M    /opt/kgm/data/ABC/in/new
1.1G    /opt/kgm/data/ABC/in
512     /opt/kgm/data/ABC/log
512     /opt/kgm/data/ABC/out
1.1G    /opt/kgm/data/ABC
129k    /opt/kgm/data/tool
512     /opt/kgm/data/etc
3.9G    /opt/kgm/data/log
2.0M    /opt/kgm/data/tmp
2.3G    /opt/kgm/data/data
25M     /opt/kgm/data/files
1.1M    /opt/kgm/data/names
136M    /opt/kgm/data/kerna
137M    /opt/kgm/data/kernb
180M    /opt/kgm/data/logtool
233k    /opt/kgm/data/buf
57M     /opt/kgm/data/patch13/unpack
75M     /opt/kgm/data/patch13
50M     /opt/kgm/data/patch14/unpack
66M     /opt/kgm/data/patch14
7.9G    /opt/kgm/data
7.9G    total

Only after moving some of the files into another filesystem, I was
able to store further files into this filesystem. So I guessed,
that it should work if I extend the reiserfs by 2G (lvextend and
resize_reiserfs). And as a matter of fact this enlargement worked.

Now, df -k /opt/kfm/data shows:

Filesystem        1k-blocks     Used Available Use% Mounted on
/dev/kgmvg/datalv  12582524  7987360   4595164  64% /opt/kgm/data

Now we can use the filesystem without problems but I fear that we
will run into problems when the filesystem reaches 77% usage again.

The directory with the biggest number of files currently contains
about 70.000 files. I don't think that this is a real challange for
reiserfs. Also the other limits as stated in the FAQ on
http://www.namesys.com/ aren't reached by now.

I'm using reiserfs version 3.6.25 [built into kernel] of a
SLES7-Distribution (SuSE) on IBM zSeries (z900, z/VM) in
64-Bit-Mode (s390x). The filesystem has been resized
3 times on the fly (in mounted state). Could it be, that this
caused the filesystem to get damaged somehow? Or is there a bug
in the df-command, that may cause it to report wrong values for
free capacities? Can it be, that the last enlargement of
the filesystem didn't do its full job so that the problem maybe
disapeared with the next enlargement? Or is it necessary to delete
and recreate the whole filesystem (which would cause us a lot of
trouble)? Or what do you suggest?

Lots of questions. I would appreciate if you could give me some
anwers for at least some of them.

Regards,
Dirk

Re: "no space left" but df -k shows plenty of free space

[Oleg Drokin]

Hello!

On Sat, May 17, 2003 at 05:34:09PM +0200, D_i_F_i@gmx.de wrote:
> while copying some data into a logical volume with reiserfs as
> filesystem, I've got an error message, that there isn't enough
> space left on the device. Even just creating a new empty file
> isn't possible anymore:
> sammy:/opt/kgm/data/log # touch abc
> touch: creating `abc': No space left on device
> But the df-command has still reported quite a lot of free capacity:
> sammy:/opt/kgm/data/log # df -k /opt/kgm/data
> Filesystem        1k-blocks     Used Available Use% Mounted on
> /dev/kgmvg/datalv  10485436  7987364   2498072  77% /opt/kgm/data

Hm, this is pretty strange.

> Only after moving some of the files into another filesystem, I was
> able to store further files into this filesystem. So I guessed,
> that it should work if I extend the reiserfs by 2G (lvextend and
> resize_reiserfs). And as a matter of fact this enlargement worked.
> Now, df -k /opt/kfm/data shows:
> Filesystem        1k-blocks     Used Available Use% Mounted on
> /dev/kgmvg/datalv  12582524  7987360   4595164  64% /opt/kgm/data
> Now we can use the filesystem without problems but I fear that we
> will run into problems when the filesystem reaches 77% usage again.

Hm, it will be interesting if you try.
May be it will stop working when there is 2498072 blocks left again.

> The directory with the biggest number of files currently contains
> about 70.000 files. I don't think that this is a real challange for

This is unlikely to be a problem indeed.

> I'm using reiserfs version 3.6.25 [built into kernel] of a
> SLES7-Distribution (SuSE) on IBM zSeries (z900, z/VM) in
> 64-Bit-Mode (s390x). The filesystem has been resized

Hm, what kernel version do they ship?

> 3 times on the fly (in mounted state). Could it be, that this
> caused the filesystem to get damaged somehow? Or is there a bug

No, this is unlikely.

> in the df-command, that may cause it to report wrong values for
> free capacities? Can it be, that the last enlargement of
> the filesystem didn't do its full job so that the problem maybe
> disapeared with the next enlargement? Or is it necessary to delete
> and recreate the whole filesystem (which would cause us a lot of
> trouble)? Or what do you suggest?

My current idea that it might have the "blocks reserved for use by fsck"
checked incorrectly.

Chris: Where can I look at the patches that came with
SLES7-Distribution's kernel?

Bye,
    Oleg

Re: "no space left" but df -k shows plenty of free space

[Chris Mason]

On Mon, 2003-05-19 at 12:02, Oleg Drokin wrote:

> My current idea that it might have the "blocks reserved for use by fsck"
> checked incorrectly.

Sorry for the delay, still digging through the email backlog from last
week.  I don't know of any known bugs here, but I'll check.  Exactly
which kernel rpm do you have installed? 
> 
> Chris: Where can I look at the patches that came with
> SLES7-Distribution's kernel?

Well, it depends ;-)  I'll hunt the rpms down.

-chris

Re: "no space left" but df -k shows plenty of free space

[Chris Mason]

On Mon, 2003-05-19 at 12:02, Oleg Drokin wrote:
> Hello!
> 
> On Sat, May 17, 2003 at 05:34:09PM +0200, D_i_F_i@gmx.de wrote:
> > while copying some data into a logical volume with reiserfs as
> > filesystem, I've got an error message, that there isn't enough
> > space left on the device. Even just creating a new empty file
> > isn't possible anymore:
> > sammy:/opt/kgm/data/log # touch abc
> > touch: creating `abc': No space left on device
> > But the df-command has still reported quite a lot of free capacity:
> > sammy:/opt/kgm/data/log # df -k /opt/kgm/data
> > Filesystem        1k-blocks     Used Available Use% Mounted on
> > /dev/kgmvg/datalv  10485436  7987364   2498072  77% /opt/kgm/data
> 
> Hm, this is pretty strange.

Could you please run debugreiserfs -p /dev/xxx | gzip -c >
df-error-fs.gz, and make it available somewhere for us to download.  If
you are using reiserfsprogs 3.x.0j or so the output can contain actual
file data, so you probably want to put the image onto a password
protected page.

This will take a metadata dump of the filesystem, I'd like to check it.

-chris

Re: "no space left" but df -k shows plenty of free space

[D_i_F_i]

Hello,

many thanks for your quick responses! Sorry that I couldn앖 use just the
reply-function of my eMail-Client, but I앐 currently on a business
travel and I don앖 have full access to my eMails.

Here are some of the informations you asked for:

>> Now we can use the filesystem without problems but I fear that we
>> will run into problems when the filesystem reaches 77% usage again.
>
>Hm, it will be interesting if you try.
>May be it will stop working when there is 2498072 blocks left again.

Sorry, but I can앖 fill up the filesystem until it압 full because we
need the filesystem for our daily work. If I realy should try this, I 
have to wait until the next weekend. Also I have to ask several collegues,
if it압 ok if I앐 going to do this test at the next weekend. But if you
think, this test will bring us a step further, I will try to get an "OK"
for this test.


>> I'm using reiserfs version 3.6.25 [built into kernel] of a
>> SLES7-Distribution (SuSE) on IBM zSeries (z900, z/VM) in
>> 64-Bit-Mode (s390x). The filesystem has been resized
>
>Hm, what kernel version do they ship?
>

We use "k_deflt-2.4.17-45", as "rpm -qi" shows:

Name        : k_deflt                      Relocations: (not relocateable)
Version     : 2.4.17                            Vendor: SuSE AG, Nuernberg,
Germany
Release     : 45                            Build Date: Thu 05 Dec 2002
09:59:14 PM CET
Install date: Fri 14 Feb 2003 02:13:21 PM CET      Build Host:
s390zvi72.suse.de
Group       : Base/Kernel                   Source RPM:
k_deflt-2.4.17-45.src.rpm
Size        : 9938987                          License: GPL + IBM OCO
LICENSE
Packager    : feedback@suse.de
Summary     : The standard SuSE kernel
Description :
The standard SuSE kernel

SuSE series: images


> Could you please run debugreiserfs -p /dev/xxx | gzip -c >
> df-error-fs.gz, and make it available somewhere for us to download.  If
> you are using reiserfsprogs 3.x.0j or so the output can contain actual
> file data, so you probably want to put the image onto a password
> protected page.
>
> This will take a metadata dump of the filesystem, I'd like to check it.

I guess, I can execute this command while the filesystem is mounted and
in use. Is this correct? Also I았e to check first, if there will be any
"confidential" data in the output of the debug-command, which I앐 not
allowed to distribute. And as the command causes some system load, I may 
have to wait until the next weekend, to generate this data.
 
By the way, We are using the following RPM:

Name        : reiserfs                     Relocations: (not relocateable)
Version     : 3.x.1b                            Vendor: SuSE AG, Nuernberg,
Germany
Release     : 16                            Build Date: Sat 20 Apr 2002
01:23:07 AM CEST
Install date: Wed 12 Feb 2003 04:47:49 PM CET      Build Host:
s390z06.suse.de
Group       : System/Filesystems            Source RPM:
reiserfs-3.x.1b-16.src.rpm
Size        : 1571177                          License: Hans Reiser
Packager    : feedback@suse.de
Summary     : reiserfs utilities
Description :
reiserfs utilities: making the file system (mkreiserfs), checking for
consistency (reiserfsck), and resizing (resize_reiserfs)

SuSE series: a



After I았e got clearance for all these questions I would send you an
approriate message.

Regards,
Dirk

-- 
+++ GMX - Mail, Messaging & more  http://www.gmx.net +++
Bitte l錄heln! Fotogalerie online mit GMX ohne eigene Homepage!

Re: "no space left" but df -k shows plenty of free space

[Oleg Drokin]

Hello!

On Tue, May 20, 2003 at 12:24:42AM +0200, D_i_F_i@gmx.de wrote:
> >> Now we can use the filesystem without problems but I fear that we
> >> will run into problems when the filesystem reaches 77% usage again.
> >Hm, it will be interesting if you try.
> >May be it will stop working when there is 2498072 blocks left again.
> Sorry, but I can?t fill up the filesystem until it?s full because we
> need the filesystem for our daily work. If I realy should try this, I 
> have to wait until the next weekend. Also I have to ask several collegues,
> if it?s ok if I?m going to do this test at the next weekend. But if you
> think, this test will bring us a step further, I will try to get an "OK"
> for this test.

Please try.
Alternatively if you can create another reiserfs filesystem that would not be in use
by anyone and try to fill it and see how soon it will report "out of space",
that should be sufficient too.

> > Could you please run debugreiserfs -p /dev/xxx | gzip -c >
> > df-error-fs.gz, and make it available somewhere for us to download.  If
> > you are using reiserfsprogs 3.x.0j or so the output can contain actual
> > file data, so you probably want to put the image onto a password
> > protected page.
> > This will take a metadata dump of the filesystem, I'd like to check it.
> I guess, I can execute this command while the filesystem is mounted and
> in use. Is this correct? Also I?ve to check first, if there will be any

Yes, you can.

> "confidential" data in the output of the debug-command, which I?m not
> allowed to distribute. And as the command causes some system load, I may 
> have to wait until the next weekend, to generate this data.

This is ok.

> After I?ve got clearance for all these questions I would send you an
> approriate message.

Ok, thank you.

Bye,
    Oleg