What will happen when disk(ext3) is full while i continue to operate files ?

What will happen when disk(ext3) is full while i continue to operate files ?

[Seaman Hu]

I am sorry if you receive it twice. since the
mail-list said that i can't send mail to it unless i
become one its member.

hi,
   My system will crash when the disk(ext3) is full
while i continue to launch 50 proceses to operate
files(such as create, rm, mv, ...). Does ext3 have
such a capability to stop journaling the changes when
it finds there is no space left? or which source file
of ext3 do i need to check this function?

   My System info:
    Redhat 7.3    Linux Kernel: 2.4.18-17
    / : ext3 1.6G   /boot : ext3 60M

   I am sorry that I will attach the oops next time
since i am still restoring my poor system.  :(

   Any opinions will be highly appriciated. Thanks in
advance. :)

Seaman

__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Seaman Hu]

Sorry. I probably didn't make it clear. 
My system is ok when first msg "EXT3-fs error (device
sd(8,2)) in ext3_new_inode: error 28" appears.
However, it will crash after millions of the same msg.
Is there some kind of buffer full to cause the crash?

thanks in advance.

Seaman


--- Seaman Hu <seaman_hu@yahoo.com> wrote:
> I am sorry if you receive it twice. since the
> mail-list said that i can't send mail to it unless i
> become one its member.
> 
> hi,
>    My system will crash when the disk(ext3) is full
> while i continue to launch 50 proceses to operate
> files(such as create, rm, mv, ...). Does ext3 have
> such a capability to stop journaling the changes
> when
> it finds there is no space left? or which source
> file
> of ext3 do i need to check this function?
> 
>    My System info:
>     Redhat 7.3    Linux Kernel: 2.4.18-17
>     / : ext3 1.6G   /boot : ext3 60M
> 
>    I am sorry that I will attach the oops next time
> since i am still restoring my poor system.  :(
> 
>    Any opinions will be highly appriciated. Thanks
> in
> advance. :)
> 
> Seaman
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Finance - Get real-time stock quotes
> http://finance.yahoo.com
> 
> 
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users@redhat.com
>
https://listman.redhat.com/mailman/listinfo/ext3-users


__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Stephen C. Tweedie]

Hi,

On Fri, Sep 20, 2002 at 02:11:14AM -0700, Seaman Hu wrote:
> Sorry. I probably didn't make it clear. 
> My system is ok when first msg "EXT3-fs error (device
> sd(8,2)) in ext3_new_inode: error 28" appears.
> However, it will crash after millions of the same msg.
> Is there some kind of buffer full to cause the crash?

Ah, that's a known problem when you run out of inodes.  Ext3
incorrectly treated it as a full fs error.  That's been fixed in -ac,
ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's
post-2.4.19 tree.

Cheers,
 Stephen

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Duncan Sands]

> Ah, that's a known problem when you run out of inodes.  Ext3
> incorrectly treated it as a full fs error.  That's been fixed in -ac,
> ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's
> post-2.4.19 tree.

Do you know a good way to recover from this when it happens?
The problem being that when you reboot it immediately happens
again...  I managed to recover from this but it required some time
and tricks (see other email to list) - perhaps you know an easy way?

All the best,

Duncan.

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Stephen C. Tweedie]

Hi,

On Fri, Sep 20, 2002 at 11:27:28AM +0200, Duncan Sands wrote:
> > Ah, that's a known problem when you run out of inodes.  Ext3
> > incorrectly treated it as a full fs error.  That's been fixed in -ac,
> > ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's
> > post-2.4.19 tree.
> 
> Do you know a good way to recover from this when it happens?
> The problem being that when you reboot it immediately happens
> again...  I managed to recover from this but it required some time
> and tricks (see other email to list) - perhaps you know an easy way?

Mounting with "errors=continue" will make the kernel ignore the error
when it occurs.

Cheers,
 Stephen

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Duncan Sands]

On Friday 20 September 2002 11:11, Seaman Hu wrote:
> Sorry. I probably didn't make it clear.
> My system is ok when first msg "EXT3-fs error (device
> sd(8,2)) in ext3_new_inode: error 28" appears.
> However, it will crash after millions of the same msg.
> Is there some kind of buffer full to cause the crash?

It looks like you have run out of inodes (df -i will tell you how
many inodes you have left).  When you format your disk as
ext2 or ext3, you specify the number of inodes to be allocated
on the disk.  This number does not change later.  The inodes
are used for storing file information.  It is possible to run out of
inodes while still having plenty of space free on the disk.  The
ext3 code in 2.4 up to and including 2.4.19 contains a bug: it
fails nastily if you run out of inodes (it fails gracefully if you run
out of free space).  This has been fixed in the latest 2.5 kernels
and in the -ac kernels I think.

The problem is that it is quite tricky to recover from this.  What
you need to do is delete files on the disk in order to have some
free inodes.  Then you can apply a kernel patch to fix the bug
(Andrew Morton sent me this patch against 2.5.20:

--- linux-2.5.20/fs/ext3/ialloc.c       Wed May 29 12:48:15 2002
+++ 25/fs/ext3/ialloc.c Mon Jun  3 16:05:36 2002
@@ -534,7 +534,8 @@ repeat:
 fail:
        unlock_super(sb);
        iput(inode);
-       ext3_std_error(sb, err);
+       if (err != -ENOSPC)
+               ext3_std_error(sb, err);
        return ERR_PTR(err);
 }

Thanks Andrew!)

Now, how to recover?  When this happened to me I did something
along these lines (I don't remember too well):
(1) force an fsck on reboot
(2) reboot quickly after the fsck and before any files get created (otherwise
you are back where you started)
(3) mount the disk as ext2 via the rootfstype=ext2 boot option
(4) delete lots of files.

Probably there is a better way...

I hope this helps,

Duncan.

Re: What will happen when disk(ext3) is full while i continue to operate files ?

[Stephen C. Tweedie]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

Hi,

On Fri, Sep 20, 2002 at 11:25:09AM +0200, Duncan Sands wrote:

> The problem is that it is quite tricky to recover from this.

Actually, mounting with "errors=continue" should let the filesystem
ignore the failure.

> What
> you need to do is delete files on the disk in order to have some
> free inodes.  Then you can apply a kernel patch to fix the bug
> (Andrew Morton sent me this patch against 2.5.20:

The official patch in 2.4 is attached.

Cheers,
 Stephen

[-- Attachment #2: 0827-inode-enospc.patch --]
[-- Type: text/plain, Size: 797 bytes --]

---------------------
PatchSet 827
Date: 2002/04/10 18:02:19
Author: sct
Log:
Don't consider ENOSPC as a fatal error when allocating an inode.  Otherwise
running out of inodes marks the fs as having an error, potentially taking
the kernel down if we are in panic-on-error fs mode.

Members: 
	fs/ext3/ialloc.c:1.19.4.4->1.19.4.5 [ext3-1_0-branch]

--- linux-ext3-2.4merge/fs/ext3/ialloc.c.=K0001=.orig	Sat Aug 17 20:09:51 2002
+++ linux-ext3-2.4merge/fs/ext3/ialloc.c	Mon Aug 19 18:48:50 2002
@@ -392,7 +392,7 @@
 
 	err = -ENOSPC;
 	if (!gdp)
-		goto fail;
+		goto out;
 
 	err = -EIO;
 	bitmap_nr = load_inode_bitmap (sb, i);
@@ -523,9 +523,10 @@
 	return inode;
 
 fail:
+	ext3_std_error(sb, err);
+out:
 	unlock_super(sb);
 	iput(inode);
-	ext3_std_error(sb, err);
 	return ERR_PTR(err);
 }