ext2_unlink fun

ext2_unlink fun

[Mohammad A. Haque]

My machine locked hard last night for an unknown reason under
2.4.3-pre4. Rebooted and it did it's fsck thing. Got alot of errors
about missing '..', fixed alot of things and moved some stuff to
/lost+found.

Some files got screwed up so I can't delete them.

[mhaque@viper html-blah]$ rm -r mac3dfx/
rm: cannot remove `mac3dfx/news/1999/08/199908231702.txt': Input/output
error
rm: cannot remove `mac3dfx/news/1999/08/199908231802.txt': Input/output
error
rm: cannot remove directory `mac3dfx/news/1999/08': Directory not empty
rm: cannot remove directory `mac3dfx/news/1999': Directory not empty
rm: cannot remove directory `mac3dfx/news': Directory not empty
rm: cannot remove `mac3dfx/dat': Input/output error
rm: cannot remove directory `mac3dfx': Directory not empty

My filesystem is probably really screwed up so I'm going to format. But
for future reference, how would one get rid of those files? Or does this
indicate that it's time to reformat and not even consider trying to
delete those files?

-- 

=====================================================================
Mohammad A. Haque                              http://www.haque.net/
                                               mhaque@haque.net

  "Alcohol and calculus don't mix.             Project Lead
   Don't drink and derive." --Unknown          http://wm.themes.org/
                                               batmanppc@themes.org
=====================================================================

Re: ext2_unlink fun

[Andreas Dilger]

Mohammad A. Haque writes:
> My machine locked hard last night for an unknown reason under
> 2.4.3-pre4. Rebooted and it did it's fsck thing. Got alot of errors
> about missing '..', fixed alot of things and moved some stuff to
> /lost+found.
> 
> Some files got screwed up so I can't delete them.
> 
> [mhaque@viper html-blah]$ rm -r mac3dfx/
> rm: cannot remove `mac3dfx/news/1999/08/199908231702.txt': Input/output
> error
> rm: cannot remove `mac3dfx/news/1999/08/199908231802.txt': Input/output
> error
> rm: cannot remove directory `mac3dfx/news/1999/08': Directory not empty
> rm: cannot remove directory `mac3dfx/news/1999': Directory not empty
> rm: cannot remove directory `mac3dfx/news': Directory not empty
> rm: cannot remove `mac3dfx/dat': Input/output error
> rm: cannot remove directory `mac3dfx': Directory not empty
> 
> My filesystem is probably really screwed up so I'm going to format. But
> for future reference, how would one get rid of those files? Or does this
> indicate that it's time to reformat and not even consider trying to
> delete those files?

It would be nice to determine _why_ you can't unlink these files.  If
it was just an issue of size > 2GB, you should get EFBIG error or so.
People have been reporting undeletable files several times now...  Before
you reformat, could you do some debugging?

It appears a reasonable spot to get the EIO from is in ext2_delete_entry()
where we are validating the de in ext2_check_dir_entry().  Can you check
your syslog for any error messages, like:

	ext2_delete_entry: bad entry in directory X: 199908231702.txt ...

It would be _nice_ to be able to delete a dir entry even if it is corrupt
(considering we want to delete it, so we don't really care about that dir
entry), but that may cause further corruption of the previous dir entry
(which we DO want to keep).

The only other place I can see we return EIO is in ext2_unlink when
comparing de->inode to inode->i_ino.  However, this could only happen
if a file of the same name was created in the directory, OR the dir entry
was changed after we had done the lookup.  Strange, but unlikely here.

Obviously e2fsck doesn't fix the problem?  Could you run debugfs on this
filesystem, cd to the "mac3dfx/news/1999/08" directory, and "stat" each
of these broken files?  This may help to identify what is wrong with the
files that e2fsck isn't fixing.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: ext2_unlink fun

[Mohammad A. Haque]

On Wed, 21 Mar 2001, Andreas Dilger wrote:

> It would be nice to determine _why_ you can't unlink these files.  If
> it was just an issue of size > 2GB, you should get EFBIG error or so.
> People have been reporting undeletable files several times now...  Before
> you reformat, could you do some debugging?

If someone guides me, I can do all the debugging you want.

>
> It appears a reasonable spot to get the EIO from is in ext2_delete_entry()
> where we are validating the de in ext2_check_dir_entry().  Can you check
> your syslog for any error messages, like:
>
> 	ext2_delete_entry: bad entry in directory X: 199908231702.txt ...

Only message I have is EXT2-fs warning (device ide0(3,3)): ext2_unlink:
Deleting nonexistent file (1343506), 0


> Obviously e2fsck doesn't fix the problem?  Could you run debugfs on this
> filesystem, cd to the "mac3dfx/news/1999/08" directory, and "stat" each
> of these broken files?  This may help to identify what is wrong with the
> files that e2fsck isn't fixing.

Correct, this was after running e2fsck. I'll try running it again when I
get home. Here is debugfs stat output for one of the broken files.
Again, I havent run e2fsck a second time yet.

debugfs:  stat 199908231702.txt
Inode: 1343489   Type: bad type    Mode:  0000   Flags: 0x0
Version/Generation: 0
User:     0   Group:     0   Size: 0
File ACL: 0    Directory ACL: 0
Links: 0   Blockcount: 0
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x00000000 -- Wed Dec 31 19:00:00 1969
atime: 0x00000000 -- Wed Dec 31 19:00:00 1969
mtime: 0x00000000 -- Wed Dec 31 19:00:00 1969
BLOCKS:



I'll send the results of what I get after running a fsck again in about
6 hours.


Thanks for your help.

-- 

=====================================================================
Mohammad A. Haque                              http://www.haque.net/
                                               mhaque@haque.net

  "Alcohol and calculus don't mix.             Project Lead
   Don't drink and derive." --Unknown          http://wm.themes.org/
                                               batmanppc@themes.org
=====================================================================

Re: ext2_unlink fun

[Andreas Dilger]

Mohammad A. Haque writes:
> On Wed, 21 Mar 2001, Andreas Dilger wrote:
> > It appears a reasonable spot to get the EIO from is in ext2_delete_entry()
> > where we are validating the de in ext2_check_dir_entry().  Can you check
> > your syslog for any error messages, like:
> >
> > 	ext2_delete_entry: bad entry in directory X: 199908231702.txt ...
> 
> Only message I have is EXT2-fs warning (device ide0(3,3)): ext2_unlink:
> Deleting nonexistent file (1343506), 0

OK, this is consistent with the debugfs output - the inode was overwritten
with zeros.  However, I'm not sure how this would lead to EIO.  In my tree
(2.4.2) for ext2_unlink() we get EIO if ext2_delete_entry() has a problem
with ext2_check_dir_entry(), but that would have put another message into
the logs.  The other EIO (de->inode != inode->i_ino) happens before this
message is printed, so it isn't possible either.

> Correct, this was after running e2fsck. I'll try running it again when I
> get home. Here is debugfs stat output for one of the broken files.
> Again, I havent run e2fsck a second time yet.
> 
> debugfs:  stat 199908231702.txt
> Inode: 1343489   Type: bad type    Mode:  0000   Flags: 0x0
> Version/Generation: 0
> User:     0   Group:     0   Size: 0
> File ACL: 0    Directory ACL: 0
> Links: 0   Blockcount: 0
> Fragment:  Address: 0    Number: 0    Size: 0
> ctime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> atime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> mtime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> BLOCKS:

Maybe you really _are_ having I/O errors?  That would explain the zero'd
inode table and the I/O error messages.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: ext2_unlink fun

[Mohammad A. Haque]

On Wed, 21 Mar 2001, Andreas Dilger wrote:

> > debugfs:  stat 199908231702.txt
> > Inode: 1343489   Type: bad type    Mode:  0000   Flags: 0x0
> > Version/Generation: 0
> > User:     0   Group:     0   Size: 0
> > File ACL: 0    Directory ACL: 0
> > Links: 0   Blockcount: 0
> > Fragment:  Address: 0    Number: 0    Size: 0
> > ctime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> > atime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> > mtime: 0x00000000 -- Wed Dec 31 19:00:00 1969
> > BLOCKS:
>
> Maybe you really _are_ having I/O errors?  That would explain the zero'd
> inode table and the I/O error messages.

*shrug* Maybe. This is beyond the scope of what I know but I'm trying to
learn.

If no one else wants anything, I'll be reformatting that partition
tonight after seeing what fsck does.

-- 

=====================================================================
Mohammad A. Haque                              http://www.haque.net/
                                               mhaque@haque.net

  "Alcohol and calculus don't mix.             Project Lead
   Don't drink and derive." --Unknown          http://wm.themes.org/
                                               batmanppc@themes.org
=====================================================================

Re: ext2_unlink fun

[Mohammad A. Haque]

Andreas Dilger wrote:
> > Correct, this was after running e2fsck. I'll try running it again when I
> > get home. Here is debugfs stat output for one of the broken files.
> > Again, I havent run e2fsck a second time yet.

Ok, I ran e2fsck twice last night. It didnt do anything the first run
but then second time around it found those files and fixed the problem.
Weird.

-- 

=====================================================================
Mohammad A. Haque                              http://www.haque.net/ 
                                               mhaque@haque.net

  "Alcohol and calculus don't mix.             Project Lead
   Don't drink and derive." --Unknown          http://wm.themes.org/
                                               batmanppc@themes.org
=====================================================================

Re: ext2_unlink fun

[Horst von Brand]

"Mohammad A. Haque" <mhaque@haque.net> said:
> My machine locked hard last night for an unknown reason under
> 2.4.3-pre4. Rebooted and it did it's fsck thing. Got alot of errors
> about missing '..', fixed alot of things and moved some stuff to
> /lost+found.
> 
> Some files got screwed up so I can't delete them.

Usually wierd permissions, in particular, spurious immutable flags. Has
happened to me when bugs in the kernel wrote garbage over inodes. Try
lsattr(1), chattr(1). Or debugfs(8) or such.
-- 
Dr. Horst H. von Brand                       mailto:vonbrand@inf.utfsm.cl
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513