ext3 file system went read-only in 2.6.18.2 (plus hacks)

ext3 file system went read-only in 2.6.18.2 (plus hacks)

[Ben Greear]

First, this kernel is tainted by one of my proprietary patches
and also has my other non-proprietary patches applied, so please
ignore as you wish.  My patches do not (purposefully) affect the file-system.

The system is a via/c3 system running a fairly stripped down FC5
installation on a 1GB CF disk.  We've been using this platform
for over a year and this is the first file system error we've
seen.  It could easily be that the CF is funky, but I thought
I'd post the message in case it proves to be something else.


eth1: Promiscuous mode enabled.
eth1: Promiscuous mode enabled.
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99247: rec_len % 4 1
Aborting journal on device hda1.
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99254: rec_len % 4 1
ext3_abort called.
EXT3-fs error (device hda1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99048: rec_len % 4 1
EXT3-fs error (device hda1): ext3_find_entry: bad entry in directory #99041: directory en1
   syslogd: /var/log/messages: Read-only file system
   syslogd: /var/log/secure: Read-only file system
   syslogd: /var/log/maillog: Read-only file system
   syslogd: /var/log/cron: Read-only file system
   syslogd: /var/log/spooler: Read-only file system
   syslogd: /var/log/boot.log: Read-only file system
   syslogd: /var/log/messages: Read-only file system
   syslogd: /var/log/secure: Read-only file system
   syslogd: /var/log/maillog: Read-only file system
   syslogd: /var/log/cron: Read-only file system
   syslogd: /var/log/spooler: Read-only file system
   syslogd: /var/log/boot.log: Read-only file system
pktgen: pktgen_mark_device marking eth0#5 for removal
pktgen: pktgen_mark_device marking eth0#0 for removal
.....

After restarting and a manual fsck, the system appears to
be back to normal.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: ext3 file system went read-only in 2.6.18.2 (plus hacks)

[Alan]

> pktgen: pktgen_mark_device marking eth0#5 for removal
> pktgen: pktgen_mark_device marking eth0#0 for removal
> .....
> 
> After restarting and a manual fsck, the system appears to
> be back to normal.

Did the fsck find any errors ?

Re: ext3 file system went read-only in 2.6.18.2 (plus hacks)

[Ben Greear]

Alan wrote:
>> pktgen: pktgen_mark_device marking eth0#5 for removal
>> pktgen: pktgen_mark_device marking eth0#0 for removal
>> .....
>>
>> After restarting and a manual fsck, the system appears to
>> be back to normal.
> 
> Did the fsck find any errors ?

Yes.  They are below:

Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/hda1
CT_ROOT contains a file system with errors, check forced.
CT_ROOT: Directory inode 99041, block 0, offset 0: directory corrupted


CT_ROOT: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
         (i.e., without -a or -p options)
[FAILED]

*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type Control-D to continue):
(Repair filesystem) 1 # fsck.ext3
anaconda-ks.cfg     .gconf/             .serverauth.7145
.bash_history       .gconfd/            .ssh/
.bash_logout        GNUstep/            .tcshrc
.bash_profile       .gtkrc              .vnc/
.bashrc             install.log.syslog  .Xauthority
.cshrc              .lesshst            .xauths0kLo8
.ethereal/          .serverauth.1869    .xinitrc
(Repair filesystem) 1 # fsck.ext3 -a -y /dev/hda1
e2fsck: Only one the options -p/-a, -n or -y may be specified.
(Repair filesystem) 2 # fsck.ext3 -y /dev/hda1
e2fsck 1.38 (30-Jun-2005)
CT_ROOT contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Directory inode 99041, block 0, offset 0: directory corrupted
Salvage? yes

First entry 'M-?' (inode=124909) in directory inode 99041 (???) should be '.'
Fix? yes

Setting filetype for entry '.' in ??? (99041) to 2.
Missing '..' in directory inode 99041.
Fix? yes

Setting filetype for entry '..' in ??? (99041) to 2.
Directory inode 99048, block 0, offset 0: directory corrupted
Salvage? yes

First entry 'M-?' (inode=129789) in directory inode 99048 (???) should be '.'
Fix? yes

Setting filetype for entry '.' in ??? (99048) to 2.
Missing '..' in directory inode 99048.
Fix? yes

Setting filetype for entry '..' in ??? (99048) to 2.
Directory inode 99247, block 0, offset 0: directory corrupted
Salvage? yes

First entry 'M-?' (inode=102335) in directory inode 99247 (???) should be '.'
Fix? yes

Setting filetype for entry '.' in ??? (99247) to 2.
Missing '..' in directory inode 99247.
Fix? yes

Setting filetype for entry '..' in ??? (99247) to 2.
Directory inode 99254, block 0, offset 0: directory corrupted
Salvage? yes

First entry 'M-?' (inode=112567) in directory inode 99254 (???) should be '.'
Fix? yes

Setting filetype for entry '.' in ??? (99254) to 2.
Missing '..' in directory inode 99254.
Fix? yes

Setting filetype for entry '..' in ??? (99254) to 2.
Pass 3: Checking directory connectivity
'..' in /lib/modules/2.6.18.2c3/kernel/drivers/mtd/maps (99041) is <The NULL inode> (0), .Fix? yes

'..' in /lib/modules/2.6.18.2c3/kernel/drivers/mtd/chips (99048) is <The NULL inode> (0),.Fix? yes

'..' in /home/lanforge/DB/day_352 (99247) is <The NULL inode> (0), should be /home/lanfor.Fix? yes

'..' in /home/lanforge/DB/day_353 (99254) is <The NULL inode> (0), should be /home/lanfor.Fix? yes

Pass 4: Checking reference counts
Inode 2 ref count is 19, should be 23.  Fix? yes

Inode 63506 ref count is 1, should be 2.  Fix? yes

Inode 95836 ref count is 58, should be 56.  Fix? yes

Unattached inode 99004
Connect to /lost+found? yes

Inode 99004 ref count is 2, should be 1.  Fix? yes

Unattached inode 99005
Connect to /lost+found? yes

Inode 99005 ref count is 2, should be 1.  Fix? yes

Unattached inode 99006
Connect to /lost+found? yes

Inode 99006 ref count is 2, should be 1.  Fix? yes

Unattached inode 99007
Connect to /lost+found? yes

Inode 99007 ref count is 2, should be 1.  Fix? yes

Unattached inode 99008
Connect to /lost+found? yes

Inode 99008 ref count is 2, should be 1.  Fix? yes

Unattached inode 99009
Connect to /lost+found? yes

Inode 99009 ref count is 2, should be 1.  Fix? yes

Unattached inode 99010
Connect to /lost+found? yes

Inode 99010 ref count is 2, should be 1.  Fix? yes

Unattached inode 99011
Connect to /lost+found? yes

Inode 99011 ref count is 2, should be 1.  Fix? yes

Unattached inode 99012
Connect to /lost+found? yes

Inode 99012 ref count is 2, should be 1.  Fix? yes

Unattached inode 99013
Connect to /lost+found? yes

Inode 99013 ref count is 2, should be 1.  Fix? yes

Unattached inode 99014
Connect to /lost+found? yes

Inode 99014 ref count is 2, should be 1.  Fix? yes

Unattached inode 99016
Connect to /lost+found? yes

Inode 99016 ref count is 2, should be 1.  Fix? yes

Unattached inode 99017
Connect to /lost+found? yes

Inode 99017 ref count is 2, should be 1.  Fix? yes

Unattached inode 99019
Connect to /lost+found? yes

Inode 99019 ref count is 2, should be 1.  Fix? yes

Inode 99020 ref count is 8, should be 6.  Fix? yes

Unattached inode 99021
Connect to /lost+found? yes

Inode 99021 ref count is 2, should be 1.  Fix? yes

Unattached inode 99022
Connect to /lost+found? yes

Inode 99022 ref count is 2, should be 1.  Fix? yes

Unattached inode 99023
Connect to /lost+found? yes

Inode 99023 ref count is 2, should be 1.  Fix? yes

Unattached inode 99248
Connect to /lost+found? yes

Inode 99248 ref count is 2, should be 1.  Fix? yes

Unattached inode 99249
Connect to /lost+found? yes

Inode 99249 ref count is 2, should be 1.  Fix? yes

Unattached inode 99250
Connect to /lost+found? yes

Inode 99250 ref count is 2, should be 1.  Fix? yes

Unattached inode 99251
Connect to /lost+found? yes

Inode 99251 ref count is 2, should be 1.  Fix? yes

Unattached inode 99252
Connect to /lost+found? yes

Inode 99252 ref count is 2, should be 1.  Fix? yes

Unattached inode 99253
Connect to /lost+found? yes

Inode 99253 ref count is 2, should be 1.  Fix? yes

Unattached inode 99255
Connect to /lost+found? yes

Inode 99255 ref count is 2, should be 1.  Fix? yes

Unattached inode 99256
Connect to /lost+found? yes

Inode 99256 ref count is 2, should be 1.  Fix? yes

Unattached inode 99257
Connect to /lost+found? yes

Inode 99257 ref count is 2, should be 1.  Fix? yes

Unattached inode 99258
Connect to /lost+found? yes

Inode 99258 ref count is 2, should be 1.  Fix? yes

Unattached inode 99259
Connect to /lost+found? yes

Inode 99259 ref count is 2, should be 1.  Fix? yes

Unattached inode 99260
Connect to /lost+found? yes

Inode 99260 ref count is 2, should be 1.  Fix? yes

Pass 5: Checking group summary information
Block bitmap differences:  +67611 -(79872--79873) -(80952--81007) -(81016--81023) -(810271Fix? yes


CT_ROOT: ***** FILE SYSTEM WAS MODIFIED *****
CT_ROOT: ***** REBOOT LINUX *****
CT_ROOT: 38495/253184 files (3.5% non-contiguous), 227976/253015 blocks
(Repair filesystem) 3 # reboot


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: ext3 file system went read-only in 2.6.18.2 (plus hacks)

[Theodore Tso]

Well, the filesystem obviously got corrupted.  The only question is
*why* it got corrupted.  It could be memory corruption, or a bug in
your propietary kernel patches, or some kind of hardware issue with
the CF device.  There's really no way to say for sure.

Were there any error messages in the system log from the device
driver?

						- Ted

Re: ext3 file system went read-only in 2.6.18.2 (plus hacks)

[Ben Greear]

Theodore Tso wrote:
> Well, the filesystem obviously got corrupted.  The only question is
> *why* it got corrupted.  It could be memory corruption, or a bug in
> your propietary kernel patches, or some kind of hardware issue with
> the CF device.  There's really no way to say for sure.

Ok.  I certainly can't guarantee my code is not somehow causing
the problem.

> Were there any error messages in the system log from the device
> driver?

I looked and did not see anything obvious.  The file system
was not overly full (49MB free, not counting the reserved
space for the root user.)

I appreciate you looking at the report.  If we see it again or
manage to find some way to reliably reproduce this on other
hardware, I will of course let you know.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com