Linux 2.4.30-rc3 md/ext3 problems

[Ville Herva <vherva@vianova.fi>]

On Mon, Mar 28, 2005 at 10:34:05AM +0300, [Ville Herva] wrote:
> 
> I just upgraded from linux-2.4.21 + vserser 0.17 to 2.4.30rc3 + vserver
> 1.2.10. The box has been running stable with 2.4.21 + vserver 0.17/0.16 for
> a few years (uptime before reboot was nearly 400 days.)
> 
> The boot went fine, but after few hours I got 
> Message from syslogd@box at Sun Mar 27 22:07:00 2005 ...
> kernel: journal commit I/O error
> 
> and dmesg is filled with 
> --8<-----------------------------------------------------------------------
> EXT3-fs error (device md(9,3)) in start_transaction: Journal has aborted
> EXT3-fs error (device md(9,3)) in start_transaction: Journal has aborted
> EXT3-fs error (device md(9,3)) in start_transaction: Journal has aborted
> EXT3-fs error (device md(9,3)) in start_transaction: Journal has aborted
> --8<-----------------------------------------------------------------------
> 
> This is roofs, on top software raid1 and two ide disks. mdstat claims it's
> healthy:
> 
> --8<-----------------------------------------------------------------------
> md3 : active raid1 hdc3[1] hda3[0]
>       37955648 blocks [2/2] [UU]
> --8<-----------------------------------------------------------------------
> 
> While dmesg has filled up and /var/log/messages is read-only - I can't see
> all the kernel messages - there appears to be no IO errors from the
> underlying devices (md, ide). smartctl -a does not report errors for hda nor
> hdc.
> 
> During reboot, fsck was run for md3, and it was clean. Now I get
> 
> --8<-----------------------------------------------------------------------
> Block bitmap differences:  -(7800660--7801060) -(7801934--7802030) -(7802370--7802602) -(7802604--7802613) -(7802681--7802700) -(7802715--7802716) -(7802726--7802732) -(7802744--7802750)-(7802914--7802927) -(7802934--7802937) -(7802946--7802964)  -(7803392--7803417) -(7805060--7808825) -(7808976--7809608) 
> Fix? no
> 
> Inode bitmap differences:  -3899400
> Fix? no
> --8<-----------------------------------------------------------------------
> 
> No errors from the badblocks part of the fsck, though.
> 
> Running fsck triggers the "journal commit I/O error" messages again, and
> still no IO errors from either md or ide.
> 
> This _could_ have something to do with the vserver patch but it doesn't
> appear so. Also, it doesn't immediately look like hardware problem. 

I rebooted (fsck took the fs errors away, no big offenders), and after a few
minutes, I got the same error ("journal commit I/O error"). So it doesn't
appear all that random memory corruption. The error happened right when I
logged out, but that might have been a coincidence. No ide nor md errors
this time either. 

I don't know what to suspect. What I gather from changelogs, there haven't
been any critical looking ext3 changes in 2.4 lately, but then again,
vserver doesn't mess with block layer / ext3 journalling either.

Any ideas?


-- v -- 

v@iki.fi