PROBLEM:

["Michael T. Babcock" <mbabcock@fibrespeed.net>]

Ext3 Journal oops & RAID-1 set losing sync.

(Sent to both EXT3 and Linux-RAID since both are in use and seem 
possibly relevant)

I have had a number of problems maintaining a software RAID-1 set on an 
IDE box I maintain; it seems that doing raidhotadd on the drive marked 
as invalid works each time though.  However, I've had both errors about 
trying to read past the end of the disk/partition and at least one oops 
relating to the ext3 journalling code.  I'm not sure either is related 
to the other, but not sure they're not either.  Any help / suggestions 
would be appreciated as this is a live system that is now being rebooted 
quite regularly ...

Output of ver_linux:
Linux {hostname} 2.4.18-vpn1.0 #5 Fri Jul 26 09:59:37 EDT 2002 i686 unknown
 
Gnu C                  2.96
Gnu make               3.79.1
util-linux             2.11n
mount                  2.11n
modutils               2.4.14
e2fsprogs              1.27
PPP                    2.4.1
isdn4k-utils           3.1pre1
Linux C Library        2.2.5
Dynamic linker (ldd)   2.2.5
Procps                 2.0.7
Net-tools              1.60
Console-tools          0.3.3
Sh-utils               2.0.11
Modules Loaded         ipsec ppp_synctty ppp_async ppp_generic slhc 
8139too via-rhine binfmt_misc epca autofs mii ipt_REJECT ipt_LOG 
ipt_state ipt_MARK iptable_mangle ipt_MASQUERADE iptable_nat 
iptable_filter usb-uhci usbcore ext3 jbd

Recent log entries (not including oops):
Sep 25 10:46:51 gw kernel: journal_bmap_Rc2009e7d: journal block not 
found at offset 7185 on md (9,3)
Sep 25 10:46:51 gw kernel: Aborting journal on device md(9,3).
Sep 25 10:46:51 gw kernel: attempt to access beyond end of device
Sep 25 10:46:51 gw kernel: 09:03: rw=1, want=1625616132, limit=4192192
Sep 25 10:46:51 gw kernel: attempt to access beyond end of device
Sep 25 10:46:51 gw kernel: 09:03: rw=1, want=67108868, limit=4192192
Sep 25 10:46:51 gw kernel: attempt to access beyond end of device
Sep 25 10:46:51 gw kernel: 09:03: rw=1, want=1740123140, limit=4192192
Sep 25 10:46:51 gw kernel: attempt to access beyond end of device
Sep 25 10:46:51 gw kernel: 09:03: rw=1, want=1912610820, limit=4192192
Sep 25 10:46:51 gw kernel: attempt to access beyond end of device
Sep 25 10:46:51 gw kernel: 09:03: rw=1, want=536870916, limit=4192192
Sep 25 10:46:51 gw kernel: Assertion failure in 
__journal_remove_journal_head() at journal.c:1730: "buffer_jbd(bh)"

ksymoops:
Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base 
says c01e0cf0, System.map says c014f3e0.  Ignoring ksyms_base entry
Warning (compare_maps): mismatch on symbol zeroes  , ipsec says 
d0a2bd00, /lib/modules/2.4.18-vpn1.0/kernel/net/ipsec/ipsec.o says 
d0a2bc00.  Ignoring /lib/modules/2.4.18-vpn1.0/kernel/net/ipsec/ipsec.o 
entry
Warning (map_ksym_to_module): cannot match loaded module ext3 to a 
unique module object.  Trace may not be reliable.
Sep 25 10:46:51 gw kernel: invalid operand: 0000
Sep 25 10:46:51 gw kernel: CPU:    0
Sep 25 10:46:51 gw kernel: EIP:    0010:[<d08181ce>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Sep 25 10:46:52 gw kernel: EFLAGS: 00010246
Sep 25 10:46:52 gw kernel: eax: 0000005c   ebx: c9e4cb60   ecx: 
c027c7e0   edx: 00000216
Sep 25 10:46:52 gw kernel: esi: cf698ca0   edi: 0000000a   ebp: 
00000000   esp: cf875e54
Sep 25 10:46:52 gw kernel: ds: 0018   es: 0018   ss: 0018
Sep 25 10:46:52 gw kernel: Process kjournald (pid: 182, stackpage=cf875000)
Sep 25 10:46:52 gw kernel: Stack: d0819e60 d0818a98 d08188bf 000006c2 
d0818ab6 c9e4cb60 cf698ca0 d0818279
Sep 25 10:46:52 gw kernel:        c9e4cb60 c9e4cb60 d0814804 cf698ca0 
00000000 00000f94 c9da206c 00000000
Sep 25 10:46:52 gw kernel:        ceaa69e0 cf698ca0 08000000 c02fb800 
c02fb7c0 c02fb800 c02fb7c0 ca165b60
Sep 25 10:46:52 gw kernel: Call Trace: [<d0819e60>] [<d0818a98>] 
[<d08188bf>] [<d0818ab6>] [<d0818279>]
Sep 25 10:46:52 gw kernel:    [<d0814804>] [<c010844d>] [<c010846c>] 
[<c010a498>] [<c0105a2e>] [<c01126b6>]
Sep 25 10:46:52 gw kernel:    [<d08169d6>] [<d08168b0>] [<c0105726>] 
[<d08168d0>]
Sep 25 10:46:52 gw kernel: Code: 0f 0b 83 c4 14 39 1e 74 23 68 c5 8a 81 
d0 68 c3 06 00 00 68

 >>EIP; d08181ce <[jbd]__journal_remove_journal_head+7e/e0>   <=====
Trace; d0819e60 <[jbd].rodata.end+1a91/4cb9>
Trace; d0818a98 <[jbd].rodata.end+6c9/4cb9>
Trace; d08188bf <[jbd].rodata.end+4f0/4cb9>
Trace; d0818ab6 <[jbd].rodata.end+6e7/4cb9>
Trace; d0818279 <[jbd]journal_unlock_journal_head+49/60>
Trace; d0814804 <[jbd]journal_commit_transaction+854/e6a>
Trace; c010844d <do_IRQ+6d/b0>
Trace; c010846c <do_IRQ+8c/b0>
Trace; c010a498 <call_do_IRQ+5/d>
Trace; c0105a2e <__switch_to+3e/d0>
Trace; c01126b6 <schedule+2c6/2f0>
Trace; d08169d6 <[jbd]kjournald+106/1a0>
Trace; d08168b0 <[jbd]commit_timeout+0/10>
Trace; c0105726 <kernel_thread+26/30>
Trace; d08168d0 <[jbd]kjournald+0/1a0>
Code;  d08181ce <[jbd]__journal_remove_journal_head+7e/e0>
00000000 <_EIP>:
Code;  d08181ce <[jbd]__journal_remove_journal_head+7e/e0>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  d08181d0 <[jbd]__journal_remove_journal_head+80/e0>
   2:   83 c4 14                  add    $0x14,%esp
Code;  d08181d3 <[jbd]__journal_remove_journal_head+83/e0>
   5:   39 1e                     cmp    %ebx,(%esi)
Code;  d08181d5 <[jbd]__journal_remove_journal_head+85/e0>
   7:   74 23                     je     2c <_EIP+0x2c> d08181fa 
<[jbd]__journal_remove_journal_head+aa/e0>
Code;  d08181d7 <[jbd]__journal_remove_journal_head+87/e0>
   9:   68 c5 8a 81 d0            push   $0xd0818ac5
Code;  d08181dc <[jbd]__journal_remove_journal_head+8c/e0>
   e:   68 c3 06 00 00            push   $0x6c3
Code;  d08181e1 <[jbd]__journal_remove_journal_head+91/e0>
  13:   68 00 00 00 00            push   $0x0


4 warnings and 2 errors issued.  Results may not be reliable.

-- 
Michael T. Babcock
C.T.O., FibreSpeed Ltd.
http://www.fibrespeed.net/~mbabcock