BUG: Assertion failure with ext3-0.95 for 2.4.7

BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

Hello ext3-users,


I tested ext3 on a Linux for S/390 with several stress and benchmark test
tests and faced a kernel bug message.
The console showed the following output:

Message from syslogd@boeaet34 at Fri Aug  3 11:34:16 2001 ...
boeaet34 kernel: Assertion failure in journal_forget() at
transaction.c:1184: "!
jh->b_committed_data"

I tried the Patch from http://www.zip.com.au/~akpm/ext3-2.4-0.9.5-247.gz
with the kernel 2.4.7 with a new LVM- patch(0.9.1)  and some S/390 specific
patches. I use mke2fs version 1.22.
S/390 is a 32bit big endian machine. After compiling and running the kernel
I created an ext3-file system on an 70GB LVM. When running the postmark
test I get (reproduceable) the message from above. dmesg shows:

kernel BUG at transaction.c:1184!
illegal operation: 0001
CPU:    1
Process bench (pid: 2453, stackpage=08CEF000)

Kernel PSW:    07080000 8007f458         =journal_forget
task: 08cee000 ksp: 08cefaa8 pt_regs: 08cefa10
Kernel GPRS:
00000000  8001c118  00000022  00000001
8007f456  00c27000  00194f9a  00000001
030d2c80  074ed294  00001899  092ca350
0001f94c  8007f2c8  8007f456  08cefaa8
Kernel ACRS:
00000000  00000000  00000000  00000000
00000001  00000000  00000000  00000000
00000000  00000000  00000000  00000000
00000000  00000000  00000000  00000000
Kernel BackChain  CallChain
       08cefaa8   [<0007f456>]                =journal_forget
       08cefb10   [<000744c6>]                =ext3_forget
       08cefb70   [<000767b4>]                =ext3_clear_blocks
       08cefbd8   [<000768d4>]                =ext3_free_data
       08cefc50   [<00076c38>]                =ext3_truncate
       08cefd08   [<00074732>]                =ext3_delete_inode
       08cefd68   [<0006659a>]                =iput
       08cefdc8   [<00063dfc>]                =d_delete

I resolved the functions using the system.map file.

Has anyone saw this message before? Any ideas, clues, hints?

Please CC me , because I am not on the list.


--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Arjan van de Ven]

Christian Borntraeger wrote:

> I tried the Patch from http://www.zip.com.au/~akpm/ext3-2.4-0.9.5-247.gz
> with the kernel 2.4.7 with a new LVM- patch(0.9.1)  and some S/390 specific
> patches. I use mke2fs version 1.22.
> S/390 is a 32bit big endian machine. After compiling and running the kernel
> I created an ext3-file system on an 70GB LVM. When running the postmark
> test I get (reproduceable) the message from above. dmesg shows:

It would be interesting to know if this still happends without a beta
version of LVM,
and without LVM at all.

Greetings,
   Arjan van de Ven

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Tom Rini]

On Wed, Aug 08, 2001 at 10:46:41AM +0200, Christian Borntraeger wrote:

> I tested ext3 on a Linux for S/390 with several stress and benchmark test
> tests and faced a kernel bug message.
> The console showed the following output:
> 
> Message from syslogd@boeaet34 at Fri Aug  3 11:34:16 2001 ...
> boeaet34 kernel: Assertion failure in journal_forget() at
> transaction.c:1184: "!
> jh->b_committed_data"

Hmm.  I managed to get that oops on my PPC box too.  Can you turn on the
buffer trace code, reproduce it and post the log of that?  I bet it looks
a lot like the one I got...

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Andrew Morton]

Christian Borntraeger wrote:
> 
> Hello ext3-users,
> 
> I tested ext3 on a Linux for S/390 with several stress and benchmark test
> tests and faced a kernel bug message.
> The console showed the following output:
> 
> Message from syslogd@boeaet34 at Fri Aug  3 11:34:16 2001 ...
> boeaet34 kernel: Assertion failure in journal_forget() at
> transaction.c:1184: "!
> jh->b_committed_data"
> 

Simple bug, subtle symptoms.  Could you please retest 0.9.5
with this patch?  Thanks.

--- ext3-0_9_5/fs/ext3/inode.c	Mon Jul 30 05:46:12 2001
+++ ext3/fs/ext3/inode.c	Thu Aug  9 00:03:34 2001
@@ -1522,7 +1523,7 @@
 	 * AKPM: turn on bforget in journal_forget()!!!
 	 */
 	for (p = first; p < last; p++) {
-		u32 nr = *p;
+		u32 nr = le32_to_cpu(*p);
 		if (nr) {
 			struct buffer_head *bh;
 

Now, if all on-disk structures were defined in terms of something
like

	struct disk32 {
		u32 x;
	}

then these things wold never happen - the compiler would catch
it.

-

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Tom Rini]

On Thu, Aug 09, 2001 at 11:58:46AM -0700, Andrew Morton wrote:
> Christian Borntraeger wrote:
> > 
> > Hello ext3-users,
> > 
> > I tested ext3 on a Linux for S/390 with several stress and benchmark test
> > tests and faced a kernel bug message.
> > The console showed the following output:
> > 
> > Message from syslogd@boeaet34 at Fri Aug  3 11:34:16 2001 ...
> > boeaet34 kernel: Assertion failure in journal_forget() at
> > transaction.c:1184: "!
> > jh->b_committed_data"
> > 
> 
> Simple bug, subtle symptoms.  Could you please retest 0.9.5
> with this patch?  Thanks.

With this patch my first oops seems to have gone away.  I'm repeating
the test again, but dbench'ing 2,4,8,16,32 and then 64 (until disk
space ran out) worked this time.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Andrew Morton]

Tom Rini wrote:
> 
> With this patch my first oops seems to have gone away.  I'm repeating
> the test again, but dbench'ing 2,4,8,16,32 and then 64 (until disk
> space ran out) worked this time.

Thanks, Tom and Christian.

Yup, it's definitely a bug and the fix will be in 0.9.6 (in fact the way
things are looking at present it'll be the only substantive change in
0.9.6).

If it's possible, could you please also test journalled data mode?

It'd be interesting to sanity test recovery as well, but doing
thorough testing of recovery is hard.  That's why the ext3 patch
places interesting debug/devel code way down inside the IDE device
driver...

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Tom Rini]

On Fri, Aug 10, 2001 at 11:09:35AM -0700, Andrew Morton wrote:
> Tom Rini wrote:
> > 
> > With this patch my first oops seems to have gone away.  I'm repeating
> > the test again, but dbench'ing 2,4,8,16,32 and then 64 (until disk
> > space ran out) worked this time.
> 
> Thanks, Tom and Christian.
> 
> Yup, it's definitely a bug and the fix will be in 0.9.6 (in fact the way
> things are looking at present it'll be the only substantive change in
> 0.9.6).
> 
> If it's possible, could you please also test journalled data mode?

Sure.  It'll take me a bit longer for that tho (I've gotta get my
spare ppc box happily booting 2.4 off the disk first..)

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

>> I tried the Patch from http://www.zip.com.au/~akpm/ext3-2.4-0.9.5-247.gz
>> with the kernel 2.4.7 with a new LVM- patch(0.9.1)  and some S/390
specific
>> patches. I use mke2fs version 1.22.
>> S/390 is a 32bit big endian machine. After compiling and running the
kernel
>> I created an ext3-file system on an 70GB LVM. When running the postmark
>> test I get (reproduceable) the message from above. dmesg shows:
>
>It would be interesting to know if this still happends without a beta
>version of LVM,
>and without LVM at all.

I will try it. But if I mount the same file system as ext2 (mount ... -t
ext2) the test succeeds, so I guess it is not LVM specific. I will inform
you if I know the result.

greetings

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

>> >It would be interesting to know if this still happends without a beta
>> >version of LVM,
>> >and without LVM at all.
>>
>> I will try it. But if I mount the same file system as ext2 (mount ... -t
>> ext2) the test succeeds, so I guess it is not LVM specific. I will
inform
>> you if I know the result.
>
>Well ext3 has more debugging checks than ext3 at the moment, and also
requires the
>underlying blocklayers (LVM/RAID etc) to not lie. So the test _IS_
relevant,
>not to caste blame, but to find the interaction.....

OK.I tested it with a single 2GB disk without LVM and there was no error.

I also tested it with a 70GB LVM and /proc/sys/fs/jbd-debug set to 5.There
was also no error. After reset to 0 the error reoccured (???)
Next, I will try,using md instead of LVM to have a disk with a similar
size.

greetings

Christan Bornträger

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Tom Rini]

On Wed, Aug 08, 2001 at 04:38:36PM +0200, Christian Borntraeger wrote:

> I also tested it with a 70GB LVM and /proc/sys/fs/jbd-debug set to 5.There
> was also no error. After reset to 0 the error reoccured (???)
> Next, I will try,using md instead of LVM to have a disk with a similar
> size.

Sounds like my crash on PPC again :)  W/ jbd-debug set to 5 there's so much
I/O going on (writing out the logs) that the bug doesn't happen, I suspect.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

>Well ext3 has more debugging checks than ext3 at the moment, and also
requires the
>underlying blocklayers (LVM/RAID etc) to not lie. So the test _IS_
relevant,
>not to caste blame, but to find the interaction.....


OK, I tried it now  with md  and the bug was there again. Now during a rm
-rf * .
As the bug ouccured with LVM __and__  with MD it looks like a ext3-problem.
Tom Rini reported the same problem on a PPC-Box. Possibly it is a big
endian problem.

The message was again:
kernel: Assertion failure in journal_forget() at transaction.c:1184:
"!jh->b_committed_data"


Here a have a new backtrace, I resolved the functions manually from
system.map -hopefully without a mistake.

kernel BUG at transaction.c:1184!
illegal operation: 0001
CPU:    1
Process rm (pid: 1917, stackpage=06125000)

Kernel PSW:    070c0000 8008023c   =journal_forget
task: 06124000 ksp: 06125938 pt_regs: 061258a0
Kernel GPRS:
00000000  8001c118  00000022  00000001
8008023a  00c2a000  00197198  00000001
12fc1880  08d06494  00001899  1575c9b0
0001f94c  800800ac  8008023a  06125938
Kernel ACRS:
00000000  00000000  00000000  00000000
00000001  00000000  00000000  00000000
00000000  00000000  00000000  00000000
00000000  00000000  00000000  00000000
Kernel BackChain  CallChain
       06125938   [<0008023a>]           =journal_forget
       061259a0   [<000747e6>]           =ext3_forget
       06125a08   [<00076c00>]           =ext3_clear_blocks
       06125a70   [<00076d20>]           =ext3_free_data
       06125ae8   [<00076ee4>]           =ext3_free_branches
       06125b60   [<00076e44>]           =ext3_free_branches
       06125bd8   [<00076e44>]           =ext3_free_branches
       06125c50   [<0007723c>]           =ext3_truncate


greetings

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

> If it's possible, could you please also test journalled data mode?
It will take a while, but it is already planned.

> It'd be interesting to sanity test recovery as well, but doing
> thorough testing of recovery is hard.  That's why the ext3 patch
> places interesting debug/devel code way down inside the IDE device
> driver...

S/390 has no IDE. That might be the reason why I faced depmod-problems if I
include ext3 and jbd-debug as module.
Nevertheless debugging seems to work if I compile ext3 in the kernel.

greetings

Re: BUG: Assertion failure with ext3-0.95 for 2.4.7

[Christian Borntraeger]

> If it's possible, could you please also test journalled data mode?

I finished some tests with journalled data mode and with ordered mode. Both
modes seems to run correct, in the meaning of stability and correct log
replay, after a forced restart on S390 architecture, but further testing is
planned.

Today I started testing ext3 on an IBM zSeries  (s390x in the kernel tree).
It might be interesting for you, that s390x is a 64bit big endian machine.
I will post the results if I face any problems.

--
Mit freundlichen Grüßen / Best Regards

Christian Bornträger
IBM Deutschland Entwicklung GmbH
eServer SW  System Evaluation + Test
email: CBORNTRA@de.ibm.com
Tel +49 7031-16-3507