[parisc-linux] c3k panics

[parisc-linux] c3k panics

[John David Anglin]

I can panic my c3750 very consistently running the binutils testsuite
(cvs source as of 20050526).  I've tried many of the default kernels
to see if I could isolate when the problem was introduced.  The last
kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
in 2.6.12-rc5-pa0 and 2.6.11-pa4.

The sad part is that panic is also broken and no error messages are
produced when the fault occurs.  Before adding panic=180 to the command
line, pressing TOC just yielded a register dump for panic itself.  That's
not too useful.  In my testing, the last working panic dump was with
2.6.9-pa1 which faulted with a HPMC during boot.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[John David Anglin]

> I can panic my c3750 very consistently running the binutils testsuite
> (cvs source as of 20050526).  I've tried many of the default kernels
> to see if I could isolate when the problem was introduced.  The last
> kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
> in 2.6.12-rc5-pa0 and 2.6.11-pa4.

Hmmm, just noticed that as-new may be using all of the machines memory
and swap space.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Carlos O'Donell]

On Sat, May 28, 2005 at 09:50:38PM -0400, John David Anglin wrote:
> > I can panic my c3750 very consistently running the binutils testsuite
> > (cvs source as of 20050526).  I've tried many of the default kernels
> > to see if I could isolate when the problem was introduced.  The last
> > kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
> > in 2.6.12-rc5-pa0 and 2.6.11-pa4.
> 
> Hmmm, just noticed that as-new may be using all of the machines memory
> and swap space.

Neither of these situations should kill the kernel.
Why doesn't OOM kill the process?

c.

_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[John David Anglin]

> On Sat, May 28, 2005 at 09:50:38PM -0400, John David Anglin wrote:
> > > I can panic my c3750 very consistently running the binutils testsuite
> > > (cvs source as of 20050526).  I've tried many of the default kernels
> > > to see if I could isolate when the problem was introduced.  The last
> > > kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
> > > in 2.6.12-rc5-pa0 and 2.6.11-pa4.
> > 
> > Hmmm, just noticed that as-new may be using all of the machines memory
> > and swap space.
> 
> Neither of these situations should kill the kernel.
> Why doesn't OOM kill the process?

I agree but we don't get any error messages, so it's hard to tell.

I can avoid the problem with ulimit -v.  However, this doesn't fix
all the stability problems.  I've had two kernel hangs since.  r2
is pointing to the return of the call to getname near the beginning
of sys_rmdir.  The IIA Offset addresses in the TOC dump point into
the fault_vector_20 jump table.  One was here:

	b,l <intr_save>,r0
==>	ldi 9, r8

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

John David Anglin wrote:
> I can panic my c3750 very consistently running the binutils testsuite
> (cvs source as of 20050526).
mmm the experimental debian bintuils-2.16 seems to works fine for me (kernel 2.6.8.1-pa11 32bit on a b180).

> I've tried many of the default kernels
> to see if I could isolate when the problem was introduced.  The last
> kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
> in 2.6.12-rc5-pa0 and 2.6.11-pa4.
> 
I observe the same differences of behaviour.

Btw, that looks like the pb I encountered many times when I tried CONFIG_DEBUG_SLAB=y?

As I tried to explain in detail (<http://lists.parisc-linux.org/pipermail/parisc-linux/2005-May/026493.html>) there is a big pb with 
this option. Unfortunately, I am not a C programer and so lake of deep knowledges to suggest a proper fix :-(

If it's your case too, can you try without this option.
(or give a try to this patch attempt: <http://lists.parisc-linux.org/pipermail/parisc-linux/2005-May/026474.html>.
it's not perfect but it helps me anyway :-) ).


> The sad part is that panic is also broken and no error messages are
> produced when the fault occurs.  Before adding panic=180 to the command
> line, pressing TOC just yielded a register dump for panic itself.  That's
> not too useful.  In my testing, the last working panic dump was with
> 2.6.9-pa1 which faulted with a HPMC during boot.
> 

The same for me, it seems that the pb appears during 2.6.9 developement;
as far as I can test, it seems that the pb appears between 2.6.9-rc2-pa2 and pa5;
For my part I already tried to revert only this pa5 (just to be sure) 
<http://lists.parisc-linux.org/pipermail/parisc-linux-cvs/2004-September/034576.html>
but it doesn't help :-)

So still have to revert few thing (but it would be hard like fixup stuff) to try to isolate the pb?

Ah still a question: is it the same pb for 32bit and 64bit twin kernel?

Thanks,
	Joel
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

Hello Randolph,

Ah that's a week that I reading code, cvs patches archives, ... and seems that something is not equal everywhere but didn't reach to 
point out since I can put together:

arch/parisc/kernel/unaligned.c
[snip]
#define FIXUP_BRANCH(lbl) \
         "\tldil L%%" #lbl ", %%r1\n"                    \
         "\tldo R%%" #lbl "(%%r1), %%r1\n"               \
         "\tbv,n %%r0(%%r1)\n"
[snip]

(since this patche: <http://cvs.parisc-linux.org/linux-2.6/arch/parisc/kernel/unaligned.c?r1=1.12&r2=1.13>
and this comment:
Revision 1.13  - (view) (download) (as text) (annotate) - [select for diffs]
Tue Sep 21 19:50:55 2004 UTC (8 months ago) by tausq
Branch: MAIN
CVS Tags: LINUS_2_6_9_RC3_MERGED, LINUS_2_6_9_RC3_PRE
Branch point for: LINUS_2_6_9_RC3
Changes since 1.12: +2 -3 lines
Diff to previous 1.12

<http://lists.parisc-linux.org/pipermail/parisc-linux/2004-September/024799.html>)

otc in arch/parisc/kernel/syscall.S
[snip]
#ifndef CONFIG_64BIT
         .macro fixup_branch,lbl
         b           \lbl
         .endm
#else
         .macro fixup_branch,lbl
         ldil        L%\lbl, %r1
         ldo         R%\lbl(%r1), %r1
         bv,n        %r0(%r1)
         .endm
#endif
[snip]

(Just for the record e.g. 64bit kernel works fine but not it's twin 32bit on b2k)

Thanks in advance for your relevant advise,
	Joel

Joel Soete wrote:
> 
> 
> John David Anglin wrote:
> 
>> I can panic my c3750 very consistently running the binutils testsuite
>> (cvs source as of 20050526).
> 
> mmm the experimental debian bintuils-2.16 seems to works fine for me 
> (kernel 2.6.8.1-pa11 32bit on a b180).
> 
>> I've tried many of the default kernels
>> to see if I could isolate when the problem was introduced.  The last
>> kernel that seems unaffected is 2.6.8.1-pa11.  The problem is present
>> in 2.6.12-rc5-pa0 and 2.6.11-pa4.
>>
> I observe the same differences of behaviour.
> 
> Btw, that looks like the pb I encountered many times when I tried 
> CONFIG_DEBUG_SLAB=y?
> 
> As I tried to explain in detail 
> (<http://lists.parisc-linux.org/pipermail/parisc-linux/2005-May/026493.html>) 
> there is a big pb with this option. Unfortunately, I am not a C 
> programer and so lake of deep knowledges to suggest a proper fix :-(
> 
> If it's your case too, can you try without this option.
> (or give a try to this patch attempt: 
> <http://lists.parisc-linux.org/pipermail/parisc-linux/2005-May/026474.html>. 
> 
> it's not perfect but it helps me anyway :-) ).
> 
> 
>> The sad part is that panic is also broken and no error messages are
>> produced when the fault occurs.  Before adding panic=180 to the command
>> line, pressing TOC just yielded a register dump for panic itself.  That's
>> not too useful.  In my testing, the last working panic dump was with
>> 2.6.9-pa1 which faulted with a HPMC during boot.
>>
> 
> The same for me, it seems that the pb appears during 2.6.9 developement;
> as far as I can test, it seems that the pb appears between 2.6.9-rc2-pa2 
> and pa5;
> For my part I already tried to revert only this pa5 (just to be sure) 
> <http://lists.parisc-linux.org/pipermail/parisc-linux-cvs/2004-September/034576.html> 
> 
> but it doesn't help :-)
> 
> So still have to revert few thing (but it would be hard like fixup 
> stuff) to try to isolate the pb?
> 
> Ah still a question: is it the same pb for 32bit and 64bit twin kernel?
> 
> Thanks,
>     Joel
> _______________________________________________
> parisc-linux mailing list
> parisc-linux@lists.parisc-linux.org
> http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
> 
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Randolph Chung]

> #ifndef CONFIG_64BIT
>         .macro fixup_branch,lbl
>         b           \lbl
>         .endm
> #else
>         .macro fixup_branch,lbl
>         ldil        L%\lbl, %r1
>         ldo         R%\lbl(%r1), %r1
>         bv,n        %r0(%r1)
>         .endm
> #endif

These two do the same thing. The 32-bit version is simpler because we
can rely on the linker to do fixups for us if the branch is too far
away. The 64-bit version always uses a long branch sequence to avoid
stub issues with the 64-bit toolchain. In the C code I have simply not
done this (micro-)optimization.

randolph
-- 
Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

[snip]
> > 
> > The same for me, it seems that the pb appears during 2.6.9 developeme=
nt;
> > as far as I can test, it seems that the pb appears between 2.6.9-rc2-=
pa2
> > and pa5;
> > For my part I already tried to revert only this pa5 (just to be sure)=


> > <http://lists.parisc-linux.org/pipermail/parisc-linux-cvs/2004-Septem=
ber/034576.html>
...
> > 
> > So still have to revert few thing (but it would be hard like fixup 
> > stuff) to try to isolate the pb?
> > 
Well again for the record, I also reverted this (always to be shure that
this is not becuase this new fixup_branch).
but the kernel alway failed the same way i.e.:
>>IASQ: 00000000 00000000 IAOQ: 1014c10c 1014c110
>> IIR: 6b800020    ISR: 00000000  IOR: 84de0906
>> CPU:        0   CR30: 15d98000 CR31: 10450000
>> ORIG_R28: 1039a000
>> IAOQ[0]: alloc_slabmgmt+0x30/0x6c
>> IAOQ[1]: alloc_slabmgmt+0x34/0x6c
>> RP(r2): cache_grow+0xd4/0x1a8
>>Kernel panic - not syncing: Kernel Fault

I also try to re-use compile option: "cflags-y        +=3D -mdisable-fpre=
gs"
but this time the system hang without any panic (nothing in pimfo after t=
he
forced reboot) nor backtrace?
so helpless :-(

Thanks,
    Joel


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[John David Anglin]

> John David Anglin wrote:
> > I can panic my c3750 very consistently running the binutils testsuite
> > (cvs source as of 20050526).
> mmm the experimental debian bintuils-2.16 seems to works fine for me (kernel 2.6.8.1-pa11 32bit on a b180).

I'm 95% certain that it's this assembler file that causes the crash:

	.data

foo:
	.block
bar:
        .block 0x3fffffff
com:

This causes the allocation of ~ 1GB of memory by the assembler.  My c3k
has 1GB of memory but the swap is only 512MB.  There aren't any limits
on this allocation (ulimit -a).  The assembler is using obstacks which
I believe use malloc for the allocation.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[John David Anglin]

>  > /usr/lib/crtn.o  -Wl,-O1 -Wl,-soname -Wl,libgcj.so.6 -o .libs/libgcj.so.6.0.0
>  > collect2: ld terminated with signal 15 [Terminated]

> Obviously this was solved by simply adding some more swap space (without panic):

No, I've not encountered this but the libgcj link is a large one, so running
out of swap space isn't unexpected.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

Hello Randolph,

>
> > #ifndef CONFIG_64BIT
> >         .macro fixup_branch,lbl
> >         b           \lbl
> >         .endm
> > #else
> >         .macro fixup_branch,lbl
> >         ldil        L%\lbl, %r1
> >         ldo         R%\lbl(%r1), %r1
> >         bv,n        %r0(%r1)
> >         .endm
> > #endif
>
> These two do the same thing. The 32-bit version is simpler because we
> can rely on the linker to do fixups for us if the branch is too far
> away. The 64-bit version always uses a long branch sequence to avoid
> stub issues with the 64-bit toolchain. In the C code I have simply not
> done this (micro-)optimization.
>
Ok clear but still confused why 32bit branch doesn't nullify the insn
in delay slot like did 64bit?

btw one more thing confusing me: the third fixup_branch macro definition
in lusercopy.S:
[snip]
         .macro fixup_branch,lbl
         ldil        L%\lbl, %r1
         ldo         R%\lbl(%r1), %r1
         bv          %r0(%r1)
         .endm

This time just bv not bv,n ?
I presume that's needed because here that's user space but couldn't we ha=
ve
2 macro name (e.g a k_fixup_br, us_fixup_br)  to avoid this confusion?

Thanks again,
    Joel


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Randolph Chung]

> Ok clear but still confused why 32bit branch doesn't nullify the insn
> in delay slot like did 64bit?

it's a bug, but fortunately that copy of the fixup_branch macro (in
syscall.S) isn't actually used ;-) I'll remove it.

> btw one more thing confusing me: the third fixup_branch macro definition
> in lusercopy.S:
> [snip]
>          .macro fixup_branch,lbl
>          ldil        L%\lbl, %r1
>          ldo         R%\lbl(%r1), %r1
>          bv          %r0(%r1)
>          .endm
> 
> This time just bv not bv,n ?

Yes, if you look at where the macro is used you will see that there is
an instruction in the delay slot for the branch that should be executed.

randolph
-- 
Randolph Chung
Debian GNU/Linux Developer, hppa/ia64 ports
http://www.tausq.org/
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

> 
> 
> > Ok clear but still confused why 32bit branch doesn't nullify the insn=

> > in delay slot like did 64bit?
> 
> it's a bug, but fortunately that copy of the fixup_branch macro (in
> syscall.S) isn't actually used ;-) I'll remove it.
> 
mmm, it seems that I am ongoing to become blind (to miss that ?)
ah no this the night (no idea where stand this dmaned bug) but I was just=

blind by a pocket torch thinking it would be a star :_(
 
> > btw one more thing confusing me: the third fixup_branch macro definit=
ion
> > in lusercopy.S:
> > [snip]
> >          .macro fixup_branch,lbl
> >          ldil        L%\lbl, %r1
> >          ldo         R%\lbl(%r1), %r1
> >          bv          %r0(%r1)
> >          .endm
> > 
> > This time just bv not bv,n ?
> 
> Yes, if you look at where the macro is used you will see that there is
> an instruction in the delay slot for the branch that should be executed=
.
> 
Cool :-)

Thanks,
    Joel


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[John David Anglin]

> > > Ok clear but still confused why 32bit branch doesn't nullify the insn
> > > in delay slot like did 64bit?
> > 
> > it's a bug, but fortunately that copy of the fixup_branch macro (in
> > syscall.S) isn't actually used ;-) I'll remove it.
> > 
> mmm, it seems that I am ongoing to become blind (to miss that ?)
> ah no this the night (no idea where stand this dmaned bug) but I was just
> blind by a pocket torch thinking it would be a star :_(

I believe that there's likely more than one bug.  After working around
the memory management bug, I tried a number of gcc builds.  Two crashes
occurred as previously reported.  I also had a third crash.  In this
one, the TOC IIA Offset value pointed into "userspace".  These all
occurred with only a few hours of running time.

I then switched from vmlinux-2.6.11-pa4-c3000_defconfig to
vmlinux-2.6.8.1-pa11-c3000_defconfig.  This kernel seems ok.  It's
survived one binutils and three full gcc builds.  However, there are
seven fails in the libstdc++ testsuite that don't occur with 2.6.11.
On the otherhand, I'm seeing a few Java processes not terminate with
2.6.11-pa4.  This problem doesn't seem to be present in 2.6.8.1-pa11
and Grant's 64-bit version of 2.6.11-pa4 on gsyprf11.

I'm going to try a few more kernels to see if it's possible to isolate
the change that introduced the problem.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] c3k panics

[Joel Soete]

>
> I believe that there's likely more than one bug.  After working around
> the memory management bug, I tried a number of gcc builds.  Two crashes=

> occurred as previously reported.  I also had a third crash.  In this
> one, the TOC IIA Offset value pointed into "userspace".  These all
> occurred with only a few hours of running time.
>
Sorry but I don't remember if I already mentioned some more results about=

test getting rid of cffc():
I reach to boot successfully this kernel which didn't panicing any more;
anyway after enough untar, rm the corresponding ext3 fs was switched to r=
ead-only
mode (corruption)?

> I then switched from vmlinux-2.6.11-pa4-c3000_defconfig to
> vmlinux-2.6.8.1-pa11-c3000_defconfig.  This kernel seems ok.  It's
> survived one binutils and three full gcc builds.  However, there are
> seven fails in the libstdc++ testsuite that don't occur with 2.6.11.
> On the otherhand, I'm seeing a few Java processes not terminate with
> 2.6.11-pa4.  This problem doesn't seem to be present in 2.6.8.1-pa11
> and Grant's 64-bit version of 2.6.11-pa4 on gsyprf11.
>
imho and according to my numerous test (that took me more time on b180) 2=
.6.8.1-pa11
(+some Kyle backport) seems to be rock solid, and iirc early 2.6.10 alrea=
dy
presented the pb (2.6.9 seems also?)

> I'm going to try a few more kernels to see if it's possible to isolate
> the change that introduced the problem.
>
Thanks (that's too much time I am working in blind on this pb, I very nee=
d
a breack)

Joel


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux