[parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> On Fri, Aug 12, 2005 at 09:54:55AM -0400, John David Anglin wrote:
> > ldw instruction at 101068ac has executed.  The sequence of
> > insns starting at 101068ac is the beginning of the epilogue for
> > handle_interruption (register restores and return).  Thus,
> > handle_interruption is going to return to 101068ac.  I don't
> > know but the insn at 101068ac might be generating TLB miss
> > faults.
> 
> Yes, I think we said almost the same thing. You are describing
> the case where we return from the fault and I'm describing
> the case where 101068ac causes the initial fault while
> handling a fault. Sounds like %sp was corrupted since references
> to %sp shouldn't ever fault...unless we went off the end of the stack?
> 
> I'm getting the impression this discussion should happen on
> the parisc-linux mailing list.
> 
> > I will try to set up a testcase.

With 2.6.8.1-pa11, cc1plus generates a SIGSEGV here:

0x00418d98 <ggc_set_mark+0>:    stw rp,-14(,sp)

The backtrace is:

(gdb) bt
#0  ggc_set_mark (p=0x404c12a0) at ../../gcc/gcc/ggc-page.c:1254
#1  0x00109ec0 in gt_ggc_mx_lang_tree_node (x_p=0x404c12a0) at gt-cp-tree.h:69
... [lots more frames]

(gdb) disass gt_ggc_mx_lang_tree_node
Dump of assembler code for function gt_ggc_mx_lang_tree_node:
0x00109e84 <gt_ggc_mx_lang_tree_node+0>:        stw rp,-14(,sp)
0x00109e88 <gt_ggc_mx_lang_tree_node+4>:        stw,ma r7,80(,sp)

(gdb) p/x $sp
$2 = 0xc0700040

So, the frame allocation by gt_ggc_mx_lang_tree_node appears to span
a page boundary and the store into the frame marker by ggc_set_mark
causes a data TLB miss and the SIGSEGV.  This causes the 2.6.13 kernel
to crash in what's apparently an infinite loop involving
handle_interruption.

So, I think the bug is that 2.6.13 (32-bit) crashes when we have a data
TLB miss for an address outside the allowed bounds for the stack.  Probably,
any simple program that recursively allocates stack frames until the
allowed stack size is exceeded will demonstrate the problem.

The C++ program that causes the system crash when compiled with cc1plus
from the GCC head is shown below.  Note that there is a change to GCC's
inlining in 4.1, and 4.0 doesn't cause the SIGSEGV.  My stack limit is
the default 8192 kbytes.  The c3k kernel is default except I had to
disable CONFIG_PDC_STABLE and CONFIG_INFINIBAND.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)

// { dg-options "-O2 -ftemplate-depth-20000" }

template <int I>
inline void g() { g<I-1>(); return; }

template <>
inline void g<0>() { int i; return; }

void h() {
  g<250>();
}
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

[parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Grant Grundler]

On Fri, Aug 12, 2005 at 08:00:50PM -0400, John David Anglin wrote:
> With 2.6.8.1-pa11, cc1plus generates a SIGSEGV here:
> 
> 0x00418d98 <ggc_set_mark+0>:    stw rp,-14(,sp)
> 
> The backtrace is:
> 
> (gdb) bt
> #0  ggc_set_mark (p=0x404c12a0) at ../../gcc/gcc/ggc-page.c:1254
> #1  0x00109ec0 in gt_ggc_mx_lang_tree_node (x_p=0x404c12a0) at gt-cp-tree.h:69
> ... [lots more frames]
> 
> (gdb) disass gt_ggc_mx_lang_tree_node
> Dump of assembler code for function gt_ggc_mx_lang_tree_node:
> 0x00109e84 <gt_ggc_mx_lang_tree_node+0>:        stw rp,-14(,sp)
> 0x00109e88 <gt_ggc_mx_lang_tree_node+4>:        stw,ma r7,80(,sp)
> 
> (gdb) p/x $sp
> $2 = 0xc0700040
> 
> So, the frame allocation by gt_ggc_mx_lang_tree_node appears to span
> a page boundary and the store into the frame marker by ggc_set_mark
> causes a data TLB miss and the SIGSEGV.  This causes the 2.6.13 kernel
> to crash in what's apparently an infinite loop involving
> handle_interruption.
> 
> So, I think the bug is that 2.6.13 (32-bit) crashes when we have a data
> TLB miss for an address outside the allowed bounds for the stack.  Probably,
> any simple program that recursively allocates stack frames until the
> allowed stack size is exceeded will demonstrate the problem.

While I believe you are probably right, I could not reproduce
this on my c3k here at home:

svenc3k:/# ls -l foo.*
-rw-r--r--  1 root root 180 Aug 13 04:40 foo.cc
-rw-r--r--  1 root root 748 Aug 13 04:42 foo.o
svenc3k:/# g++-4.0 -O2 -ftemplate-depth-20000 -c foo.cc
svenc3k:/# ls -l foo.*
-rw-r--r--  1 root root 180 Aug 13 04:40 foo.cc
-rw-r--r--  1 root root 748 Aug 13 04:44 foo.o
svenc3k:/# ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) unlimited
virtual memory        (kbytes, -v) unlimited
svenc3k:/#

svenc3k:/# gcc-4.0 -v
Using built-in specs.
Target: hppa-linux-gnu
Configured with: ../src/configure -v --enable-languages=c,c++,java,f95,objc,ada,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --enable-nls --without-included-gettext --enable-threads=posix --program-suffix=-4.0 --enable-__cxa_atexit --enable-libstdcxx-allocator=mt --enable-clocale=gnu --enable-libstdcxx-debug --enable-java-gc=boehm --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-4.0-1.4.2.0/jre --enable-mpfr --disable-werror --enable-checking=release hppa-linux-gnu
Thread model: posix
gcc version 4.0.2 20050806 (prerelease) (Debian 4.0.1-4)

> The C++ program that causes the system crash when compiled with cc1plus
> from the GCC head is shown below.  Note that there is a change to GCC's
> inlining in 4.1, and 4.0 doesn't cause the SIGSEGV.

I expect this is the difference...I need to use the same g++ version.
Does someone want to hack a small test program that consumes
stack space until it bombs out becuase of the ulimit?


>   My stack limit is
> the default 8192 kbytes.  The c3k kernel is default except I had to
> disable CONFIG_PDC_STABLE and CONFIG_INFINIBAND.

I'm not certain I have a perfectly clean -rc3-pa1 kernel either.
I expect I have some PCI changes and the same .config changes you
have listed above.

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

[parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> While I believe you are probably right, I could not reproduce
> this on my c3k here at home:

One additional detail is the kernel was built with
gcc version 4.0.1 (Debian 4.0.1-2).

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> svenc3k:/# gcc-4.0 -v
> Using built-in specs.
> Target: hppa-linux-gnu
> Configured with: ../src/configure -v --enable-languages=c,c++,java,f95,objc,ada,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --enable-nls --without-included-gettext --enable-threads=posix --program-suffix=-4.0 --enable-__cxa_atexit --enable-libstdcxx-allocator=mt --enable-clocale=gnu --enable-libstdcxx-debug --enable-java-gc=boehm --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-4.0-1.4.2.0/jre --enable-mpfr --disable-werror --enable-checking=release hppa-linux-gnu
> Thread model: posix
> gcc version 4.0.2 20050806 (prerelease) (Debian 4.0.1-4)

I think that i said in the report that it's a change to the inlining
in gcc 4.1.0 that triggers this.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[James Bottomley]

On Fri, 2005-08-12 at 23:02 -0600, Grant Grundler wrote:
> I expect this is the difference...I need to use the same g++ version.
> Does someone want to hack a small test program that consumes
> stack space until it bombs out becuase of the ulimit?

Here's a test program that does this:

#include <stdio.h>

char *stack_base_estimate;

void
recurse(void)
{
	char a[256];

	memset(a, '\0', sizeof(*a));
	printf("Estimating stack consumed 0x%lx\n",
	       (unsigned long)(a - stack_base_estimate));
	recurse();
}

int
main(int argc, char *argv[])
{
	char b;

	stack_base_estimate = &b;
	printf("Estimating stack base at 0x%p\n", stack_base_estimate);
	recurse();
	return 0;
}
		
But the results are as expected (on 2.6.13-rc6-pa1):

jejb@raven> ulimit -a
core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) unlimited
virtual memory        (kbytes, -v) unlimited
jejb@raven> a.out
Estimating stack base at 0x0xc0114548
Estimating stack consumed 0x80
[...]
Estimating stack consumed 0x7fefc0
Estimating stack consumed 0x7ff100
Segmentation fault

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> On Fri, 2005-08-12 at 23:02 -0600, Grant Grundler wrote:
> > I expect this is the difference...I need to use the same g++ version.
> > Does someone want to hack a small test program that consumes
> > stack space until it bombs out becuase of the ulimit?
> 
> Here's a test program that does this:
> But the results are as expected (on 2.6.13-rc6-pa1):

It doesn't trigger the fault on my c3k with 2.6.13-rc3-pa1 either.

However,

cc1plus -O2 -ftemplate-depth-20000 inline4.C

consistently crashes my c3k.  There must be something more than
just running out of stack space.  Strangely, the system doesn't
crash when cc1plus is run under gdb.

dave@hiauly6:~/gnu/gcc-4.0/objdir/gcc$ ./cc1plus --version
GNU C++ version 4.1.0 20050813 (experimental) (hppa-linux)
        compiled by GNU C version 4.1.0 20050813 (experimental).
GGC heuristics: --param ggc-min-expand=30 --param ggc-min-heapsize=4096

Another TOC gave the following:

r2	0x1010c260	search_exception_tables+0x34
IIA	0x1020c76c	search_extable+0x10

I've seen this once before.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> However,
> 
> cc1plus -O2 -ftemplate-depth-20000 inline4.C
> 
> consistently crashes my c3k.

I tried this with builds of 2.6.13-rc6-pa1 with debian gcc-3.3, gcc-3.4
and gcc-4.0.  They all crash in the same manner.

I will send cc1plus by separate mail to anyone interested in trying to
duplicate this.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

[parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> > > Odd that it uses "rp" for the insn at 101068ac and "%r2"
> > > for the b,l insn at 101068a4.
> > 
> > I'll look at it.  It's a small inconsistency in hppa-dis.c,
> > affecting objdump, etc.
> 
> ok. minor problem though.

This is now fixed in binutils head along with a couple of other
special cases where we printed '%' before register names.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Randolph Chung]

Can somebody with a 32-bit machine try this slightly modified test case?
In the presence of a signal handler for SIGSEGV, the program will loop
segfaulting because there's not enough stack space left for the signal
handler. On an a500, this program just loops, but perhaps on a 32-bit
machine it crashes the machine if run long enough. Seems to suggest a
problem in our kernel entry/exit paths?

randolph

================================================================

#include <stdio.h>
#include <signal.h>
#include <stdlib.h>

char *stack_base_estimate;

static void sighandler(int sig)
{
        printf("Got signal %d\n", sig);
        exit(0);
}

void
recurse(void)
{
        char a[256];

        memset(a, '\0', sizeof(*a));
        printf("Estimating stack consumed 0x%lx\n",
               (unsigned long)(a - stack_base_estimate));
        recurse();
}

int
main(int argc, char *argv[])
{
        char b;

        signal(SIGSEGV, sighandler);

        stack_base_estimate = &b;
        printf("Estimating stack base at 0x%p\n", stack_base_estimate);
        recurse();
        return 0;
}

_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> static void sighandler(int sig)
> {
>         printf("Got signal %d\n", sig);
>         exit(0);

I think you need to use _exit in a signal handler although this may
not be an issue when there is only one thread.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> > static void sighandler(int sig)
> > {
> >         printf("Got signal %d\n", sig);
> >         exit(0);
> 
> I think you need to use _exit in a signal handler although this may
> not be an issue when there is only one thread.

GCC uses exit.  Either way, the kernel shouldn't hang.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Joel Soete]

> > > static void sighandler(int sig)
> > > {
> > >         printf("Got signal %d\n", sig);
> > >         exit(0);
> > 
> > I think you need to use _exit in a signal handler although this may
> > not be an issue when there is only one thread.
> 
> GCC uses exit.  Either way, the kernel shouldn't hang.
> 
Randolph , Dave,

Eventhought it does:
on a b180 model runing the latest cvs 2.6.13-rc6-pa1 (compiled with debia=
n
stock gcc-4.0, as it's an unstable install but downgraded libc6 2.3.2), f=
oo
runs untils:
...
Estimating stack consumed 0x7fee40
Estimating stack consumed 0x7fef80
Estimating stack consumed 0x7ff0c0
=3D=3D=3D=3D<hang>=3D=3D=3D

The ping still responding but not anymore the console (nor other process
apparently)... no more heartbit/disk leds flashing; pressing TOC buton la=
unch
the reboot but what is bad: nothing was saved in piminfo (all 0 and No va=
lid
timestamp) :-(

If some other test will be helpfull: don't hesistate.

Joel=0A=0A---------------------------------------------------------------=
=0AA free anti-spam and anti-virus filter on all Scarlet mailboxes=0AMore=
 info on http://www.scarlet.be/

_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Randolph Chung]

> on a b180 model runing the latest cvs 2.6.13-rc6-pa1 (compiled with debian
> stock gcc-4.0, as it's an unstable install but downgraded libc6 2.3.2), foo
> runs untils:
> ...
> Estimating stack consumed 0x7fee40
> Estimating stack consumed 0x7fef80
> Estimating stack consumed 0x7ff0c0
> ====<hang>===
> 
> The ping still responding but not anymore the console (nor other process
> apparently)... no more heartbit/disk leds flashing; pressing TOC buton launch
> the reboot but what is bad: nothing was saved in piminfo (all 0 and No valid
> timestamp) :-(

Yup, this is the same as what others are seeing. It's definitely a
problem with infinitely looping signal handlers.

I'm not sure what's the "correct" behavior in this case. I recall now
that we saw a similar problem with expect where the signal handler was
recursing.

Is the kernel responsible for blocking SIGSEGV handlers that generate
SIGSEGV?

randolph

_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> I'm not sure what's the "correct" behavior in this case. I recall now
> that we saw a similar problem with expect where the signal handler was
> recursing.
> 
> Is the kernel responsible for blocking SIGSEGV handlers that generate
> SIGSEGV?

See http://www.opengroup.org/onlinepubs/009695399/functions/signal.html.

This is implementation defined.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Randolph Chung]

> See http://www.opengroup.org/onlinepubs/009695399/functions/signal.html.
> 
> This is implementation defined.

I remember reading that, but does that mean hanging the system is ok? :-)

randolph
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Michael S. Zick]

On Tue August 16 2005 09:34, Randolph Chung wrote:
> > See http://www.opengroup.org/onlinepubs/009695399/functions/signal.html.
> > 
> > This is implementation defined.
> 
> I remember reading that, but does that mean hanging the system is ok? :-)
> 
The answer seems spread across several paragraphs...

If you get a signal while handling a signal, you can do a SIGDFL OR
block handling the new signal until the original signal is handled.
That applies to SIGSEGV also.

The paragraph listing SIGSEGV as a special case is preceded by:
"If and when the function returns..."

So, by my reading, if the SIGSEGV handler never returned -
then looping in the SIGSEGV handler could not happen.

Translation: Seems to be a design choice, if hanging the system
is considered to be the meaning of: "behavior is undefined"
which is reached if the SIGSEGV handler does return.

Mike
> randolph
> _______________________________________________
> parisc-linux mailing list
> parisc-linux@lists.parisc-linux.org
> http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
> 
> 
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[Randolph Chung]

>>Is the kernel responsible for blocking SIGSEGV handlers that generate
>>SIGSEGV?
> 
> See http://www.opengroup.org/onlinepubs/009695399/functions/signal.html.
> 
> This is implementation defined.

James spotted where we were not doing the right thing when setting up a 
signal handler frame. I've checked in his suggested fix to 
2.6.13-rc6-pa2. I think it should solve the problem.

randolph
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> >>Is the kernel responsible for blocking SIGSEGV handlers that generate
> >>SIGSEGV?
> > 
> > See http://www.opengroup.org/onlinepubs/009695399/functions/signal.html.
> > 
> > This is implementation defined.
> 
> James spotted where we were not doing the right thing when setting up a 
> signal handler frame. I've checked in his suggested fix to 
> 2.6.13-rc6-pa2. I think it should solve the problem.

We also don't seem to be using force_sigsegv.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Re: gsyprf11 and 2.6.13-rc3-pa1

[John David Anglin]

> > James spotted where we were not doing the right thing when setting up a 
> > signal handler frame. I've checked in his suggested fix to 
> > 2.6.13-rc6-pa2. I think it should solve the problem.
> 
> We also don't seem to be using force_sigsegv.

Doh, I should have looked at your change before mailing!

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux