"Segfault/illegal instruction" - udevd - ntpd - glibc

"Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Hi all,

I am working with two MIPS based boards (one is MIPS and the other is
MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a consecutive
reboot test on these boards. After some number of reboots (say 80) I  am
getting "segmentaion fault/illegal instruction" while running udevd and
ntpd during bootup. Upon observing the core dump, it is noted that the
segfault occured from the _init function of libnss_dns.so (in the case
of ntpd) and libnss_compat.so (in the case of udevd). I assume that
there might be a problem somewhere in the call_init function in
glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
debugging in the call_init function, there is no segfault/illegal
instruction in the reboot testing. I have also used gdb to debug the
problem but the "segfault/illegal instruction" doesn't occur during the
reboot test. Could anyone please help me to sort out this problem. The
gdb output using coredump is attached.

Any help will be appreciated
Firos

-------------------------GDB O/P---------------------

-bash-3.00# gdb -c core udevd
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=mipsel-linux-gnu"...
Core was generated by `/sbin/udevd --daemon'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from
/lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
#0  0x2ad02ef8 in _init () from /lib/libnss_compat.so.2
(gdb)
(gdb) bt
#0  0x2ad02ef8 in _init () from /lib/libnss_compat.so.2
#1  0x00000001 in ?? ()
Cannot access memory at address 0xfffffffe
(gdb)

 

"Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Hi all,

I am working with two MIPS based boards (one is MIPS and the other is
MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a consecutive
reboot test on these boards. After some number of reboots (say 80) I  am
getting "segmentaion fault/illegal instruction" while running udevd and
ntpd during bootup. Upon observing the core dump, it is noted that the
segfault occured from the _init function of libnss_dns.so (in the case
of ntpd) and libnss_compat.so (in the case of udevd). I assume that
there might be a problem somewhere in the call_init function in
glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
debugging in the call_init function, there is no segfault/illegal
instruction in the reboot testing. I have also used gdb to debug the
problem but the "segfault/illegal instruction" doesn't occur during the
reboot test. Could anyone please help me to sort out this problem. The
gdb output using coredump is attached.

Any help will be appreciated
Firos

-------------------------GDB O/P---------------------

-bash-3.00# gdb -c core udevd
GNU gdb 6.2.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=mipsel-linux-gnu"...
Core was generated by `/sbin/udevd --daemon'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libnss_compat.so.2...done.
Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from
/lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
#0  0x2ad02ef8 in _init () from /lib/libnss_compat.so.2
(gdb)
(gdb) bt
#0  0x2ad02ef8 in _init () from /lib/libnss_compat.so.2
#1  0x00000001 in ?? ()
Cannot access memory at address 0xfffffffe
(gdb)

 

Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Ralf Baechle]

On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:

> I am working with two MIPS based boards (one is MIPS and the other is
> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a consecutive
> reboot test on these boards. After some number of reboots (say 80) I  am
> getting "segmentaion fault/illegal instruction" while running udevd and
> ntpd during bootup. Upon observing the core dump, it is noted that the
> segfault occured from the _init function of libnss_dns.so (in the case
> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
> there might be a problem somewhere in the call_init function in
> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
> debugging in the call_init function, there is no segfault/illegal
> instruction in the reboot testing. I have also used gdb to debug the
> problem but the "segfault/illegal instruction" doesn't occur during the
> reboot test. Could anyone please help me to sort out this problem. The
> gdb output using coredump is attached.

Normally the address space layout and most other variables during a
program load should be identical each time so userspace should behave
identical.   So I sense the scent of a TLB or more likely cache managment
problem.

What 2.6.18 variant exactly are you running, that is where & when did
download it, what CPU?

  Ralf

RE: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Thanks Ralf for the reply.

Regarding kernel, I am using the final release version of linux 2.6.18
kernel with some customizations. Out of the two boards which I have one
has MSP8150 Multi-Service Processor(MIPS 64) and the other has ITE 8172
system controller with RM7035C, 64-bit MIPS RISC microprocessor(MIPS V).

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org] 
Sent: Sunday, June 17, 2007 1:04 AM
To: Sadarul Firos
Cc: linux-mips@linux-mips.org
Subject: Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:

> I am working with two MIPS based boards (one is MIPS and the other is
> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a
consecutive
> reboot test on these boards. After some number of reboots (say 80) I
am
> getting "segmentaion fault/illegal instruction" while running udevd
and
> ntpd during bootup. Upon observing the core dump, it is noted that the
> segfault occured from the _init function of libnss_dns.so (in the case
> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
> there might be a problem somewhere in the call_init function in
> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
> debugging in the call_init function, there is no segfault/illegal
> instruction in the reboot testing. I have also used gdb to debug the
> problem but the "segfault/illegal instruction" doesn't occur during
the
> reboot test. Could anyone please help me to sort out this problem. The
> gdb output using coredump is attached.

Normally the address space layout and most other variables during a
program load should be identical each time so userspace should behave
identical.   So I sense the scent of a TLB or more likely cache
managment
problem.

What 2.6.18 variant exactly are you running, that is where & when did
download it, what CPU?

  Ralf

RE: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Thanks Ralf for the reply.

Regarding kernel, I am using the final release version of linux 2.6.18
kernel with some customizations. Out of the two boards which I have one
has MSP8150 Multi-Service Processor(MIPS 64) and the other has ITE 8172
system controller with RM7035C, 64-bit MIPS RISC microprocessor(MIPS V).

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org] 
Sent: Sunday, June 17, 2007 1:04 AM
To: Sadarul Firos
Cc: linux-mips@linux-mips.org
Subject: Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:

> I am working with two MIPS based boards (one is MIPS and the other is
> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a
consecutive
> reboot test on these boards. After some number of reboots (say 80) I
am
> getting "segmentaion fault/illegal instruction" while running udevd
and
> ntpd during bootup. Upon observing the core dump, it is noted that the
> segfault occured from the _init function of libnss_dns.so (in the case
> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
> there might be a problem somewhere in the call_init function in
> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
> debugging in the call_init function, there is no segfault/illegal
> instruction in the reboot testing. I have also used gdb to debug the
> problem but the "segfault/illegal instruction" doesn't occur during
the
> reboot test. Could anyone please help me to sort out this problem. The
> gdb output using coredump is attached.

Normally the address space layout and most other variables during a
program load should be identical each time so userspace should behave
identical.   So I sense the scent of a TLB or more likely cache
managment
problem.

What 2.6.18 variant exactly are you running, that is where & when did
download it, what CPU?

  Ralf

Re: [SPAM] Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Markus Gothe]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ralf Baechle wrote:
> On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:
>
>> I am working with two MIPS based boards (one is MIPS and the other is
>> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a consecutive
>> reboot test on these boards. After some number of reboots (say 80) I  am
>> getting "segmentaion fault/illegal instruction" while running udevd and
>> ntpd during bootup. Upon observing the core dump, it is noted that the
>> segfault occured from the _init function of libnss_dns.so (in the case
>> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
>> there might be a problem somewhere in the call_init function in
>> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
>> debugging in the call_init function, there is no segfault/illegal
>> instruction in the reboot testing. I have also used gdb to debug the
>> problem but the "segfault/illegal instruction" doesn't occur during the
>> reboot test. Could anyone please help me to sort out this problem. The
>> gdb output using coredump is attached.
>
> Normally the address space layout and most other variables during a
> program load should be identical each time so userspace should behave
> identical.   So I sense the scent of a TLB or more likely cache managment
> problem.
>
> What 2.6.18 variant exactly are you running, that is where & when did
> download it, what CPU?
>
>   Ralf
>
Seems to me like a timing issue if printf'ing helps. You should try
invoking udevd by gdb (without the printf's).

- --
_______________________________________

Mr Markus Gothe
Software Engineer

Phone: +46 (0)13 21 81 20 (ext. 1046)
Fax: +46 (0)13 21 21 15
Mobile: +46 (0)73 718 72 80
Diskettgatan 11, SE-583 35 Linköping, Sweden
www.27m.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGeB3S6I0XmJx2NrwRCNxBAJ9vZb0+ngxP3nW02nXvKlEKY7DDQwCglXkW
ntaDNcvzn2pNdYruCxyZqt8=
=2MG9
-----END PGP SIGNATURE-----

RE: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Hello Ralf,

As an immediate workaround I've added a dummy _init/_fini functions in
nss_dns and nss_compat libraries and no segfault/illegal instruction
error was observed for over 800 successive reboots. Since I knew
overriding _init/_fini is dangerous, added constructor/destructor fns
replacing the dummy _init/_fini. But this time again I got the
segmentation fault/illegal instruction errors. I think this will give
you more insight. I'm really stuck up in the problem :(

-----Original Message-----
From: Sadarul Firos 
Sent: Monday, June 18, 2007 3:21 PM
To: 'ralf@linux-mips.org'
Cc: 'linux-mips@linux-mips.org'
Subject: RE: "Segfault/illegal instruction" - udevd - ntpd - glibc


Thanks Ralf for the reply.

Regarding kernel, I am using the final release version of linux 2.6.18
kernel with some customizations. Out of the two boards which I have one
has MSP8150 Multi-Service Processor(MIPS 64) and the other has ITE 8172
system controller with RM7035C, 64-bit MIPS RISC microprocessor(MIPS V).

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org] 
Sent: Sunday, June 17, 2007 1:04 AM
To: Sadarul Firos
Cc: linux-mips@linux-mips.org
Subject: Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:

> I am working with two MIPS based boards (one is MIPS and the other is
> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a
consecutive
> reboot test on these boards. After some number of reboots (say 80) I
am
> getting "segmentaion fault/illegal instruction" while running udevd
and
> ntpd during bootup. Upon observing the core dump, it is noted that the
> segfault occured from the _init function of libnss_dns.so (in the case
> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
> there might be a problem somewhere in the call_init function in
> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
> debugging in the call_init function, there is no segfault/illegal
> instruction in the reboot testing. I have also used gdb to debug the
> problem but the "segfault/illegal instruction" doesn't occur during
the
> reboot test. Could anyone please help me to sort out this problem. The
> gdb output using coredump is attached.

Normally the address space layout and most other variables during a
program load should be identical each time so userspace should behave
identical.   So I sense the scent of a TLB or more likely cache
managment
problem.

What 2.6.18 variant exactly are you running, that is where & when did
download it, what CPU?

  Ralf

RE: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Sadarul Firos]

Hello Ralf,

As an immediate workaround I've added a dummy _init/_fini functions in
nss_dns and nss_compat libraries and no segfault/illegal instruction
error was observed for over 800 successive reboots. Since I knew
overriding _init/_fini is dangerous, added constructor/destructor fns
replacing the dummy _init/_fini. But this time again I got the
segmentation fault/illegal instruction errors. I think this will give
you more insight. I'm really stuck up in the problem :(

-----Original Message-----
From: Sadarul Firos 
Sent: Monday, June 18, 2007 3:21 PM
To: 'ralf@linux-mips.org'
Cc: 'linux-mips@linux-mips.org'
Subject: RE: "Segfault/illegal instruction" - udevd - ntpd - glibc


Thanks Ralf for the reply.

Regarding kernel, I am using the final release version of linux 2.6.18
kernel with some customizations. Out of the two boards which I have one
has MSP8150 Multi-Service Processor(MIPS 64) and the other has ITE 8172
system controller with RM7035C, 64-bit MIPS RISC microprocessor(MIPS V).

-----Original Message-----
From: Ralf Baechle [mailto:ralf@linux-mips.org] 
Sent: Sunday, June 17, 2007 1:04 AM
To: Sadarul Firos
Cc: linux-mips@linux-mips.org
Subject: Re: "Segfault/illegal instruction" - udevd - ntpd - glibc

On Fri, Jun 15, 2007 at 09:23:34PM +0530, Sadarul Firos wrote:

> I am working with two MIPS based boards (one is MIPS and the other is
> MIPSEL) running linux-2.6.18/glibc-2.3.5. I am performing a
consecutive
> reboot test on these boards. After some number of reboots (say 80) I
am
> getting "segmentaion fault/illegal instruction" while running udevd
and
> ntpd during bootup. Upon observing the core dump, it is noted that the
> segfault occured from the _init function of libnss_dns.so (in the case
> of ntpd) and libnss_compat.so (in the case of udevd). I assume that
> there might be a problem somewhere in the call_init function in
> glibc-2.3.5/elf/dl-init.c. After I put some printf statements for
> debugging in the call_init function, there is no segfault/illegal
> instruction in the reboot testing. I have also used gdb to debug the
> problem but the "segfault/illegal instruction" doesn't occur during
the
> reboot test. Could anyone please help me to sort out this problem. The
> gdb output using coredump is attached.

Normally the address space layout and most other variables during a
program load should be identical each time so userspace should behave
identical.   So I sense the scent of a TLB or more likely cache
managment
problem.

What 2.6.18 variant exactly are you running, that is where & when did
download it, what CPU?

  Ralf

RE: "Segfault/illegal instruction" - udevd - ntpd - glibc

[Shane McDonald]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

Hello Sadarul:

  I'm not sure if you've found a solution to your problem with udevd and
ntpd dying during your repetitive bootups.  As Ralf has pointed out, it
sounds like a TLB or cache management problem.  I believe it is the same
problem that I've run across using the PMC-Sierra Xiao Hu board, whose
solution is described here:
http://www.linux-mips.org/archives/linux-mips/2007-07/msg00205.html.

  The XH is basically the same platform as your MIPSEL board.  Looking at
the code to your BSP, release 1.8.3, I can see that the described workaround
will not be used for either of your boards.  I know definitely that the
workaround is required for your MIPSEL platform; I suspect it is also
required for the MIPS board you have, as that is using the RM9150 which uses
the E9000 core.

  Anyways, give that fix a try and see how it goes.  I'll bet it will solve
your problem.

Shane

[-- Attachment #2: Type: text/html, Size: 1056 bytes --]