RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions" during builds

RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions" during builds

[Tom Epperly]

I am getting intermittent "Illegal instruction" errors during builds
of the software I am developing, and it appears to be kernel
related. I have been investigating this problem for several weeks, and
I have exhausted all the means of investigation known to me (detailed
below). There is evidence to suggest it is not a RAM problem or a
random hardware problem (see below). I can "solve" the problem by
running the non-SMP kernel and ignoring the second processor, but this
is not a particularly satisfying solution.  I am wondering if someone
can suggest some additional things I can do to understand and fix this
problem.  I would appreciate if you could CC me on replies.

EVIDENCE OF THE PROBLEM
=======================

Here is an excerpt from the make log to show the effects of the problem:

make[2]: Entering directory
`/home/epperly/tmp/nightly_qc/cronjobs/tom-linux-gcc2.96/babel/doc/talks'
rm -rf .libs _libs
rm -f *.lo
make[2]: Leaving directory
`/home/epperly/tmp/nightly_qc/cronjobs/tom-linux-gcc2.96/babel/doc/talks'
Making clean in papers
make[1]: *** [clean-recursive] Illegal instruction
make[1]: Leaving directory
`/home/epperly/tmp/nightly_qc/cronjobs/tom-linux-gcc2.96/babel/doc'
make: *** [clean-recursive] Error 1
****** make clean failed ******

Sometimes, the build runs to completion and succeeds.  When it fails, it
fails in a different spot each time. It doesn't always list "Illegal
instruction" as the error.  Here is another error message I've seen:

make[3]: *** [installcheck-local] Error 132

I could show you a lot more examples, but they don't seem to indicate more 
than the examples I've shown here.  The package we build can be downloaded 
here: http://www.llnl.gov/casc/components/docs/babel-0.6.3.tar.gz
The build uses the autotools suite, Sun's JDK 1.3.1_02, gcc, g++, g77, and 
Python.

On one occasion, random processes started dying on the machine. I had to 
reboot to recover.

The failure rate of our nightly build is between 20-40%.  These failures
exclude any that relate to things we can trace back to our coding
mistakes. The nightly builds do a sequence of configure, build and
regression testing.

MACHINE DETAILS
===============

HARDWARE	Dell Precision Workstation 530
PROCESSORS	Dual Intel Xeon Processors 1500MHz
RAM		512MB ECC RAM
O/S		RH 7.2 (upgraded from 7.1) running RH's 2.4.9-21 SMP 

$ /sbin/lsmod
Module                  Size  Used by
nfsd                   71232   8 (autoclean)
autofs                 11556   1 (autoclean)
nfs                    79840   3 (autoclean)
lockd                  53184   1 (autoclean) [nfsd nfs]
sunrpc                 64816   1 (autoclean) [nfsd nfs lockd]
3c59x                  26504   1
usb-uhci               21668   0 (unused)
usbcore                51808   1 [usb-uhci]
aic7xxx               114624   6
sd_mod                 11900   6
scsi_mod               98584   2 [aic7xxx sd_mod]

An identical machine has the same intermittent problems that my box does.

WHAT I HAVE ALREADY TRIED
=========================

1. Upgrade to from an earlier kernel to RH 2.4.9-21 SMP

   The new kernel didn't change anything.

2. Ran Dell's memory checker on the RAM for an hour. It checked out
   fine.

   The fact that another machine next door has the exact same problems 
   suggests that it isn't a random hardware problem unless they both came 
   from the same bad batch.

3. Open my case to vent additional heat.

   Someone suggested that the CPUs might be overheating and that opening
   the case might solve the problem.  It didn't solve the problem. My
   machine is in a well air conditioned room, and it didn't seem 
   excessively hot when I opened the case.  I haven't overclocked the
   machine or anything like that.

4. Disable X11 server and reboot to avoid loading nVIDIA kernel module.

   This may have lowered the frequency of problems, but it did not 
   eliminate the problem. Our nightly build still failed roughly
   20% of the time with "Illegal instruction" errors or other
   unexplainable failures.

5. Run the non-SMP 2.4.9-21 effectively turning off the second processor 
   (X11 still disabled)

   This seems to have "solved" the problem.  I've run over 22 nightly 
   builds, two at a time, on the system without a single failure.
   Running with just one processor is better than running an unstable
   two processor system, but it seems like I should be able to figure
   out how to have a stable two processor system.

I have not tried compiling my own kernel because I don't have root on this
machine yet. I work in an environment where they try to centralize machine
administration, so I need special permission to get root.  There is also a
desire to stick with generic RH software components. Going through the
process above is part of what I've done to justify getting root, so I can
try installing a more recent kernel.

Do you agree that this is likely to be a kernel problem?  Is upgrading
the kernel my best course of action?

Here is what I get when running the non-SMP kernel. 
$ uname -a
Linux tux06.llnl.gov 2.4.9-21 #1 Thu Jan 17 14:16:30 EST 
2002 i686 unknown
$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 0
model name	: Intel(R) Xeon(TM) CPU 1500MHz
stepping	: 10
cpu MHz		: 1495.463
cache size	: 256 KB
fdiv_bug	: no
hlt_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips	: 2981.88

Thanks in advance,

Tom

--
------------------------------------------------------------------------
Tom Epperly
Center for Applied Scientific Computing   Phone: 925-424-3159
Lawrence Livermore National Laboratory      Fax: 925-424-2477
L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly@llnl.gov
------------------------------------------------------------------------

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[Alan Cox]

> Do you agree that this is likely to be a kernel problem?  Is upgrading
> the kernel my best course of action?

Almost every other report I have ever seen that looked like that one has always
turned out to be hardware related. The randomness in paticular tends to be 
a pointer to thinks like cache faults.

You do have ECC main memory which is good.

What other hardware is in the machine ?

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[Tom Epperly]

On Thu, 21 Feb 2002, Alan Cox wrote:

> Almost every other report I have ever seen that looked like that one has
> always turned out to be hardware related. The randomness in paticular
> tends to be a pointer to thinks like cache faults.

Is there any way to test this?
 
> You do have ECC main memory which is good.
> 
> What other hardware is in the machine ?

1. SCSI harddisk plugged into the motherboard
2. Floppy drive plugged into the motherboard
3. CD-ROM and a never used Zip disk drive plugged into IDE2 on the 
   motherboard
4. Sound Blaster live sound code plugged into a PCI slot.
5. nVidia Corp NV15 GL (Quadro2) plugged into the AGP slot.

No USB ports are in use. The video output, mouse and keyboard are plugged 
into a KVM switch.  There is also a CAT-5 cable attached to the network 
jack.  Otherwise, there are no other external connections outside of the 
power cable.

By running without the X11 server, I hoped to remove the nVidia board as a 
source of trouble.

$ /sbin/lspci
00:00.0 Host bridge: Intel Corporation 82850 860 (Wombat) Chipset Host Bridge (MCH) (rev 04)
00:01.0 PCI bridge: Intel Corporation 82850 850 (Tehama) Chipset AGP Bridge (rev 04)
00:02.0 PCI bridge: Intel Corporation 82860 860 (Wombat) Chipset AGP Bridge (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801BAM PCI (rev 04)
00:1f.0 ISA bridge: Intel Corporation 82801BA ISA Bridge (ICH2) (rev 04)
00:1f.1 IDE interface: Intel Corporation 82801BA IDE U100 (rev 04)
00:1f.2 USB Controller: Intel Corporation 82801BA(M) USB (Hub A) (rev 04)
00:1f.3 SMBus: Intel Corporation 82801BA(M) SMBus (rev 04)
00:1f.4 USB Controller: Intel Corporation 82801BA(M) USB (Hub B) (rev 04)
00:1f.5 Multimedia audio controller: Intel Corporation 82801BA(M) AC'97 Audio (rev 04)
01:00.0 VGA compatible controller: nVidia Corporation NV15 GL (Quadro2) (rev a4)
02:1f.0 PCI bridge: Intel Corporation 82806AA PCI64 Hub PCI Bridge (rev 03)
03:00.0 PIC: Intel Corporation 82806AA PCI64 Hub Advanced Programmable Interrupt Controller (rev 01)
03:0e.0 SCSI storage controller: Adaptec 7892P (rev 02)
04:0b.0 Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 78)
04:0c.0 FireWire (IEEE 1394): Texas Instruments: Unknown device 8020
04:0d.0 Multimedia audio controller: Creative Labs SB Live! EMU10000 (rev 08)
04:0d.1 Input device controller: Creative Labs SB Live! (rev 08)

Tom

--
------------------------------------------------------------------------
Tom Epperly
Center for Applied Scientific Computing   Phone: 925-424-3159
Lawrence Livermore National Laboratory      Fax: 925-424-2477
L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly@llnl.gov
------------------------------------------------------------------------

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[arjan]

In article <Pine.LNX.4.44.0202211010270.19681-100000@tux06.llnl.gov> you wrote:
> 5. nVidia Corp NV15 GL (Quadro2) plugged into the AGP slot.

> By running without the X11 server, I hoped to remove the nVidia board as a 
> source of trouble.

did you ever install the NVidia driver ?

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[Tom Epperly]

On Thu, 21 Feb 2002 arjan@fenrus.demon.nl wrote:

> In article <Pine.LNX.4.44.0202211010270.19681-100000@tux06.llnl.gov> you wrote:
> > 5. nVidia Corp NV15 GL (Quadro2) plugged into the AGP slot.
> 
> > By running without the X11 server, I hoped to remove the nVidia board as a 
> > source of trouble.
> 
> did you ever install the NVidia driver ?

The NVidia drivers (kernel module and X11) are installed, but I have
rebooted since disabling the X11 server. /sbin/lsmod does not list the
NVdriver in the running system.  Will the kernel load NVdriver if the X11
server is never started after a reboot?  /etc/modules.conf has this line

alias char-major-195 NVdriver

Tom

--
------------------------------------------------------------------------
Tom Epperly
Center for Applied Scientific Computing   Phone: 925-424-3159
Lawrence Livermore National Laboratory      Fax: 925-424-2477
L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly@llnl.gov
------------------------------------------------------------------------

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[Alan Cox]

> NVdriver in the running system.  Will the kernel load NVdriver if the X11
> server is never started after a reboot?  /etc/modules.conf has this line

Shouldn't do

> alias char-major-195 NVdriver

Change to 

alias char-major-195 off

and it should be fine

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[J Sloan]

I really doubt the nvidia is the problem
unless it's actually bad hardware doing
strange things to the bus.

FWIW I've torture tested linux with voodoo,
nvidia and radeon cards, and interestingly
enough the only problems I've seen are
with the radeon.

However to satisfy all demands you could
change the modules.conf line to read

alias char-major-195 off

I predict that will change nothing...

Joe

Tom Epperly wrote:

>On Thu, 21 Feb 2002 arjan@fenrus.demon.nl wrote:
>
>>In article <Pine.LNX.4.44.0202211010270.19681-100000@tux06.llnl.gov> you wrote:
>>
>>>5. nVidia Corp NV15 GL (Quadro2) plugged into the AGP slot.
>>>
>>>By running without the X11 server, I hoped to remove the nVidia board as a 
>>>source of trouble.
>>>
>>did you ever install the NVidia driver ?
>>
>
>The NVidia drivers (kernel module and X11) are installed, but I have
>rebooted since disabling the X11 server. /sbin/lsmod does not list the
>NVdriver in the running system.  Will the kernel load NVdriver if the X11
>server is never started after a reboot?  /etc/modules.conf has this line
>
>alias char-major-195 NVdriver
>
>Tom
>
>--
>------------------------------------------------------------------------
>Tom Epperly
>Center for Applied Scientific Computing   Phone: 925-424-3159
>Lawrence Livermore National Laboratory      Fax: 925-424-2477
>L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly@llnl.gov
>------------------------------------------------------------------------
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[Richard B. Johnson]

On Thu, 21 Feb 2002, Tom Epperly wrote:

> On Thu, 21 Feb 2002, Alan Cox wrote:
> 
> > Almost every other report I have ever seen that looked like that one has
> > always turned out to be hardware related. The randomness in paticular
> > tends to be a pointer to thinks like cache faults.
> 

> 5. nVidia Corp NV15 GL (Quadro2) plugged into the AGP slot.
     ^^^^^^^^^^^^^^^^^^^^^_________ Bingo!

Just for kicks, grab some other screen card.


Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (797.90 BogoMips).

        111,111,111 * 111,111,111 = 12,345,678,987,654,321

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[Alan Cox]

> > tends to be a pointer to thinks like cache faults.
> Is there any way to test this?

Well in theory the cpu should report it with a machine check (which we
would report). Sometimes it doesnt

> By running without the X11 server, I hoped to remove the nVidia board as a 
> source of trouble.

If you booted and never ran X or hand loaded the nvidia module you did that

> 00:00.0 Host bridge: Intel Corporation 82850 860 (Wombat) Chipset Host Bridge (MCH) (rev 04)

when you see the nicknames it makes you wonder if intel know that wombat is
"waste of money brains and time" in business speak 8)

Nothing else in the hardware really stands out. You can avoid the sb live!
for testing I guess but I wouldnt have expected it to be a problem.

Possibly one hardware thing to try (depending on who and how the box is
maintained) is swapping the cpus over and seeing if it then works single
cpu..

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal

[Tom Epperly]

On Thu, 21 Feb 2002, Alan Cox wrote:

> Possibly one hardware thing to try (depending on who and how the box is
> maintained) is swapping the cpus over and seeing if it then works single
> cpu..

I tried swapping the cpus (i.e. swapping chips), and it still runs fine 
non-SMP.

Tom

--
------------------------------------------------------------------------
Tom Epperly
Center for Applied Scientific Computing   Phone: 925-424-3159
Lawrence Livermore National Laboratory      Fax: 925-424-2477
L-661, P.O. Box 808, Livermore, CA 94551  Email: tepperly@llnl.gov
------------------------------------------------------------------------

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[Tom Epperly]

Alan Cox wrote:

>>Do you agree that this is likely to be a kernel problem?  Is upgrading
>>the kernel my best course of action?
>>
>
>Almost every other report I have ever seen that looked like that one has always
>turned out to be hardware related. The randomness in paticular tends to be 
>a pointer to thinks like cache faults.
>
>You do have ECC main memory which is good.
>
>What other hardware is in the machine ?
>
>
To recap from an earlier email, my nightly build & regression tests (a 
roughly 2 hour process involving Sun's JDK, GNU make, gcc, g++, g77 & 
Bourne shell scripts) has been failing intermittently on a dual-Xeon 
system usually with an "Illegal intruction" signal. I've tried removing 
the sound card and disabling the X11 server to avoid loading the nVidia 
kernel mod. The intermittent failures disappear when I run non-SMP. I've 
tried swapping the processors on the motherboard, and both processors 
appear to work fine individually. Most of the boxes I've run on have >= 
512MB ECC RAM. I've run Dell's hardware diagnostics (especially the 
memory ones) twice. The diagnostics don't seem to have SMP tests where 
both CPUs are being stressed.

FYI when I upgraded to the 2.4.18-1smp kernel, the failure rate went 
from 20% to 100%. I have tried running the nightly build & regression on 
roughly 6 different dual processors Pentium III or better machines 
(cylcing it over and over), and they all have intermittent failures of 
one kind or another. All these machines are made by Dell, but they 
provide some evidence that it is not a hardware problem.

Tom

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[Alan Cox]

> FYI when I upgraded to the 2.4.18-1smp kernel, the failure rate went 
> from 20% to 100%. I have tried running the nightly build & regression on 
> roughly 6 different dual processors Pentium III or better machines 
> (cylcing it over and over), and they all have intermittent failures of 
> one kind or another. All these machines are made by Dell, but they 
> provide some evidence that it is not a hardware problem.

It does sound like either a bad compiler (eg gcc 3) which is unlikely but
possible or your workload triggers something important and unusual which is
buggy. Do you know which bit of the workload is the trigger - is it practical
to find out ?

Alan

Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal instructions"

[James Washer]

Tom,

If I send you a patch to do a bit of debugging, would you be able to run
it? Basically, my plan is just to call a die() like routine for  trap=6, so
that we get a good stack frame (much like an oops).. From there, we should
be able to figure out WHY the program is getting an illegal op..

My guesses at this point in time are....  bad file io( i.e. the executable
is corrupt) , or bad mem... (check to see if the same phys page is being
used, for example)

 - jim

"Tom Epperly" <tepperly@llnl.gov>@vger.kernel.org on 03/11/2002 08:07:05 AM

Sent by:    linux-kernel-owner@vger.kernel.org


To:    Alan Cox <alan@lxorguk.ukuu.org.uk>
cc:    linux-kernel@vger.kernel.org
Subject:    Re: RH7.2 running 2.4.9-21-SMP (dual Xeon's) yields "Illegal
       instructions"



Alan Cox wrote:

>>Do you agree that this is likely to be a kernel problem?  Is upgrading
>>the kernel my best course of action?
>>
>
>Almost every other report I have ever seen that looked like that one has
always
>turned out to be hardware related. The randomness in paticular tends to be
>a pointer to thinks like cache faults.
>
>You do have ECC main memory which is good.
>
>What other hardware is in the machine ?
>
>
To recap from an earlier email, my nightly build & regression tests (a
roughly 2 hour process involving Sun's JDK, GNU make, gcc, g++, g77 &
Bourne shell scripts) has been failing intermittently on a dual-Xeon
system usually with an "Illegal intruction" signal. I've tried removing
the sound card and disabling the X11 server to avoid loading the nVidia
kernel mod. The intermittent failures disappear when I run non-SMP. I've
tried swapping the processors on the motherboard, and both processors
appear to work fine individually. Most of the boxes I've run on have >=
512MB ECC RAM. I've run Dell's hardware diagnostics (especially the
memory ones) twice. The diagnostics don't seem to have SMP tests where
both CPUs are being stressed.

FYI when I upgraded to the 2.4.18-1smp kernel, the failure rate went
from 20% to 100%. I have tried running the nightly build & regression on
roughly 6 different dual processors Pentium III or better machines
(cylcing it over and over), and they all have intermittent failures of
one kind or another. All these machines are made by Dell, but they
provide some evidence that it is not a hardware problem.

Tom

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/