Re: kernel panic at load average of 24 is it acceptable ?

Re: kernel panic at load average of 24 is it acceptable ?

[Robert Hancock]

Vikas Kedia wrote:
> Here is the output of mcelog:
> root@srv1:~# less /var/log/mcelog
> MCE 0
> CPU 0 0 data cache TSC 6988ae18046
> ADDR f87f5ec0
>  Data cache ECC error (syndrome ce)
>       bit46 = corrected ecc error
>  bus error 'local node origin, request didn't time out
>      data read mem transaction
>      memory access, level generic'
> STATUS 9467400000000833 MCGSTATUS 0
> MCE 0
> CPU 0 0 data cache TSC 723b38a3633
> ADDR 3d9fc0
>  Data cache ECC error (syndrome ce)
>       bit46 = corrected ecc error
>       bit62 = error overflow (multiple errors)
>  bus error 'local node origin, request didn't time out
>      data read mem transaction
>      memory access, level generic'
> STATUS d467400000000833 MCGSTATUS 0
> 
> Since it shows ECC error is the hypothesis correct that its the RAM
> problem and replacing it should solve the problem.

That looks like a data cache ECC error, which would point to a CPU 
problem in that case..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: kernel panic at load average of 24 is it acceptable ?

[Andreas Mohr]

Hi,

On Mon, Jul 17, 2006 at 08:28:03AM -0600, Robert Hancock wrote:
> Vikas Kedia wrote:
> >Since it shows ECC error is the hypothesis correct that its the RAM
> >problem and replacing it should solve the problem.
> 
> That looks like a data cache ECC error, which would point to a CPU 
> problem in that case..

(I already suspected the same privately)

IOW, pull the CPU(s), remove any and all dust you can find and reseat
them firmly, then pray and try again.
And a good idea would be to alert your local hardware shop in advance that
you might need new hardware real quick...

Andreas Mohr

kernel panic at load average of 24 is it acceptable ?

[Vikas Kedia]

I gave the following command:
$stress --cpu 12 --io 10 --vm 10 --vm-bytes 128M --timeout 100000s --verbose

and then my top shows:
top - 01:23:10 up 3:13, 3 users, load average: 24.29, 24.32, 24.50

and then the system gave a kernel panic.

Here are some of the entries in the log files:
dmesg
[ 0.000000] ACPI: Unable to locate RSDP
[ 0.000000] Virtual Wire compatibility mode.
[ 0.000000] OEM ID: TYAN <6>Product ID: S2881 <6>APIC at: 0xFEE00000
[ 0.000000] Processor #0 15:5 APIC version 16
[ 0.000000] I/O APIC #1 Version 17 at 0xFEC00000.
[ 0.000000] I/O APIC #2 Version 17 at 0xFEBFF000.
[ 0.000000] I/O APIC #3 Version 17 at 0xFEBFE000.
[ 33.869489] Using IO-APIC 1
[ 33.869495] Using IO-APIC 2
[ 33.869497] Using IO-APIC 3
[ 34.120904] ACPI: Subsystem revision 20051216
[ 34.120906] ACPI: Interpreter disabled.
[ 34.123482] PCI: MSI quirk detected. pci_msi_quirk set.
[ 34.123486] PCI: MSI quirk detected. pci_msi_quirk set.



/var/log/messages
Jul 16 17:02:36 srv1 kernel: [ 2435.535458] Machine check events logged
Jul 16 17:07:36 srv1 kernel: [ 2735.003362] Machine check events logged
Jul 16 17:12:36 srv1 kernel: [ 3034.461297] Machine check events logged
Jul 16 17:17:36 srv1 kernel: [ 3333.929201] Machine check events logged
Jul 16 17:22:36 srv1 kernel: [ 3633.397107] Machine check events logged
Jul 16 17:27:36 srv1 kernel: [ 3932.875013] Machine check events logged
Jul 16 17:32:36 srv1 kernel: [ 4232.352889] Machine check events logged
Jul 16 17:37:36 srv1 kernel: [ 4531.820809] Machine check events logged
Jul 16 17:42:36 srv1 kernel: [ 4831.288708] Machine check events logged
Jul 16 17:47:36 srv1 kernel: [ 5130.756612] Machine check events logged
Jul 16 17:52:36 srv1 kernel: [ 5430.224522] Machine check events logged
Jul 16 17:57:36 srv1 kernel: [ 5729.692441] Machine check events logged
Jul 16 19:10:05 srv1 kernel: [ 34.995486] PCI: MSI quirk detected.
pci_msi_quirk set.
Jul 16 19:10:05 srv1 kernel: [ 34.995489] PCI: MSI quirk detected.
pci_msi_quirk set.

The screen shot of the console (through the KVM) is available here:
http://www.visitlab.com/styles/basic/images/ubuntu-error1.JPG
http://www.visitlab.com/styles/basic/images/ubuntu-error2.JPG

The memtest ran fine for 8 hours:
http://www.visitlab.com/styles/basic/images/memtest.JPG

My questions are:
1. Kernel panic at load average of 24 is it acceptable ?
2. If not how do I go about debugging this kernel panic ?

Regards,

v

Re: kernel panic at load average of 24 is it acceptable ?

[Andreas Mohr]

Hi,

On Mon, Jul 17, 2006 at 12:08:41AM -0700, Vikas Kedia wrote:
> The memtest ran fine for 8 hours:
> http://www.visitlab.com/styles/basic/images/memtest.JPG
> 
> My questions are:
> 1. Kernel panic at load average of 24 is it acceptable ?

Kernel panic is _NEVER_ acceptable.
I've seen loads in the couple hundreds with no problem.

However you actually have a mce_panic() crash here.
Make sure to figure out why this Machine Check Exception got raised,
otherwise you might hose the box if you continue without investigation.
It could easily be due to mal-working CPU fan etc.pp., especially since it
happened exactly while you stress-tested the machine.

> 2. If not how do I go about debugging this kernel panic ?

Read up on MCE debugging methods on Linux or so, that should hopefully help.

Good luck!

Andreas Mohr

Re: kernel panic at load average of 24 is it acceptable ?

[Vikas Kedia]

> Read up on MCE debugging methods on Linux or so, that should hopefully help.

Here is the output of mcelog:
root@srv1:~# less /var/log/mcelog
MCE 0
CPU 0 0 data cache TSC 6988ae18046
ADDR f87f5ec0
  Data cache ECC error (syndrome ce)
       bit46 = corrected ecc error
  bus error 'local node origin, request didn't time out
      data read mem transaction
      memory access, level generic'
STATUS 9467400000000833 MCGSTATUS 0
MCE 0
CPU 0 0 data cache TSC 723b38a3633
ADDR 3d9fc0
  Data cache ECC error (syndrome ce)
       bit46 = corrected ecc error
       bit62 = error overflow (multiple errors)
  bus error 'local node origin, request didn't time out
      data read mem transaction
      memory access, level generic'
STATUS d467400000000833 MCGSTATUS 0

Since it shows ECC error is the hypothesis correct that its the RAM
problem and replacing it should solve the problem.

Regards,

Vikas

On 7/17/06, Andreas Mohr <andi@rhlx01.fht-esslingen.de> wrote:
> Hi,
>
> On Mon, Jul 17, 2006 at 12:08:41AM -0700, Vikas Kedia wrote:
> > The memtest ran fine for 8 hours:
> > http://www.visitlab.com/styles/basic/images/memtest.JPG
> >
> > My questions are:
> > 1. Kernel panic at load average of 24 is it acceptable ?
>
> Kernel panic is _NEVER_ acceptable.
> I've seen loads in the couple hundreds with no problem.
>
> However you actually have a mce_panic() crash here.
> Make sure to figure out why this Machine Check Exception got raised,
> otherwise you might hose the box if you continue without investigation.
> It could easily be due to mal-working CPU fan etc.pp., especially since it
> happened exactly while you stress-tested the machine.
>
> > 2. If not how do I go about debugging this kernel panic ?
>
> Read up on MCE debugging methods on Linux or so, that should hopefully help.
>
> Good luck!
>
> Andreas Mohr
>

Re: kernel panic at load average of 24 is it acceptable ?

[Andrey Borzenkov]

Vikas Kedia wrote:

>> Read up on MCE debugging methods on Linux or so, that should hopefully
>> help.
> 
> Here is the output of mcelog:
> root@srv1:~# less /var/log/mcelog
> MCE 0
> CPU 0 0 data cache TSC 6988ae18046
> ADDR f87f5ec0
>   Data cache ECC error (syndrome ce)
>        bit46 = corrected ecc error
>   bus error 'local node origin, request didn't time out
>       data read mem transaction
>       memory access, level generic'
> STATUS 9467400000000833 MCGSTATUS 0
> MCE 0
> CPU 0 0 data cache TSC 723b38a3633
> ADDR 3d9fc0
>   Data cache ECC error (syndrome ce)
>        bit46 = corrected ecc error
>        bit62 = error overflow (multiple errors)
>   bus error 'local node origin, request didn't time out
>       data read mem transaction
>       memory access, level generic'
> STATUS d467400000000833 MCGSTATUS 0
> 
> Since it shows ECC error is the hypothesis correct that its the RAM
> problem and replacing it should solve the problem.
> 

I am not sure if this is a question, but it shows _data cache_ multibit
error which makes it rather CPU not memory.

-andrey