How to debug random bus errors?

How to debug random bus errors?

[Orion Poplawski]

We're seeing programs die with "bus error" (SIGBUS) randomly on a dual 
processor Opteron machine.  I've run memtest86+ and cpuburn stress tests 
with no failure.  gdb on a core file seems uninteresting.  Is there some 
way to trace the kernel to try to get more insight?

Thanks!

-- 
Orion Poplawski
System Administrator                  303-415-9701 x222
NWRA/CoRA Division                    FAX: 303-415-9702
3380 Mitchell Lane                  orion@cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com

Re: How to debug random bus errors?

[Linus Torvalds]

On Fri, 22 Sep 2006, Orion Poplawski wrote:
>
> We're seeing programs die with "bus error" (SIGBUS) randomly on a dual
> processor Opteron machine.  I've run memtest86+ and cpuburn stress tests with
> no failure.  gdb on a core file seems uninteresting.  Is there some way to
> trace the kernel to try to get more insight?

Which kernel?

Opterons in SMP environments had a TLB flushing bug if the tlb flush 
filter was enabled.

We've had that fixed for a long time, but if you have a kernel older that 
2.6.14-rc2 or so (but still new enough to use the four-level page tables: 
it's not a big window), you can hit the bug. We fixed it pretty much 
exactly a year go ("Sat Sep 17, 2005").

Other than that, does it tend to happen to the same particular program? It 
might just be a user space bug that needs a specific set of things to 
happen. Some of those bugs go away when you disable address space 
randomization etc, since they can depend on just pure luck (or rather, 
lack there-of).

If it happens to lots of different programs, and you sometimes see 
SIGSEGV's and occasionally perhaps data corruption, that does tend to 
indicate memory problems (or wild kernel pointers, but on the whole memory 
problems tend to be more common).

			Linus

Re: How to debug random bus errors?

[Orion Poplawski]

Linus Torvalds wrote:
> 
> On Fri, 22 Sep 2006, Orion Poplawski wrote:
>> We're seeing programs die with "bus error" (SIGBUS) randomly on a dual
>> processor Opteron machine.  I've run memtest86+ and cpuburn stress tests with
>> no failure.  gdb on a core file seems uninteresting.  Is there some way to
>> trace the kernel to try to get more insight?
> 
> Which kernel?
> 

Sorry, 2.6.17-1.2142_FC4smp, so fairly recent.

> Other than that, does it tend to happen to the same particular program? It 
> might just be a user space bug that needs a specific set of things to 
> happen. Some of those bugs go away when you disable address space 
> randomization etc, since they can depend on just pure luck (or rather, 
> lack there-of).

So far just one program.  Forgot about address space randomization, may 
need to look into turning that off to help debugging.

Thanks!

-- 
Orion Poplawski
System Administrator                  303-415-9701 x222
NWRA/CoRA Division                    FAX: 303-415-9702
3380 Mitchell Lane                  orion@cora.nwra.com
Boulder, CO 80301              http://www.cora.nwra.com