Re: EDAC linux-2.6.34-rc5 non correctable errors not reported on AMD64 opteron

[Keith Mannthey <kmannth@us.ibm.com>]

On Thu, 2010-04-29 at 15:31 -0700, Prasanna S. Panchamukhi wrote:
> On Thu, Apr 29, 2010 at 03:13:42PM -0700, Keith Mannthey wrote:
> > On Thu, 2010-04-29 at 11:30 -0700, Prasanna S. Panchamukhi wrote:
> > > Hi Doug,
> > > 
> > > I am testing Linux-2.6.34-rc5 EDAC driver on AMD64 Opteron.
> > > I am able to inject single bit errors and get the edac driver report the 
> > > correctable errors.
> > > But when I inject 2-bit errors, I did not see any notification or kernel 
> > > log, the system simply hangs.
> > > This happens with or without edac_mc_panic_on_ue enabled.
> > > Please let me know if I am missing something.
> > > Below are the details.
> > 
> > I would have to recheck the specs to be 100% sure but I would consider
> > double bit errors to be fatal on normal Opteron boxes. There is a good
> > chance your BIOS detects the fatal error and freezes the box to prevent
> > data corruption.
> 
> Shouldn't the edac driver be reporting Uncorrectable Errors even 
> before ..BIOS detects the fatal error and freezes the box?
> Did someone already tested the 2-bit error injection and reporting
> on AMD64?

I don't know your hardware or firmware but a SMI can be uses when the
fatal error is triggered. BIOS can get instant notification. 


> Did the edac driver reported Uncorrectable Errors on other architectures
> Powerpc/Intel?

This is a very specific question that requires firmware level knowledge
of your system.  On my AMD and Intel system the bios steps in and kills
the box when faced with a double bit error and this is the safest
possible scenario for a double bit error.  From your report it sounds
like you box may be doing something similar. 

I didn't get around to playing with the error injection code could be
something going on there.  I had real debug dims with switches on them
to generate an error.  

You are triggering a double bit error and waiting around for over a min
for the box to hang? If so I doubt it is the BIOS that is killing your
box as that should happen right away. 

Thanks,
  Keith 

> Thanks
> Prasanna
> 
> 
> > 
> > Thanks,
> >   Keith 
> > 
> > > Thanks
> > > Prasanna
> > > 
> > > Steps to reproduce the problem:
> > > 
> > > 
> > > 1. Build Linux-2.6.34-rc5 using x86_64_defconfig with following additional config options enabled:
> > > CONFIG_EDAC_DECODE_MCE=y
> > > CONFIG_EDAC_MM_EDAC=y
> > > CONFIG_EDAC_AMD64=m
> > > CONFIG_EDAC_AMD64_ERROR_INJECTION=y
> > > CONFIG_EDAC_E752X=m
> > > CONFIG_EDAC_I82975X=m
> > > CONFIG_EDAC_I3000=m
> > > CONFIG_EDAC_I3200=m
> > > CONFIG_EDAC_X38=m
> > > CONFIG_EDAC_I5400=m
> > > CONFIG_EDAC_I5000=m
> > > CONFIG_EDAC_I5100=m
> > > 
> > > 2. insert the kernel module
> > > #insmod amd64_edac_mod.ko
> > > 
> > > 3. Inject errors
> > > 
> > > # echo 3 > /sys/devices/system/edac/mc/mc0/inject_section  
> > > # echo 7 > /sys/devices/system/edac/mc/mc0/inject_word
> > > # echo 0x88 > /sys/devices/system/edac/mc/mc0/inject_ecc_vector
> > > # echo 1 > /sys/devices/system/edac/mc/mc0/inject_read
> > > # echo 1 > /sys/devices/system/edac/mc/mc0/inject_write
> > > 
> > > 4. Should hang the system in few minutes.
> > > 
> > > Additional info:
> > > - AMD64 opteron
> > > # cat /proc/cpuinfo
> > > processor    : 0
> > > vendor_id    : AuthenticAMD
> > > cpu family   : 16
> > > model          : 2
> > > model name    : Quad-Core AMD Opteron(tm) Processor 2346 HE
> > > stepping       : 3
> > > cpu MHz      : 1800.023
> > > cache size    : 512 KB
> > > physical id   : 0
> > > siblings       : 4
> > > core id        : 0
> > > cpu cores    : 4
> > > apicid          : 0
> > > initial apicid    : 0
> > > fpu        : yes
> > > fpu_exception    : yes
> > > cpuid level    : 5
> > > wp        : yes
> > > flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt 
> > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc 
> > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic 
> > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock
> > > bogomips    : 3600.04
> > > TLB size    : 1024 4K pages
> > > clflush size    : 64
> > > cache_alignment    : 64
> > > address sizes    : 48 bits physical, 48 bits virtual
> > > power management: ts ttp tm stc 100mhzsteps hwpstate
> > > 
> > > processor    : 1
> > > vendor_id    : AuthenticAMD
> > > cpu family    : 16
> > > model        : 2
> > > model name    : Quad-Core AMD Opteron(tm) Processor 2346 HE
> > > stepping    : 3
> > > cpu MHz        : 1800.023
> > > cache size    : 512 KB
> > > physical id    : 0
> > > siblings    : 4
> > > core id        : 1
> > > cpu cores    : 4
> > > apicid        : 1
> > > initial apicid    : 1
> > > fpu        : yes
> > > fpu_exception    : yes
> > > cpuid level    : 5
> > > wp        : yes
> > > flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt 
> > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc 
> > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic 
> > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock
> > > bogomips    : 3600.08
> > > TLB size    : 1024 4K pages
> > > clflush size    : 64
> > > cache_alignment    : 64
> > > address sizes    : 48 bits physical, 48 bits virtual
> > > power management: ts ttp tm stc 100mhzsteps hwpstate
> > > 
> > > processor    : 2
> > > vendor_id    : AuthenticAMD
> > > cpu family    : 16
> > > model        : 2
> > > model name    : Quad-Core AMD Opteron(tm) Processor 2346 HE
> > > stepping    : 3
> > > cpu MHz        : 1800.023
> > > cache size    : 512 KB
> > > physical id    : 0
> > > siblings    : 4
> > > core id        : 2
> > > cpu cores    : 4
> > > apicid        : 2
> > > initial apicid    : 2
> > > fpu        : yes
> > > fpu_exception    : yes
> > > cpuid level    : 5
> > > wp        : yes
> > > flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt 
> > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc 
> > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic 
> > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock
> > > bogomips    : 3599.96
> > > TLB size    : 1024 4K pages
> > > clflush size    : 64
> > > cache_alignment    : 64
> > > address sizes    : 48 bits physical, 48 bits virtual
> > > power management: ts ttp tm stc 100mhzsteps hwpstate
> > > 
> > > processor    : 3
> > > vendor_id    : AuthenticAMD
> > > cpu family    : 16
> > > model        : 2
> > > model name    : Quad-Core AMD Opteron(tm) Processor 2346 HE
> > > stepping    : 3
> > > cpu MHz        : 1800.023
> > > cache size    : 512 KB
> > > physical id    : 0
> > > siblings    : 4
> > > core id        : 3
> > > cpu cores    : 4
> > > apicid        : 3
> > > initial apicid    : 3
> > > fpu        : yes
> > > fpu_exception    : yes
> > > cpuid level    : 5
> > > wp        : yes
> > > flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt 
> > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc 
> > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic 
> > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock
> > > bogomips    : 3600.01
> > > TLB size    : 1024 4K pages
> > > clflush size    : 64
> > > cache_alignment    : 64
> > > address sizes    : 48 bits physical, 48 bits virtual
> > > power management: ts ttp tm stc 100mhzsteps hwpstate
> > > 
> > > 
> > > 
> > > 
> > > ------------------------------------------------------------------------------
> > > _______________________________________________
> > > bluesmoke-devel mailing list
> > > bluesmoke-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/bluesmoke-devel
> > 


------------------------------------------------------------------------------