From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keith Mannthey Subject: Re: EDAC linux-2.6.34-rc5 non correctable errors not reported on AMD64 opteron Date: Thu, 29 Apr 2010 17:38:38 -0700 Message-ID: <1272587918.3792.59.camel@keith-laptop> References: <20100429183003.GB20328@ppanchamukhi> <1272579222.3792.18.camel@keith-laptop> <20100429223121.GA20899@ppanchamukhi> <1272583087.3792.35.camel@keith-laptop> <20100430001241.GB20899@ppanchamukhi> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20100430001241.GB20899@ppanchamukhi> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bluesmoke-devel-bounces@lists.sourceforge.net To: "Prasanna S. Panchamukhi" Cc: Rob Becker , "bluesmoke-devel@lists.sourceforge.net" , Arthur Jones , "dougthompson@xmission.com" List-Id: edac.vger.kernel.org On Thu, 2010-04-29 at 17:12 -0700, Prasanna S. Panchamukhi wrote: > On Thu, Apr 29, 2010 at 04:18:07PM -0700, Keith Mannthey wrote: > > On Thu, 2010-04-29 at 15:31 -0700, Prasanna S. Panchamukhi wrote: > > > On Thu, Apr 29, 2010 at 03:13:42PM -0700, Keith Mannthey wrote: > > > > On Thu, 2010-04-29 at 11:30 -0700, Prasanna S. Panchamukhi wrote: > > > > > Hi Doug, > > > > > > > > > > I am testing Linux-2.6.34-rc5 EDAC driver on AMD64 Opteron. > > > > > I am able to inject single bit errors and get the edac driver report the > > > > > correctable errors. > > > > > But when I inject 2-bit errors, I did not see any notification or kernel > > > > > log, the system simply hangs. > > > > > This happens with or without edac_mc_panic_on_ue enabled. > > > > > Please let me know if I am missing something. > > > > > Below are the details. > > > > > > > > I would have to recheck the specs to be 100% sure but I would consider > > > > double bit errors to be fatal on normal Opteron boxes. There is a good > > > > chance your BIOS detects the fatal error and freezes the box to prevent > > > > data corruption. > > > > > > Shouldn't the edac driver be reporting Uncorrectable Errors even > > > before ..BIOS detects the fatal error and freezes the box? > > > Did someone already tested the 2-bit error injection and reporting > > > on AMD64? > > > > I don't know your hardware or firmware but a SMI can be uses when the > > fatal error is triggered. BIOS can get instant notification. > > > > Thanks Keith for your response. > My current hardware is AMD64 Opteron Family F10. This behavior is mostly firmware/bios dependent. The cpu error handling and gets programed by your BIOS. > > > > > Did the edac driver reported Uncorrectable Errors on other architectures > > > Powerpc/Intel? > > > > This is a very specific question that requires firmware level knowledge > > of your system. On my AMD and Intel system the bios steps in and kills > > the box when faced with a double bit error and this is the safest > > possible scenario for a double bit error. From your report it sounds > > like you box may be doing something similar. > > Killing of the system is expected. But even before the system > gets killed, the edac driver should log saying..double bit UE detected. > And from your above statement, it looks like neither you saw > the Uncorrectable errors being reported but just the system gets killed. Correct, on my systems the box dies and the bios reports the error properly. The OS is not involved in anywhat it is a hard power cycle. > I would expect the edac driver to report Uncorrectable errors > before it kills the box. In a real error your box may panic before EDAC gets a chance to poll and process the error. IE you might catch the fatal ecc error on load into the data/instruction cache and run some very errant command before the next EDAC poll. With your error injection you probably don't have to worry about that. You might look into the current driver I heard there was a way to offline specific pages on error reports. I have no idea if the current AMD driver is doing this. > Also there is "edac_mc_panic_on_ue" module param. if enabled it > should result in system panic. Correct there are lots of different ue errors. Panic is generally a good idea although I have seen error labeled fatal that were safe to ignore on one system (Intel TMID) > > > > I didn't get around to playing with the error injection code could be > > something going on there. I had real debug dims with switches on them > > to generate an error. > > > > AMD error injection is a very cool feature, that helps us to check > the system capabilites to report and detect the memory errors. Yes I understand what it is used for. Be sure to properly investigate mapping edac errors to human usable error information if you are relying on EDAC for reporting in a system of any complexity. > > You are triggering a double bit error and waiting around for over a min > > for the box to hang? If so I doubt it is the BIOS that is killing your > > box as that should happen right away. > > My system does not have any services running, hence there is a delay > before the system hangs. But generally its pretty quick. Your system is panicing within a second (less that 10s for sure) caused by EDAC processing the double bit errors as it stands now? You are just wanting an edac UE error notification in dmesg? Thanks, Keith > Thanks > Prasanna > > > > > Thanks, > > Keith > > > > > Thanks > > > Prasanna > > > > > > > > > > > > > > Thanks, > > > > Keith > > > > > > > > > Thanks > > > > > Prasanna > > > > > > > > > > Steps to reproduce the problem: > > > > > > > > > > > > > > > 1. Build Linux-2.6.34-rc5 using x86_64_defconfig with following additional config options enabled: > > > > > CONFIG_EDAC_DECODE_MCE=y > > > > > CONFIG_EDAC_MM_EDAC=y > > > > > CONFIG_EDAC_AMD64=m > > > > > CONFIG_EDAC_AMD64_ERROR_INJECTION=y > > > > > CONFIG_EDAC_E752X=m > > > > > CONFIG_EDAC_I82975X=m > > > > > CONFIG_EDAC_I3000=m > > > > > CONFIG_EDAC_I3200=m > > > > > CONFIG_EDAC_X38=m > > > > > CONFIG_EDAC_I5400=m > > > > > CONFIG_EDAC_I5000=m > > > > > CONFIG_EDAC_I5100=m > > > > > > > > > > 2. insert the kernel module > > > > > #insmod amd64_edac_mod.ko > > > > > > > > > > 3. Inject errors > > > > > > > > > > # echo 3 > /sys/devices/system/edac/mc/mc0/inject_section > > > > > # echo 7 > /sys/devices/system/edac/mc/mc0/inject_word > > > > > # echo 0x88 > /sys/devices/system/edac/mc/mc0/inject_ecc_vector > > > > > # echo 1 > /sys/devices/system/edac/mc/mc0/inject_read > > > > > # echo 1 > /sys/devices/system/edac/mc/mc0/inject_write > > > > > > > > > > 4. Should hang the system in few minutes. > > > > > > > > > > Additional info: > > > > > - AMD64 opteron > > > > > # cat /proc/cpuinfo > > > > > processor : 0 > > > > > vendor_id : AuthenticAMD > > > > > cpu family : 16 > > > > > model : 2 > > > > > model name : Quad-Core AMD Opteron(tm) Processor 2346 HE > > > > > stepping : 3 > > > > > cpu MHz : 1800.023 > > > > > cache size : 512 KB > > > > > physical id : 0 > > > > > siblings : 4 > > > > > core id : 0 > > > > > cpu cores : 4 > > > > > apicid : 0 > > > > > initial apicid : 0 > > > > > fpu : yes > > > > > fpu_exception : yes > > > > > cpuid level : 5 > > > > > wp : yes > > > > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > > > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > > > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > > > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > > > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock > > > > > bogomips : 3600.04 > > > > > TLB size : 1024 4K pages > > > > > clflush size : 64 > > > > > cache_alignment : 64 > > > > > address sizes : 48 bits physical, 48 bits virtual > > > > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > > > > > > > > processor : 1 > > > > > vendor_id : AuthenticAMD > > > > > cpu family : 16 > > > > > model : 2 > > > > > model name : Quad-Core AMD Opteron(tm) Processor 2346 HE > > > > > stepping : 3 > > > > > cpu MHz : 1800.023 > > > > > cache size : 512 KB > > > > > physical id : 0 > > > > > siblings : 4 > > > > > core id : 1 > > > > > cpu cores : 4 > > > > > apicid : 1 > > > > > initial apicid : 1 > > > > > fpu : yes > > > > > fpu_exception : yes > > > > > cpuid level : 5 > > > > > wp : yes > > > > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > > > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > > > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > > > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > > > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock > > > > > bogomips : 3600.08 > > > > > TLB size : 1024 4K pages > > > > > clflush size : 64 > > > > > cache_alignment : 64 > > > > > address sizes : 48 bits physical, 48 bits virtual > > > > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > > > > > > > > processor : 2 > > > > > vendor_id : AuthenticAMD > > > > > cpu family : 16 > > > > > model : 2 > > > > > model name : Quad-Core AMD Opteron(tm) Processor 2346 HE > > > > > stepping : 3 > > > > > cpu MHz : 1800.023 > > > > > cache size : 512 KB > > > > > physical id : 0 > > > > > siblings : 4 > > > > > core id : 2 > > > > > cpu cores : 4 > > > > > apicid : 2 > > > > > initial apicid : 2 > > > > > fpu : yes > > > > > fpu_exception : yes > > > > > cpuid level : 5 > > > > > wp : yes > > > > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > > > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > > > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > > > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > > > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock > > > > > bogomips : 3599.96 > > > > > TLB size : 1024 4K pages > > > > > clflush size : 64 > > > > > cache_alignment : 64 > > > > > address sizes : 48 bits physical, 48 bits virtual > > > > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > > > > > > > > processor : 3 > > > > > vendor_id : AuthenticAMD > > > > > cpu family : 16 > > > > > model : 2 > > > > > model name : Quad-Core AMD Opteron(tm) Processor 2346 HE > > > > > stepping : 3 > > > > > cpu MHz : 1800.023 > > > > > cache size : 512 KB > > > > > physical id : 0 > > > > > siblings : 4 > > > > > core id : 3 > > > > > cpu cores : 4 > > > > > apicid : 3 > > > > > initial apicid : 3 > > > > > fpu : yes > > > > > fpu_exception : yes > > > > > cpuid level : 5 > > > > > wp : yes > > > > > flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > > > > > cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt > > > > > pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nonstop_tsc > > > > > extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic > > > > > cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs npt lbrv svm_lock > > > > > bogomips : 3600.01 > > > > > TLB size : 1024 4K pages > > > > > clflush size : 64 > > > > > cache_alignment : 64 > > > > > address sizes : 48 bits physical, 48 bits virtual > > > > > power management: ts ttp tm stc 100mhzsteps hwpstate > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > _______________________________________________ > > > > > bluesmoke-devel mailing list > > > > > bluesmoke-devel@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/bluesmoke-devel > > > > > > ------------------------------------------------------------------------------