Re: [PATCH] x86: Add an option to disable decoding of MCE

[Mike Waychison <mikew@google.com>]

On Tue, Jan 11, 2011 at 12:49 PM, Borislav Petkov <bp@amd64.org> wrote:
> Ok, let me preface this with an even easier suggestion: Can you simply
> not compile EDAC (which includes CONFIG_EDAC_DECODE_MCE) in your kernels
> and the whole issue with decoding disappears simply because no module
> registers as a decoder...?

The trouble here is that default_decode_mce() is still getting called
no matter what :(  It didn't really cause problems until you added
atomic_notifier_call_chain(&x86_mce_decoder_chain, 0, &m) to
machine_check_poll().

> On Tue, Jan 11, 2011 at 02:56:50PM -0500, Mike Waychison wrote:
>> >> On our systems, we do not want to have any "decoders" called on machine
>> >> check events.  These decoders can easily spam our logs and cause space
>> >> problems on machines that have a lot of correctable error events.  We
>> >> _do_ however want to get the messages delivered via /dev/mcelog for
>> >> userland processing.
>> >
>> > Ok, question: how do you guys process DRAM ECCs? And more specifically,
>> > with a large number of machines, how do you do the mapping from the DRAM
>> > ECC error address reported by MCA to a DIMM that's failing in userspace
>> > on a particular machine?
>>
>> We process machine checks in userland, using carnal knowledge of the
>> memory controller and the board specific addressing of the SPDs on the
>> various i2c busses to deswizzle and make sense of the addresses and
>> symptoms.  We then expose this digested data on the network, which is
>> dealt with at the cluster level.
>
> Right, and this means that you need to know all the memory controller
> topologies of all the different architectures and also the SPD accessing
> based on a board type could be a pain. One of the main reasons for
> fleshing out MCE decoding in the kernel was to avoid needless trouble
> like that.

It's not that painful for us.  Our firmware guys own this userland
code :)  I can see it being a pain for others however.

Doing this all in userland does have it's upsides as well fwiw.  For
example, MC4 is usually kept around across a reset, which means that
the firmware can pick it up when the system goes down due to a
processor context corruption.  We rely on these libraries as well to
decode egregious uncorrectable memory errors as well as bus errors
(like hypertransport sync floods).

*snip*

> I've
> heard similar troubles reported by other big server farm people and
> what I'm currently working on is a RAS daemon that hooks into perf thus
> enabling persistent performance events. This way, you could open a
> debugfs file (this'll move to sysfs someday) and read the same decoded
> data by mmaping the perf ringbuffer.

I'll definitely keep an eye out for your developments with RAS :)

>
> I think this is also easy disabled by not configuring EDAC, as I said
> above. Basically, if you don't enable EDAC, you can drop that patch
> too and run your kernels without any modification, or am I missing
> something..?

See the comment above.  It's the spamming for the default decoder that
I wanted to disable specifically, though my approach was a knob to
disable decoders generally.