Re: [PATCH] RAS/CEC: Reduce default threshold to offline a page to "2"

[Borislav Petkov <bp@alien8.de>]

On Thu, Jun 30, 2022 at 10:02:36AM -0700, Luck, Tony wrote:
> Yes. The cost to offline a page is low (4KB reduction in system capacity
> on a system with 10's or 100's of GB memory).

*If* that page is going to go bad at all.

> The risk to the system if the page does develop an uncorected error is
> high (process is killed, or system crashes).

That's not what the papers say.

> The question is whether the default threshold should be "do I feel
> lucky?" and those corrected errors are nothing to worry about. Or
> "do I want to take the safe path?" and premptively offline pages
> at the first sign of trouble.

Well, we can't decide that for every possible situation so if Intel's
recommendation is to do that on Intel systems, then users can set that.

/sys/kernel/debug/ras/cec/action_threshold is perhaps not the perfect
interface for that but we can make something more user-friendly.

> Is there a study about "wobbly" DIMMs?

Most of the papers I looked at say that the majority of errors are CE
and that there's a likelihood that those errors can turn UE but none is
quantifying that likelihood. One paper says that a huge number of the
errors are transient. If you offline such a page just because two alpha
particles flew through it, you're offlining a perfectly good page.

DRAM vendor is also important as different DRAM vendors show different
error stats. And so on and so on.

So you can't simply go and decide for all and say, the answer is 2.

> We now have some real data. Instead of a "finger in the air guess"
> that was made (on a different generation of DIMM technology ... the
> AMD paper you reference below says DDR4 is 5.5x worse than DDR3).

In the next sentence it says that the hardware handles those errors just
fine!

> Second most common on DDR4 DIMMs is "row failure". Which current ECC
> systems don't handle well.

This is not what we're talking about here - we're talking about
offlining pages after 2 CEs.

As to the row offlining - yes, no question there, we need to address
that.

> While that's low from one perspective (0.6% servers affected) it's high
> enough to be interesting to the CSP - because they lose revenue and
> reputation when they have to tell their customers: "sorry the VM you
> rented from us just crashed". Note that one physical system crashing
> may take down dozens of VMs.

So that whitepaper doesn't specify what they call "fault". Because
in one of the papers in the Reference section, they explain the
terminology:

"A fault is the underlying cause of an error, such as a stuck-at bit or
high-energy particle strike. Faults can be active (causing errors), or
dormant (not causing errors).

An error is an incorrect portion of state resulting from an active
fault, such as an incorrect value in memory. Errors may be detected and
possibly corrected by higher level mechanisms such as parity or error
correcting codes (ECC). They may also go uncorrected, or in the worst
case, completely undetected (i.e., silent)."

So even if we put on the most pessimistic glasses and say that 0.6%
of the faults result in system crashes, then CSP can go and set the
threshold to something lower for their use case after following
recommendations by DRAM and CPU vendor and so on.

> While anyone can tune the RAS_CEC threshold. The default value should
> be something reasonable. I'm sticking with "2" being much more
> reasonable default than 1023.

You can make that configurable or Intel-only or whatever - but not
unconditional for everyone.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette