[PATCH 0/2] x86/mce: Rate-limit storm notices and add a corrected error ceiling

[PATCH 0/2] x86/mce: Rate-limit storm notices and add a corrected error ceiling

[Breno Leitao]

Some hosts on the Meta fleet have machine check banks that report
TOO MANY corrected errors, faster than the kernel drains them.

For instance, one host flapped a single bank in and out of storm mode
nearly ten million times over four days. The storm detected and subsided
lines were 82% of everything it wrote to the kernel log, crowding out
whatever would explain the failure.

Patch 1 rate-limits the notices.

Patch 2 adds mce=panic_on_ce_count, off by default. Such a host is not
one to keep in service; a clean panic is the better outcome.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (2):
      x86/mce: Rate-limit the CMCI storm transition notices
      x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

 Documentation/ABI/testing/sysfs-mce             | 12 ++++++++
 Documentation/admin-guide/kernel-parameters.txt | 10 +++++++
 arch/x86/kernel/cpu/mce/core.c                  | 14 +++++++--
 arch/x86/kernel/cpu/mce/internal.h              |  6 ++++
 arch/x86/kernel/cpu/mce/threshold.c             | 39 +++++++++++++++++++++++--
 5 files changed, 77 insertions(+), 4 deletions(-)
---
base-commit: 7079a12d7506b07fb53b54a664bfad5fa9b16d70
change-id: 20260820-mce-panic-on-storm-78cd09a4dcbe

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH 1/2] x86/mce: Rate-limit the CMCI storm transition notices

[Breno Leitao]

mce_track_storm() prints a line every time a bank enters storm mode and
another every time it leaves. Neither is bounded, so the log volume
follows whatever rate the bank flaps at.

One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
"CMCI storm subsided" lines for a single bank over four days, which is
a bit excessive for all monitoring purposes.

Put both transitions behind one ratelimit.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
index 6c370d5af5bd2..8591715ce2430 100644
--- a/arch/x86/kernel/cpu/mce/threshold.c
+++ b/arch/x86/kernel/cpu/mce/threshold.c
@@ -4,6 +4,7 @@
  */
 #include <linux/interrupt.h>
 #include <linux/kernel.h>
+#include <linux/ratelimit.h>
 
 #include <asm/irq_vectors.h>
 #include <asm/traps.h>
@@ -111,6 +112,13 @@ void cmci_storm_end(unsigned int bank)
 		mce_timer_kick(false);
 }
 
+/*
+ * Shared by both transitions so that a bank flapping between them cannot
+ * outrun the console.
+ */
+static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
 void mce_track_storm(struct mce *mce)
 {
 	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
@@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
 	if (storm->banks[mce->bank].in_storm_mode) {
 		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, false);
 		cmci_storm_end(mce->bank);
 	} else {
 		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, true);
 		cmci_storm_begin(mce->bank);
 	}

-- 
2.53.0-Meta

RE: [PATCH 1/2] x86/mce: Rate-limit the CMCI storm transition notices

[Luck, Tony]

> One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
> "CMCI storm subsided" lines for a single bank over four days, which is
> a bit excessive for all monitoring purposes.

Marvelous use of understatement!

Patch 1:

Acked-by: Tony Luck <tony.luck@intel.com>

-Tony

Re: [PATCH 1/2] x86/mce: Rate-limit the CMCI storm transition notices

[Breno Leitao]

On Fri, Aug 21, 2026 at 04:18:42PM +0000, Luck, Tony wrote:
> > One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
> > "CMCI storm subsided" lines for a single bank over four days, which is
> > a bit excessive for all monitoring purposes.
> 
> Marvelous use of understatement!

lol. There are benefits not using LLMs for everything :-)

[PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Breno Leitao]

A machine check bank producing corrected errors faster than the kernel can
drain them is not a machine anyone wants to keep in service, but nothing
takes it out. mce_track_storm() throttles CMCI for the bank and the
machine stays up.

Count corrected errors per bank and add mce=panic_on_ce_count=<count>
and panic the host if we have more events than set.

Off by default, for obvious reasons.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 Documentation/ABI/testing/sysfs-mce             | 12 ++++++++++++
 Documentation/admin-guide/kernel-parameters.txt | 10 ++++++++++
 arch/x86/kernel/cpu/mce/core.c                  | 14 ++++++++++++--
 arch/x86/kernel/cpu/mce/internal.h              |  6 ++++++
 arch/x86/kernel/cpu/mce/threshold.c             | 23 +++++++++++++++++++++++
 5 files changed, 63 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-mce b/Documentation/ABI/testing/sysfs-mce
index 83172f50e27c6..02009b5d581a4 100644
--- a/Documentation/ABI/testing/sysfs-mce
+++ b/Documentation/ABI/testing/sysfs-mce
@@ -95,3 +95,15 @@ Contact:	Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
 Date:		Jun 2009
 Description:
 		Disables the CMCI feature.
+
+What:		/sys/devices/system/machinecheck/machinecheckX/panic_on_ce_count
+Contact:	Breno Leitao <leitao@debian.org>
+Date:		Aug 2026
+Description:
+		Panic once a machine check bank has logged this many corrected
+		errors. 0, the default, disables it.
+
+		The setting is global rather than per-CPU, and it is compared
+		against a running total kept per CPU and per bank. Lowering it
+		below a total a bank has already reached takes the machine
+		down on that bank's next corrected error.
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index d55524e3b7246..d5005bfefb7be 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3924,6 +3924,16 @@ Kernel parameters
 		print_all
 			print all machine check logs to the console.
 
+		panic_on_ce_count=<count>
+			panic once a machine check bank has logged this many
+			corrected errors. 0, the default, disables it.
+
+			This is a running total per CPU and per bank for the
+			life of the boot, not a rate, so a machine that logs
+			a slow trickle for long enough will reach any value
+			eventually. Pick one a failing part reaches in
+			minutes and a healthy one does not reach at all.
+
 		monarchtimeout (number)
 			sets the time in us to wait for other CPUs on machine
 			checks. 0 to disable.
diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
index ab469605fc893..43dcdef1a7bbe 100644
--- a/arch/x86/kernel/cpu/mce/core.c
+++ b/arch/x86/kernel/cpu/mce/core.c
@@ -261,7 +261,7 @@ static const char *mce_dump_aux_info(struct mce *m)
 	return NULL;
 }
 
-static noinstr void mce_panic(const char *msg, struct mce_hw_err *final, char *exp)
+noinstr void mce_panic(const char *msg, struct mce_hw_err *final, char *exp)
 {
 	struct llist_node *pending;
 	struct mce_evt_llist *l;
@@ -813,6 +813,10 @@ void machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
 		barrier();
 		m->status = mce_rdmsrq(mca_msr_reg(i, MCA_STATUS));
 
+		/* The boot time poll replays errors from before this boot. */
+		if (!(flags & MCP_QUEUE_LOG))
+			mce_track_ce_count(m);
+
 		/*
 		 * Update storm tracking here, before checking for the
 		 * MCI_STATUS_VAL bit. Valid corrected errors count
@@ -2326,6 +2330,7 @@ void mce_disable_bank(int bank)
  * mce=nobootlog Don't log MCEs from before booting.
  * mce=bios_cmci_threshold Don't program the CMCI threshold
  * mce=recovery force enable copy_mc_fragile()
+ * mce=panic_on_ce_count=<n> Panic after n corrected errors on one bank
  */
 static int __init mcheck_enable(char *str)
 {
@@ -2349,7 +2354,10 @@ static int __init mcheck_enable(char *str)
 		cfg->print_all = true;
 	else if (!strcmp(str, "ignore_ce"))
 		cfg->ignore_ce = true;
-	else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog"))
+	else if (str_has_prefix(str, "panic_on_ce_count=")) {
+		str += strlen("panic_on_ce_count=");
+		get_option(&str, &cfg->panic_on_ce_count);
+	} else if (!strcmp(str, "bootlog") || !strcmp(str, "nobootlog"))
 		cfg->bootlog = (str[0] == 'b');
 	else if (!strcmp(str, "bios_cmci_threshold"))
 		cfg->bios_cmci_threshold = 1;
@@ -2617,6 +2625,7 @@ static ssize_t store_int_with_restart(struct device *s,
 }
 
 static DEVICE_INT_ATTR(monarch_timeout, 0644, mca_cfg.monarch_timeout);
+static DEVICE_INT_ATTR(panic_on_ce_count, 0644, mca_cfg.panic_on_ce_count);
 static DEVICE_BOOL_ATTR(dont_log_ce, 0644, mca_cfg.dont_log_ce);
 static DEVICE_BOOL_ATTR(print_all, 0644, mca_cfg.print_all);
 
@@ -2641,6 +2650,7 @@ static struct device_attribute *mce_device_attrs[] = {
 	&dev_attr_trigger,
 #endif
 	&dev_attr_monarch_timeout.attr,
+	&dev_attr_panic_on_ce_count.attr,
 	&dev_attr_dont_log_ce.attr,
 	&dev_attr_print_all.attr,
 	&dev_attr_ignore_ce.attr,
diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h
index a31cf984619ca..7ebf87f6ca346 100644
--- a/arch/x86/kernel/cpu/mce/internal.h
+++ b/arch/x86/kernel/cpu/mce/internal.h
@@ -37,6 +37,7 @@ struct llist_node *mce_gen_pool_prepare_records(void);
 
 int mce_severity(struct mce *a, struct pt_regs *regs, char **msg, bool is_excp);
 struct dentry *mce_get_debugfs_dir(void);
+noinstr void mce_panic(const char *msg, struct mce_hw_err *final, char *exp);
 
 extern mce_banks_t mce_banks_ce_disabled;
 
@@ -64,6 +65,7 @@ void mce_timer_kick(bool storm);
 void cmci_storm_begin(unsigned int bank);
 void cmci_storm_end(unsigned int bank);
 void mce_track_storm(struct mce *mce);
+void mce_track_ce_count(struct mce *mce);
 void mce_inherit_storm(unsigned int bank);
 bool mce_get_storm_mode(void);
 void mce_set_storm_mode(bool storm);
@@ -72,6 +74,7 @@ u32  mce_get_apei_thr_limit(void);
 static inline void cmci_storm_begin(unsigned int bank) {}
 static inline void cmci_storm_end(unsigned int bank) {}
 static inline void mce_track_storm(struct mce *mce) {}
+static inline void mce_track_ce_count(struct mce *mce) {}
 static inline void mce_inherit_storm(unsigned int bank) {}
 static inline bool mce_get_storm_mode(void) { return false; }
 static inline void mce_set_storm_mode(bool storm) {}
@@ -83,12 +86,14 @@ static inline u32  mce_get_apei_thr_limit(void) { return 0; }
  *			represents an error seen.
  *
  * timestamp:		Last time (in jiffies) that the bank was polled.
+ * ce_count:		Corrected errors logged since boot.
  * in_storm_mode:	Is this bank in storm mode?
  * poll_only:		Bank does not support CMCI, skip storm tracking.
  */
 struct storm_bank {
 	u64 history;
 	u64 timestamp;
+	u64 ce_count;
 	bool in_storm_mode;
 	bool poll_only;
 };
@@ -183,6 +188,7 @@ struct mca_config {
 	bool ignore_ce;
 	bool print_all;
 
+	int panic_on_ce_count;
 	int monarch_timeout;
 	int panic_timeout;
 	u32 rip_msr;
diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
index 8591715ce2430..208dd6c8a5b18 100644
--- a/arch/x86/kernel/cpu/mce/threshold.c
+++ b/arch/x86/kernel/cpu/mce/threshold.c
@@ -119,6 +119,29 @@ void cmci_storm_end(unsigned int bank)
 static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL,
 			      DEFAULT_RATELIMIT_BURST);
 
+/*
+ * A bank that keeps reporting corrected errors is repairing them faster than
+ * anything acts on it. Count them and let the admin put a ceiling on it.
+ */
+void mce_track_ce_count(struct mce *mce)
+{
+	struct storm_bank *bank = &this_cpu_ptr(&storm_desc)->banks[mce->bank];
+	int limit = READ_ONCE(mca_cfg.panic_on_ce_count);
+
+	if (limit <= 0)
+		return;
+
+	if (!(mce->status & MCI_STATUS_VAL) || !mce_is_correctable(mce))
+		return;
+
+	if (++bank->ce_count < (u64)limit)
+		return;
+
+	printk_deferred(KERN_EMERG "CPU%d BANK%d logged %llu corrected errors\n",
+			smp_processor_id(), mce->bank, bank->ce_count);
+	mce_panic("Too many corrected errors", NULL, NULL);
+}
+
 void mce_track_storm(struct mce *mce)
 {
 	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);

-- 
2.53.0-Meta

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

On Fri, Aug 21, 2026 at 03:24:08AM -0700, Breno Leitao wrote:
> A machine check bank producing corrected errors faster than the kernel can
> drain them is not a machine anyone wants to keep in service, but nothing
> takes it out. mce_track_storm() throttles CMCI for the bank and the
> machine stays up.
> 
> Count corrected errors per bank and add mce=panic_on_ce_count=<count>
> and panic the host if we have more events than set.

FYI. I don't think this needs to be fixed, but you should be aware and
perhaps document the shared bank details.

This won't count accurately for banks that are shared by multiple logical
CPUs (you've inherited this from the storm detection code that introduces
this problem).

E.g. a machine check bank reporting L2 errors is shared by both logical CPUs
on a core on P-core systems, and by all cores on a module on E-core systems.

But the storm code keeps <per-CPU,per-bank> counts. So if an L2 instance
is throwing out many errors, some will be counted by one of the CPUs, while
other errors are counted separately by the other CPUs sharing the bank.

The net effect is that a storm won't be trigged until one of the CPUs tracking
a shared bank hits the threshold.

Similarly there may be more errors logged than you expect before your
panic fires.

-Tony

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Breno Leitao]

On Fri, Aug 21, 2026 at 09:50:19AM -0700, Luck, Tony wrote:

> > Count corrected errors per bank and add mce=panic_on_ce_count=<count>
> > and panic the host if we have more events than set.
> 
> FYI. I don't think this needs to be fixed, but you should be aware and
> perhaps document the shared bank details.
> 
> This won't count accurately for banks that are shared by multiple logical
> CPUs (you've inherited this from the storm detection code that introduces
> this problem).
> 
> E.g. a machine check bank reporting L2 errors is shared by both logical CPUs
> on a core on P-core systems, and by all cores on a module on E-core systems.

Good point, thanks. I had not thought about the shared bank case.

I will document it in v2, roughly like this under panic_on_ce_count=
in kernel-parameters.txt:

        The count is kept per CPU and per bank. A bank shared by
        several logical CPUs -- an L2 bank is shared by the SMT
	siblings, for instance -- has its errors split across those
	CPUs.

plus a line in the changelog noting that the count inherits the
<per-CPU,per-bank> granularity of the storm tracking. Thus, there may be
more errors logged than you expect before your panic fires. Would it be better?

One question while I am here: is the approach itself something you are
willing to take, or, just a just send patch [1/2]?

Thanks for the review,
--breno

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

On Mon, Aug 24, 2026 at 01:29:18AM -0700, Breno Leitao wrote:
> On Fri, Aug 21, 2026 at 09:50:19AM -0700, Luck, Tony wrote:
> 
> > > Count corrected errors per bank and add mce=panic_on_ce_count=<count>
> > > and panic the host if we have more events than set.
> > 
> > FYI. I don't think this needs to be fixed, but you should be aware and
> > perhaps document the shared bank details.
> > 
> > This won't count accurately for banks that are shared by multiple logical
> > CPUs (you've inherited this from the storm detection code that introduces
> > this problem).
> > 
> > E.g. a machine check bank reporting L2 errors is shared by both logical CPUs
> > on a core on P-core systems, and by all cores on a module on E-core systems.
> 
> Good point, thanks. I had not thought about the shared bank case.
> 
> I will document it in v2, roughly like this under panic_on_ce_count=
> in kernel-parameters.txt:
> 
>         The count is kept per CPU and per bank. A bank shared by
>         several logical CPUs -- an L2 bank is shared by the SMT
> 	siblings, for instance -- has its errors split across those
> 	CPUs.
> 
> plus a line in the changelog noting that the count inherits the
> <per-CPU,per-bank> granularity of the storm tracking. Thus, there may be
> more errors logged than you expect before your panic fires. Would it be better?

Yes. Documentation like this in the commit, and under Documentation/*
gives fair notice to users.

> One question while I am here: is the approach itself something you are
> willing to take, or, just a just send patch [1/2]?

Boris is the maintainer here. So he has the final decision. My
opinion is that this is a bit niche for the kernel. Maybe this
could be handled by some user agent (mcelog? rasdaemon?) to
raise awareness to system operators about high rates of corrected
errors.
> 
> Thanks for the review,
> --breno

-Tony

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

On Mon, Aug 24, 2026 at 09:31:02AM -0700, Luck, Tony wrote:
> On Mon, Aug 24, 2026 at 01:29:18AM -0700, Breno Leitao wrote:
> > On Fri, Aug 21, 2026 at 09:50:19AM -0700, Luck, Tony wrote:
> > 
> > > > Count corrected errors per bank and add mce=panic_on_ce_count=<count>
> > > > and panic the host if we have more events than set.
> > > 
> > > FYI. I don't think this needs to be fixed, but you should be aware and
> > > perhaps document the shared bank details.
> > > 
> > > This won't count accurately for banks that are shared by multiple logical
> > > CPUs (you've inherited this from the storm detection code that introduces
> > > this problem).
> > > 
> > > E.g. a machine check bank reporting L2 errors is shared by both logical CPUs
> > > on a core on P-core systems, and by all cores on a module on E-core systems.
> > 
> > Good point, thanks. I had not thought about the shared bank case.
> > 
> > I will document it in v2, roughly like this under panic_on_ce_count=
> > in kernel-parameters.txt:
> > 
> >         The count is kept per CPU and per bank. A bank shared by
> >         several logical CPUs -- an L2 bank is shared by the SMT
> > 	siblings, for instance -- has its errors split across those
> > 	CPUs.
> > 
> > plus a line in the changelog noting that the count inherits the
> > <per-CPU,per-bank> granularity of the storm tracking. Thus, there may be
> > more errors logged than you expect before your panic fires. Would it be better?
> 
> Yes. Documentation like this in the commit, and under Documentation/*
> gives fair notice to users.
> 
> > One question while I am here: is the approach itself something you are
> > willing to take, or, just a just send patch [1/2]?
> 
> Boris is the maintainer here. So he has the final decision. My
> opinion is that this is a bit niche for the kernel. Maybe this
> could be handled by some user agent (mcelog? rasdaemon?) to
> raise awareness to system operators about high rates of corrected
> errors.

I chatted with someone internally. They'd be very interested in being
able to see these running counts via some /sys files.

Perhaps your use case would be better for that too? Rather than:

"Everything is fine"
...
"Everything is fine"
...
"Everything is fine"
...
"Everything is fine"
...
"Count exceeded, PANIC!"

> > Thanks for the review,
> > --breno
> 
> -Tony

-Tony
> 

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Breno Leitao]

On Mon, Aug 24, 2026 at 01:20:20PM -0700, Luck, Tony wrote:
> > > One question while I am here: is the approach itself something you are
> > > willing to take, or, just a just send patch [1/2]?
> > 
> > Boris is the maintainer here. So he has the final decision. My
> > opinion is that this is a bit niche for the kernel. Maybe this
> > could be handled by some user agent (mcelog? rasdaemon?) to
> > raise awareness to system operators about high rates of corrected
> > errors.
> 
> I chatted with someone internally. They'd be very interested in being
> able to see these running counts via some /sys files.
> 
> Perhaps your use case would be better for that too? Rather than:
> 
> "Everything is fine"

Sounds good, I will restructure v2 that way: export the running counts
first, and add the panic on top as a separate patch.

The count is already kept per CPU and per bank in struct storm_bank, so
the natural fit is one read-only file per bank, next to the existing
bank<N> knobs:

      /sys/devices/system/machinecheck/machinecheckX/ce_count<N>

Is this the right approach?

Thanks
--breno

RE: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

> > I chatted with someone internally. They'd be very interested in being
> > able to see these running counts via some /sys files.
> >
> > Perhaps your use case would be better for that too? Rather than:
> >
> > "Everything is fine"
>
> Sounds good, I will restructure v2 that way: export the running counts
> first, and add the panic on top as a separate patch.
>
> The count is already kept per CPU and per bank in struct storm_bank, so
> the natural fit is one read-only file per bank, next to the existing
> bank<N> knobs:
>
>       /sys/devices/system/machinecheck/machinecheckX/ce_count<N>
>
> Is this the right approach?

Breno,

Yes, that looks like a natural place to expose the running counts.

I'd argue that the files should be read/write. My rationale is that corrected
error counts may be increasing slowly over time for transient errors caused
by particle strikes. If a system runs for many months, it might reach the
threshold and trigger a panic. Allowing the system administrator to zero
the counts would avoid this.

-Tony

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Borislav Petkov]

On Tue, Aug 25, 2026 at 03:27:55PM +0000, Luck, Tony wrote:
> > > I chatted with someone internally. They'd be very interested in being
> > > able to see these running counts via some /sys files.
> > >
> > > Perhaps your use case would be better for that too? Rather than:
> > >
> > > "Everything is fine"
> >
> > Sounds good, I will restructure v2 that way: export the running counts
> > first, and add the panic on top as a separate patch.
> >
> > The count is already kept per CPU and per bank in struct storm_bank, so
> > the natural fit is one read-only file per bank, next to the existing
> > bank<N> knobs:
> >
> >       /sys/devices/system/machinecheck/machinecheckX/ce_count<N>
> >
> > Is this the right approach?
> 
> Breno,
> 
> Yes, that looks like a natural place to expose the running counts.
> 
> I'd argue that the files should be read/write. My rationale is that corrected
> error counts may be increasing slowly over time for transient errors caused
> by particle strikes. If a system runs for many months, it might reach the
> threshold and trigger a panic. Allowing the system administrator to zero
> the counts would avoid this.

How are you going to handle this on a large fleet? Surely not log into every
box. IOW, if anything this should be automatic and not a sysfs knob.

Why isn't this whole effort part of the drivers/ras/cec.c thing where we can
do all kinds of configurable policy and so on and which is exactly for things
like shutting up banks and not overflowing dmesg and memory_failure-ing pages
and so on?

Why isn't the strategy here page offlining and shutting up the source of the
error instead of doing silly counting and not doing anything to contain the
errors in the first place?

Weird...

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

> Why isn't the strategy here page offlining and shutting up the source of the
> error instead of doing silly counting and not doing anything to contain the
> errors in the first place?

While Breno wasn't specific about the source of the errors in these messages, I'm
guessing that they might be coming from cache errors rather than DDR memory.

There isn't a good way for software[1] to suppress these errors. Taking memory
pages offline when the problem is the cache will just deplete available memory
without solving the problem.

-Tony

[1] I'll be posting something later today that adds a driver to help with a hardware
solution to correctable cache errors.

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Borislav Petkov]

On Tue, Aug 25, 2026 at 04:26:26PM +0000, Luck, Tony wrote:
> > Why isn't the strategy here page offlining and shutting up the source of the
> > error instead of doing silly counting and not doing anything to contain the
> > errors in the first place?
> 
> While Breno wasn't specific about the source of the errors in these messages, I'm
> guessing that they might be coming from cache errors rather than DDR memory.
> 
> There isn't a good way for software[1] to suppress these errors. Taking memory
> pages offline when the problem is the cache will just deplete available memory
> without solving the problem.

I have been thinking about this *years* ago. If it is cache errors, we should
simply offline the core or cores using that cache. We have a lot of cores
nowadays :)

In general, us being a lot more resilient and applying automatic containment
and recovery actions should be the goal IMO.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

RE: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

> > > Why isn't the strategy here page offlining and shutting up the source of the
> > > error instead of doing silly counting and not doing anything to contain the
> > > errors in the first place?
> >
> > While Breno wasn't specific about the source of the errors in these messages, I'm
> > guessing that they might be coming from cache errors rather than DDR memory.
> >
> > There isn't a good way for software[1] to suppress these errors. Taking memory
> > pages offline when the problem is the cache will just deplete available memory
> > without solving the problem.
>
> I have been thinking about this *years* ago. If it is cache errors, we should
> simply offline the core or cores using that cache. We have a lot of cores
> nowadays :)

You aren't the only one. Andi Kleen built this exact idea into mcelog in 2009:

https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git/commit/?id=ffd10014622d20eb08fd35b03f756966f80a08bd

This can work well for L1/L2 cache errors. Perhaps also for L3 on modern AMD
CPUs where only small fraction of cores share each L3 cache instance.

L3 on Intel is shared by the whole socket. So you'd lose 50% of cores for an L3 cache
issue on a typical two socket system (plus we'd have to bring back offline of CPU 0
if you want this to work for socket 0).

> In general, us being a lot more resilient and applying automatic containment
> and recovery actions should be the goal IMO.

Taking cores offline likely needs a bunch more plumbing outside of the kernel.
Bare metal systems sometime isolate critical workloads on specific cores. VMM
systems may bind guests to specific cores to provide consistent performance.
Maybe there are already some udev events that could be used to trigger actions
when cores go away?

-Tony

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Borislav Petkov]

On Tue, Aug 25, 2026 at 08:01:23PM +0000, Luck, Tony wrote:
> L3 on Intel is shared by the whole socket. So you'd lose 50% of cores for an L3 cache
> issue on a typical two socket system

Would panicking the whole system be better?

> (plus we'd have to bring back offline of CPU 0 if you want this to work for
> socket 0).

As long as you offline whatever you can and cordon off the accesses to the
faulty area as much as possible...

> Taking cores offline likely needs a bunch more plumbing outside of the kernel.
> Bare metal systems sometime isolate critical workloads on specific cores. VMM
> systems may bind guests to specific cores to provide consistent performance.

Well, nothing's free, right? If you want to run with degraded performance, you
should put that into the list of testing scenarios.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

On Fri, Aug 21, 2026 at 03:24:08AM -0700, Breno Leitao wrote:
> +/*
> + * A bank that keeps reporting corrected errors is repairing them faster than
> + * anything acts on it. Count them and let the admin put a ceiling on it.
> + */
> +void mce_track_ce_count(struct mce *mce)
> +{
> +	struct storm_bank *bank = &this_cpu_ptr(&storm_desc)->banks[mce->bank];
> +	int limit = READ_ONCE(mca_cfg.panic_on_ce_count);
> +
> +	if (limit <= 0)
> +		return;
> +
> +	if (!(mce->status & MCI_STATUS_VAL) || !mce_is_correctable(mce))
> +		return;
> +
> +	if (++bank->ce_count < (u64)limit)

If errors are happening faster than Linux can service CMCI interrupts,
then the corrected error count in bits {52:38} will be some number
bigger than "1".[*] You should add that number here, rather than simply
incrementing +bank->ce_count.

> +		return;
> +
> +	printk_deferred(KERN_EMERG "CPU%d BANK%d logged %llu corrected errors\n",
> +			smp_processor_id(), mce->bank, bank->ce_count);
> +	mce_panic("Too many corrected errors", NULL, NULL);
> +}
> +
>  void mce_track_storm(struct mce *mce)
>  {
>  	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
> 
> -- 
> 2.53.0-Meta

-Tony

[*] My personal record is several hundred. But that was on a debug
system that had been poked to signal an error on every cache access!