[PATCH RESEND v2] x86/mce: Rate-limit the CMCI storm transition notices

[PATCH RESEND v2] x86/mce: Rate-limit the CMCI storm transition notices

[Breno Leitao]

mce_track_storm() prints a line every time a bank enters storm mode and
another every time it leaves. Neither is bounded, so the log volume
follows whatever rate the bank flaps at.

One host in our fleet logged 9970726 "CMCI storm detected" and 9970734
"CMCI storm subsided" lines for a single bank over four days, which is
a bit excessive for all monitoring purposes.

Put both transitions behind one ratelimit.

Signed-off-by: Breno Leitao <leitao@debian.org>
Acked-by: Tony Luck <tony.luck@intel.com>
---
Changes in v2:
- rebased on linux-next
- Link to v1: https://patch.msgid.link/20260827-mci_ratelimit-v1-1-ac43df769b36@debian.org
---
 arch/x86/kernel/cpu/mce/threshold.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mce/threshold.c b/arch/x86/kernel/cpu/mce/threshold.c
index 6c370d5af5bd23..8591715ce24306 100644
--- a/arch/x86/kernel/cpu/mce/threshold.c
+++ b/arch/x86/kernel/cpu/mce/threshold.c
@@ -4,6 +4,7 @@
  */
 #include <linux/interrupt.h>
 #include <linux/kernel.h>
+#include <linux/ratelimit.h>
 
 #include <asm/irq_vectors.h>
 #include <asm/traps.h>
@@ -111,6 +112,13 @@ void cmci_storm_end(unsigned int bank)
 		mce_timer_kick(false);
 }
 
+/*
+ * Shared by both transitions so that a bank flapping between them cannot
+ * outrun the console.
+ */
+static DEFINE_RATELIMIT_STATE(storm_rs, DEFAULT_RATELIMIT_INTERVAL,
+			      DEFAULT_RATELIMIT_BURST);
+
 void mce_track_storm(struct mce *mce)
 {
 	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
@@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
 	if (storm->banks[mce->bank].in_storm_mode) {
 		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, false);
 		cmci_storm_end(mce->bank);
 	} else {
 		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
 			return;
-		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
+		if (__ratelimit(&storm_rs))
+			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
+					smp_processor_id(), mce->bank);
 		mce_handle_storm(mce->bank, true);
 		cmci_storm_begin(mce->bank);
 	}

---
base-commit: aaa2f954c21b205a0bcd0961e5b2be4e82e58455
base-tree: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
change-id: 20260827-mci_ratelimit-064421e2d8cd

Best regards,
--  
Breno Leitao <leitao@debian.org>

Re: [PATCH RESEND v2] x86/mce: Rate-limit the CMCI storm transition notices

[Borislav Petkov]

On Thu, Nov 12, 2026 at 01:56:39AM -0800, Breno Leitao wrote:
>  void mce_track_storm(struct mce *mce)
>  {
>  	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
> @@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
>  	if (storm->banks[mce->bank].in_storm_mode) {
>  		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
>  			return;
> -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
> +		if (__ratelimit(&storm_rs))
> +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
> +					smp_processor_id(), mce->bank);
>  		mce_handle_storm(mce->bank, false);
>  		cmci_storm_end(mce->bank);
>  	} else {
>  		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
>  			return;
> -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
> +		if (__ratelimit(&storm_rs))
> +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
> +					smp_processor_id(), mce->bank);
>  		mce_handle_storm(mce->bank, true);
>  		cmci_storm_begin(mce->bank);
>  	}

Should we merge all the MCE ratelimiting into a single effort?

https://lore.kernel.org/r/20261108143751.3af49e48@fangorn

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH RESEND v2] x86/mce: Rate-limit the CMCI storm transition notices

[Breno Leitao]

On Thu, Nov 12, 2026 at 10:21:25AM -0800, Borislav Petkov wrote:
> On Thu, Nov 12, 2026 at 01:56:39AM -0800, Breno Leitao wrote:
> >  void mce_track_storm(struct mce *mce)
> >  {
> >  	struct mca_storm_desc *storm = this_cpu_ptr(&storm_desc);
> > @@ -150,13 +158,17 @@ void mce_track_storm(struct mce *mce)
> >  	if (storm->banks[mce->bank].in_storm_mode) {
> >  		if (history & GENMASK_ULL(STORM_END_POLL_THRESHOLD, 0))
> >  			return;
> > -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n", smp_processor_id(), mce->bank);
> > +		if (__ratelimit(&storm_rs))
> > +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm subsided\n",
> > +					smp_processor_id(), mce->bank);
> >  		mce_handle_storm(mce->bank, false);
> >  		cmci_storm_end(mce->bank);
> >  	} else {
> >  		if (hweight64(history) < STORM_BEGIN_THRESHOLD)
> >  			return;
> > -		printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n", smp_processor_id(), mce->bank);
> > +		if (__ratelimit(&storm_rs))
> > +			printk_deferred(KERN_NOTICE "CPU%d BANK%d CMCI storm detected\n",
> > +					smp_processor_id(), mce->bank);
> >  		mce_handle_storm(mce->bank, true);
> >  		cmci_storm_begin(mce->bank);
> >  	}
> 
> Should we merge all the MCE ratelimiting into a single effort?
> 
> https://lore.kernel.org/r/20261108143751.3af49e48@fangorn

Sounds good. I will send v3 as one series: Rik's patch first, since it
is a stable fix and can go in on its own, and the storm notice ratelimit
on top of it.