[PATCH 0/2] mm/memory-failure: add panic option for unrecoverable pages

[PATCH 0/2] mm/memory-failure: add panic option for unrecoverable pages

[Breno Leitao]

When the memory failure handler encounters an in-use kernel page that it
cannot recover (slab, page tables, kernel stacks, vmalloc, etc.), it
currently logs the error as "Ignored" and continues operation.

This leaves corrupted data accessible to the kernel, which will inevitably
cause either silent data corruption or a delayed crash when the poisoned memory
is next accessed.

This is a common problem on large fleets. We frequently observe multi-bit ECC
errors hitting kernel slab pages, where memory_failure() fails to recover them
and the system crashes later at an unrelated code path, making root cause
analysis unnecessarily difficult.

Here is one specific example from production on an arm64 server: a multi-bit
ECC error hit a dentry cache slab page, memory_failure() failed to recover it
(slab pages are not supported by the hwpoison recovery mechanism), and 67
seconds later d_lookup() accessed the poisoned cache line causing a synchronous
external abort:

    [88690.479680] [Hardware Error]: error_type: 3, multi-bit ECC
    [88690.498473] Memory failure: 0x40272d: unhandlable page.
    [88690.498619] Memory failure: 0x40272d: recovery action for
                   get hwpoison page: Ignored
    ...
    [88757.847126] Internal error: synchronous external abort:
                   0000000096000410 [#1] SMP
    [88758.061075] pc : d_lookup+0x5c/0x220

This series adds a new sysctl vm.panic_on_unrecoverable_memory_failure
(default 0) that, when enabled, panics immediately on unrecoverable
memory failures. This provides a clean crash dump at the time of the
error, which is far more useful for diagnosis than a random crash later
at an unrelated code path.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (2):
      mm/memory-failure: add panic_on_unrecoverable_memory_failure sysctl
      Documentation: document panic_on_unrecoverable_memory_failure sysctl

 Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
 mm/memory-failure.c                     | 15 +++++++++++++++
 2 files changed, 42 insertions(+)
---
base-commit: 63f5f5ffdf63d9c75a438c92be58177744b4c69c
change-id: 20260323-ecc_panic-4e473b83087c

Best regards,
--  
Breno Leitao <leitao@debian.org>

[PATCH 1/2] mm/memory-failure: add panic_on_unrecoverable_memory_failure sysctl

[Breno Leitao]

When memory_failure() encounters an in-use kernel page that cannot be
recovered (slab, page tables, kernel stacks, reserved, vmalloc, etc.),
it currently logs MF_IGNORED and continues. This leaves corrupted data
accessible to the kernel, risking silent data corruption or a delayed
crash when the poisoned cache line is next accessed.

For example, a multi-bit ECC error on a dentry cache slab page was
ignored by memory_failure(), and 67 seconds later d_lookup() accessed
the poisoned cache line, causing a synchronous external abort:

  [88690.479680] [Hardware Error]: error_type: 3, multi-bit ECC
  [88690.498473] Memory failure: 0x40272d: unhandlable page.
  [88690.498619] Memory failure: 0x40272d: recovery action for
                 get hwpoison page: Ignored
  ...
  [88757.847126] Internal error: synchronous external abort:
                 0000000096000410 [#1] SMP
  [88758.061075] pc : d_lookup+0x5c/0x220

Add a new sysctl vm.panic_on_unrecoverable_memory_failure (default 0)
that, when set to 1, panics immediately on unrecoverable memory
failures. This provides a clean crash dump at the time of the error
rather than a delayed crash with potential silent corruption in between.

The panic is placed in action_result() so that all call sites that log
MF_MSG_GET_HWPOISON with MF_IGNORED are covered, including the hugetlb
path in try_memory_failure_hugetlb().

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 mm/memory-failure.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index ee42d43613097..25bd043497195 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -74,6 +74,8 @@ static int sysctl_memory_failure_recovery __read_mostly = 1;
 
 static int sysctl_enable_soft_offline __read_mostly = 1;
 
+static int sysctl_panic_on_unrecoverable_mf __read_mostly;
+
 atomic_long_t num_poisoned_pages __read_mostly = ATOMIC_LONG_INIT(0);
 
 static bool hw_memory_failure __read_mostly = false;
@@ -155,6 +157,15 @@ static const struct ctl_table memory_failure_table[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= SYSCTL_ONE,
+	},
+	{
+		.procname	= "panic_on_unrecoverable_memory_failure",
+		.data		= &sysctl_panic_on_unrecoverable_mf,
+		.maxlen		= sizeof(sysctl_panic_on_unrecoverable_mf),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE,
 	}
 };
 
@@ -1298,6 +1309,10 @@ static int action_result(unsigned long pfn, enum mf_action_page_type type,
 	pr_err("%#lx: recovery action for %s: %s\n",
 		pfn, action_page_types[type], action_name[result]);
 
+	if (sysctl_panic_on_unrecoverable_mf &&
+	    type == MF_MSG_GET_HWPOISON && result == MF_IGNORED)
+		panic("Memory failure: %#lx: unrecoverable page", pfn);
+
 	return (result == MF_RECOVERED || result == MF_DELAYED) ? 0 : -EBUSY;
 }
 

-- 
2.52.0

[PATCH 2/2] Documentation: document panic_on_unrecoverable_memory_failure sysctl

[Breno Leitao]

Document the new vm.panic_on_unrecoverable_memory_failure sysctl in the
admin guide, following the same format as panic_on_unrecovered_nmi.

Signed-off-by: Breno Leitao <leitao@debian.org>
---
 Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
index 97e12359775c9..3310fb8272fb9 100644
--- a/Documentation/admin-guide/sysctl/vm.rst
+++ b/Documentation/admin-guide/sysctl/vm.rst
@@ -67,6 +67,7 @@ Currently, these files are in /proc/sys/vm:
 - page-cluster
 - page_lock_unfairness
 - panic_on_oom
+- panic_on_unrecoverable_memory_failure
 - percpu_pagelist_high_fraction
 - stat_interval
 - stat_refresh
@@ -925,6 +926,32 @@ panic_on_oom=2+kdump gives you very strong tool to investigate
 why oom happens. You can get snapshot.
 
 
+panic_on_unrecoverable_memory_failure
+======================================
+
+When a hardware memory error (e.g. multi-bit ECC) hits an in-use kernel
+page that cannot be recovered by the memory failure handler, the default
+behaviour is to ignore the error and continue operation.  This is
+dangerous because the corrupted data remains accessible to the kernel,
+risking silent data corruption or a delayed crash when the poisoned
+memory is next accessed.
+
+Pages that reach this path include slab objects (dentry cache, inode
+cache, etc.), page tables, kernel stacks, and other kernel allocations
+that lack the reverse mapping needed to isolate all references.
+
+For many environments it is preferable to panic immediately with a clean
+crash dump that captures the original error context, rather than to
+continue and face a random crash later whose cause is difficult to
+diagnose.
+
+= ===================================================================
+0 Try to continue operation (default).
+1 Panic immediately.  If the ``panic`` sysctl is also non-zero then the
+  machine will be rebooted.
+= ===================================================================
+
+
 percpu_pagelist_high_fraction
 =============================
 

-- 
2.52.0

Re: [PATCH 2/2] Documentation: document panic_on_unrecoverable_memory_failure sysctl

[Randy Dunlap]

On 3/23/26 8:29 AM, Breno Leitao wrote:
> Document the new vm.panic_on_unrecoverable_memory_failure sysctl in the
> admin guide, following the same format as panic_on_unrecovered_nmi.
> 
> Signed-off-by: Breno Leitao <leitao@debian.org>
> ---
>  Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
> index 97e12359775c9..3310fb8272fb9 100644
> --- a/Documentation/admin-guide/sysctl/vm.rst
> +++ b/Documentation/admin-guide/sysctl/vm.rst


> +
> += ===================================================================
> +0 Try to continue operation (default).
> +1 Panic immediately.  If the ``panic`` sysctl is also non-zero then the
> +  machine will be rebooted.
> += ===================================================================

The table begin and end lines must be at least as long as the text (may be
longer). Please extend the =========== lines by a few characters.

-- 
~Randy

Re: [PATCH 2/2] Documentation: document panic_on_unrecoverable_memory_failure sysctl

[Breno Leitao]

Hello Randy,

On Mon, Mar 23, 2026 at 09:51:55AM -0700, Randy Dunlap wrote:
> On 3/23/26 8:29 AM, Breno Leitao wrote:
> > Document the new vm.panic_on_unrecoverable_memory_failure sysctl in the
> > admin guide, following the same format as panic_on_unrecovered_nmi.
> >
> > Signed-off-by: Breno Leitao <leitao@debian.org>
> > ---
> >  Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
> >  1 file changed, 27 insertions(+)
> >
> > diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
> > index 97e12359775c9..3310fb8272fb9 100644
> > --- a/Documentation/admin-guide/sysctl/vm.rst
> > +++ b/Documentation/admin-guide/sysctl/vm.rst
>
>
> > +
> > += ===================================================================
> > +0 Try to continue operation (default).
> > +1 Panic immediately.  If the ``panic`` sysctl is also non-zero then the
> > +  machine will be rebooted.
> > += ===================================================================
>
> The table begin and end lines must be at least as long as the text (may be
> longer). Please extend the =========== lines by a few characters.

The HTML renders correctly in Sphinx (likely due to automatic column
expansion), but I agree the raw table format should be properly
structured.

I'll send v2 with this corrected.

Thanks for the review,
--breno

Re: [PATCH 2/2] Documentation: document panic_on_unrecoverable_memory_failure sysctl

[Akira Yokosawa]

Hi,

On Tue, 24 Mar 2026 03:09:25 -0700, Breno Leitao wrote:
> Hello Randy,
> 
> On Mon, Mar 23, 2026 at 09:51:55AM -0700, Randy Dunlap wrote:
>> On 3/23/26 8:29 AM, Breno Leitao wrote:
>> > Document the new vm.panic_on_unrecoverable_memory_failure sysctl in the
>> > admin guide, following the same format as panic_on_unrecovered_nmi.
>> >
>> > Signed-off-by: Breno Leitao <leitao@debian.org>
>> > ---
>> >  Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
>> >  1 file changed, 27 insertions(+)
>> >
>> > diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
>> > index 97e12359775c9..3310fb8272fb9 100644
>> > --- a/Documentation/admin-guide/sysctl/vm.rst
>> > +++ b/Documentation/admin-guide/sysctl/vm.rst
>>
>>
>> > +
>> > += ===================================================================
>> > +0 Try to continue operation (default).
>> > +1 Panic immediately.  If the ``panic`` sysctl is also non-zero then the
>> > +  machine will be rebooted.
>> > += ===================================================================
>>
>> The table begin and end lines must be at least as long as the text (may be
>> longer). Please extend the =========== lines by a few characters.
> 
> The HTML renders correctly in Sphinx (likely due to automatic column
> expansion), but I agree the raw table format should be properly
> structured.

Just to be clear, Sphinx is behaving as expected here.

The table is in the form of so-called "simple tables" in the reST
(or docutils) parlance.  The rightmost column can exceed the width
indicated by "==========".

Quote from [1]:

   The rightmost column is unbounded; text may continue past the
   edge of the table (as indicated by the table borders). However,
   it is recommended that borders be made long enough to contain
   the entire text.

[1]: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#simple-tables

So, it's just a recommendation, rather than a requirement.

"Grid tables" have a stricter rule.

Hope this helps.

Regards,
Akira

> 
> I'll send v2 with this corrected.
> 
> Thanks for the review,
> --breno

Re: [PATCH 2/2] Documentation: document panic_on_unrecoverable_memory_failure sysctl

[Randy Dunlap]

On 3/24/26 4:48 AM, Akira Yokosawa wrote:
> Hi,
> 
> On Tue, 24 Mar 2026 03:09:25 -0700, Breno Leitao wrote:
>> Hello Randy,
>>
>> On Mon, Mar 23, 2026 at 09:51:55AM -0700, Randy Dunlap wrote:
>>> On 3/23/26 8:29 AM, Breno Leitao wrote:
>>>> Document the new vm.panic_on_unrecoverable_memory_failure sysctl in the
>>>> admin guide, following the same format as panic_on_unrecovered_nmi.
>>>>
>>>> Signed-off-by: Breno Leitao <leitao@debian.org>
>>>> ---
>>>>  Documentation/admin-guide/sysctl/vm.rst | 27 +++++++++++++++++++++++++++
>>>>  1 file changed, 27 insertions(+)
>>>>
>>>> diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst
>>>> index 97e12359775c9..3310fb8272fb9 100644
>>>> --- a/Documentation/admin-guide/sysctl/vm.rst
>>>> +++ b/Documentation/admin-guide/sysctl/vm.rst
>>>
>>>
>>>> +
>>>> += ===================================================================
>>>> +0 Try to continue operation (default).
>>>> +1 Panic immediately.  If the ``panic`` sysctl is also non-zero then the
>>>> +  machine will be rebooted.
>>>> += ===================================================================
>>>
>>> The table begin and end lines must be at least as long as the text (may be
>>> longer). Please extend the =========== lines by a few characters.
>>
>> The HTML renders correctly in Sphinx (likely due to automatic column
>> expansion), but I agree the raw table format should be properly
>> structured.
> 
> Just to be clear, Sphinx is behaving as expected here.
> 
> The table is in the form of so-called "simple tables" in the reST
> (or docutils) parlance.  The rightmost column can exceed the width
> indicated by "==========".
> 
> Quote from [1]:
> 
>    The rightmost column is unbounded; text may continue past the
>    edge of the table (as indicated by the table borders). However,
>    it is recommended that borders be made long enough to contain
>    the entire text.
> 
> [1]: https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#simple-tables
> 
> So, it's just a recommendation, rather than a requirement.
> 
> "Grid tables" have a stricter rule.
> 
> Hope this helps.
> 
> Regards,
> Akira
> 
>>
>> I'll send v2 with this corrected.
>>
>> Thanks for the review,
>> --breno

Thanks to both of you for the edumcation.

-- 
~Randy