Re: [PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

Re: [PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

[Baoquan He]

On 07/31/26 at 05:46pm, Jan Sebastian Götte wrote:
> I'm using linux on an embedded target in a Hardware Security Module-like
> application. One requirement is that I want the system to be able to
> quickly erase its memory when it detects physical tampering. I'm
> approaching that by using kdump to load into a small payload that
> instead of dumping RAM, erases RAM from start to end. However, writing
> all of RAM, especially on an embedded target, is rather slow. For this
> reason, I propose the mechanism in this patch series:
> 
> Add CONFIG_CRASH_ZEROIZE (default off), which when enabled makes various
> subsystems handling secret data do a quick, targeted wipe of these
> secrets before kdump. This behavior might also be interesting in cases
> where you run a normal kdump kernel but you still want to keep things
> like fde crypto keys out of these dumps.

Please check below patchset, you both seem to have the similar
requirement. Please go there to discuss.

[RFC PATCH 0/4] panic: a pre kdump notifier list for hypervisor upcalls

Note that we usually dont' want to run a lot of work after panic and
before jumping into kdump kernel.

> 
> CONFIG_CRASH_ZEROIZE is a best effort, defense in depth solution. There
> are circumstances, such as when a panic is triggered after memory
> corruption, or when a panic interrupts some operation that mutates data
> structures under locks, when the kernel cannot safely wipe some memory
> areas. The handlers proposed in this series will just print a warning
> and skip the affected areas in this case.
> 
> This series introduces two handlers as a starting point: One for kernel
> keyrings, and one for secretmem. Future places where such handlers could
> be added would be for example drivers for crypto accelerators. 
> 
> The patch series applies on top of linux-next but should work on 7.0.0,
> too. I've tested the patches on a Arduino uno Q (Qualcomm QRB2210)
> embedded target.
> 
> Jan Sebastian Götte (4):
>   of/kexec: fix typo in comment (usable-memory-range)
>   kexec: add CRASH_ZEROIZE to wipe secrets before kdump
>   mm/secretmem: zeroize secret pages before kdump
>   security/keys: zeroize key payloads before kdump
> 
>  drivers/of/kexec.c                        |  2 +-
>  include/linux/crash_core.h                |  5 +++
>  include/linux/key-type.h                  |  9 ++++
>  kernel/Kconfig.kexec                      |  8 ++++
>  kernel/crash_core.c                       | 18 ++++++++
>  mm/secretmem.c                            | 50 +++++++++++++++++++++++
>  security/keys/big_key.c                   | 15 +++++++
>  security/keys/encrypted-keys/encrypted.c  | 12 ++++++
>  security/keys/key.c                       | 44 ++++++++++++++++++++
>  security/keys/trusted-keys/trusted_core.c | 14 +++++++
>  security/keys/user_defined.c              | 11 +++++
>  11 files changed, 187 insertions(+), 1 deletion(-)
> 
> -- 
> 2.53.0
> 

Re: [PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

[Jan Sebastian Götte]

Hi there,

On 8/1/26 16:03, Baoquan He wrote:
> On 07/31/26 at 05:46pm, Jan Sebastian Götte wrote:
>> Add CONFIG_CRASH_ZEROIZE (default off), which when enabled makes various
>> subsystems handling secret data do a quick, targeted wipe of these
>> secrets before kdump. This behavior might also be interesting in cases
>> where you run a normal kdump kernel but you still want to keep things
>> like fde crypto keys out of these dumps.
> 
> Please check below patchset, you both seem to have the similar
> requirement. Please go there to discuss.
> 
> [RFC PATCH 0/4] panic: a pre kdump notifier list for hypervisor upcalls

Thank you for the pointer. That's indeed similar, but I think this 
patchset here makes sense as an independent patchset.

* The two notifier chains run at different points, and theirs runs 
inside panic independent of kdump. This one here has to be run after 
machine_crash_shutdown(), and so makes most sense inside 
__crash_kexec(). The wipe code in this patchset must be one of the last 
things to run before the kexec since the wiped data structures could 
easily lead to problems if something tried using them later.

* I think this patch series here is a bit cleaner. I don't think this 
needs a custom notifier implementation.

* Since there's lots of places that may need wiping after crash, I think 
it makes sense to have an explicit notifier list for that purpose, and 
to not mix these with other things like hypercalls. I think ordering is 
important here: The wiping should be the last thing before the kexec 
jump. Having it on a generic notifier chain risks that later, other 
callbacks get added that when interleaved could cause problems.

* Since a good fraction of users may not care about wiping secrets, I 
think it should be gated behind an explicit enable setting.

> Note that we usually dont' want to run a lot of work after panic and
> before jumping into kdump kernel.

I understand. For this reason, I think it's best to keep this 
default-off. As-is, the notifier list call is timed and on the (slow) 
ARM64 target I'm using, it takes about 3-5 ms to run. I took the "try 
lock, skip if locked" approach to keep the risk of this code crashing 
during panic minimal. In my application, the kdump payload is code that 
then does a full wipe, taking a couple hundred milliseconds.

Let me know what you think.

Re: [PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

[Dave Young]

On 8/2/26 12:31 AM, Jan Sebastian Götte wrote:
> Hi there,
> 
> On 8/1/26 16:03, Baoquan He wrote:
>> On 07/31/26 at 05:46pm, Jan Sebastian Götte wrote:
>>> Add CONFIG_CRASH_ZEROIZE (default off), which when enabled makes various
>>> subsystems handling secret data do a quick, targeted wipe of these
>>> secrets before kdump. This behavior might also be interesting in cases
>>> where you run a normal kdump kernel but you still want to keep things
>>> like fde crypto keys out of these dumps.
>>
>> Please check below patchset, you both seem to have the similar
>> requirement. Please go there to discuss.
>>
>> [RFC PATCH 0/4] panic: a pre kdump notifier list for hypervisor upcalls
> 
> Thank you for the pointer. That's indeed similar, but I think this patchset here makes sense as an independent patchset.
> 
> * The two notifier chains run at different points, and theirs runs inside panic independent of kdump. This one here has to be run after machine_crash_shutdown(), and so makes most sense inside __crash_kexec(). The wipe code in this patchset must be one of the last things to run before the kexec since the wiped data structures could easily lead to problems if something tried using them later.
> 
> * I think this patch series here is a bit cleaner. I don't think this needs a custom notifier implementation.
> 
> * Since there's lots of places that may need wiping after crash, I think it makes sense to have an explicit notifier list for that purpose, and to not mix these with other things like hypercalls. I think ordering is important here: The wiping should be the last thing before the kexec jump. Having it on a generic notifier chain risks that later, other callbacks get added that when interleaved could cause problems.
> 
> * Since a good fraction of users may not care about wiping secrets, I think it should be gated behind an explicit enable setting.
> 
>> Note that we usually dont' want to run a lot of work after panic and
>> before jumping into kdump kernel.
> 
> I understand. For this reason, I think it's best to keep this default-off. As-is, the notifier list call is timed and on the (slow) ARM64 target I'm using, it takes about 3-5 ms to run. I took the "try lock, skip if locked" approach to keep the risk of this code crashing during panic minimal. In my application, the kdump payload is code that then does a full wipe, taking a couple hundred milliseconds.

Not only about the time used, the panicked kernel is not reliable, any more extra logic can make it even not reliable, any pre-kdump extra logic is not a good idea unless it is a must to ensure kdump working.

Cleaning up secret data can be done with makedumpfile + eppic scripts (see the manual of makedumpfile), or it is even possible to do so in kdump kernel with Tao Liu's improvments for makedumpfile previously  (I don't know the status, probably dropped for the time being, but it is possible, cced him).

Thanks
Dave

> 
> Let me know what you think.

Re: [PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

[Jan Sebastian Götte]

On 8/2/26 07:08, Dave Young wrote:
> On 8/2/26 12:31 AM, Jan Sebastian Götte wrote:
>> On 8/1/26 16:03, Baoquan He wrote:
>>> Note that we usually dont' want to run a lot of work after panic and
>>> before jumping into kdump kernel.
>>
>> I understand. For this reason, I think it's best to keep this default-off. As-is, the notifier list call is timed and on the (slow) ARM64 target I'm using, it takes about 3-5 ms to run. I took the "try lock, skip if locked" approach to keep the risk of this code crashing during panic minimal. In my application, the kdump payload is code that then does a full wipe, taking a couple hundred milliseconds.
> 
> Not only about the time used, the panicked kernel is not reliable, any more extra logic can make it even not reliable, any pre-kdump extra logic is not a good idea unless it is a must to ensure kdump working.
> 
> Cleaning up secret data can be done with makedumpfile + eppic scripts (see the manual of makedumpfile), or it is even possible to do so in kdump kernel with Tao Liu's improvments for makedumpfile previously  (I don't know the status, probably dropped for the time being, but it is possible, cced him).

Thank you for the pointer!

There's two scenarios worth considering. First, in the standard scenario 
where you enable this option, then drop into a standard kdump kernel, I 
don't think it makes a big difference *when* you do this cleanup since 
someone is going to have to dereference these pointers. IMHO a good 
reason to do it in the old kernel is that there, the code knows about 
the layout of all the data structures. To retroactively do this in the 
kdump kernel is much more complicated, since there you have to 
reconstruct the structure layouts from symbols or hardcoded struct 
layouts, and you have to keep this symbol/layout information perfectly 
in sync with the running kernel.

The second scenario is what I'm working on here: I'm not using a normal 
kdump kernel, but instead a custom payload that wipes all RAM from start 
to end. This payload will wipe all these keys too, but my critical 
concern is speed: On the embedded SoCs I'm targeting, the full memory 
wipe takes too long (hundreds of ms) for an HSM application, so I want 
to do a targeted wipe of just the keys first. The old kernel I think is 
the natural place to do this. Adding to that, in my scenario the most 
likely trigger of a panic is not something like memory corruption, but a 
trigger of the system's tamper alarms, which would leave the old kernel 
relatively stable during panic.

I can imagine several possible mitigations for the stability concerns 
beyond the default off config option:

* Since the wipe handlers are all really simple, it would be possible to 
manually guard every memory access there to ensure they can't fail and 
that they don't write to sensitive areas like the dump kernel or the 
remaining panic'ing stack. Doing that would only rely on information 
(more or less intact stack pointer, kdump kernel area boundaries) that 
would be necessary for kdump to succeed anyway.

* And/Or I could extend the patchset to include a mechanism similar to 
that in Bradley Morgan's patchset that catches segfaults during wipe, 
and then skips the handler causing the fault.

* A last option would be to have the alive kernel prepare some kind of 
"wipe this first" structure in its memory during normal operation that 
the dump kernel then can pick up to do the actual dirty work. I disfavor 
that since it adds double bookkeeping to a lot of places, some of which 
could be performance critical.

I've also picked up that I should remove the timing logic since that 
could cause instability.

Thanks,
Jan Sebastian

[PATCH 0/4] CRASH_ZEROIZE: Wipe secrets before kdump

[Jan Sebastian Götte]

I'm using linux on an embedded target in a Hardware Security Module-like
application. One requirement is that I want the system to be able to
quickly erase its memory when it detects physical tampering. I'm
approaching that by using kdump to load into a small payload that
instead of dumping RAM, erases RAM from start to end. However, writing
all of RAM, especially on an embedded target, is rather slow. For this
reason, I propose the mechanism in this patch series:

Add CONFIG_CRASH_ZEROIZE (default off), which when enabled makes various
subsystems handling secret data do a quick, targeted wipe of these
secrets before kdump. This behavior might also be interesting in cases
where you run a normal kdump kernel but you still want to keep things
like fde crypto keys out of these dumps.

CONFIG_CRASH_ZEROIZE is a best effort, defense in depth solution. There
are circumstances, such as when a panic is triggered after memory
corruption, or when a panic interrupts some operation that mutates data
structures under locks, when the kernel cannot safely wipe some memory
areas. The handlers proposed in this series will just print a warning
and skip the affected areas in this case.

This series introduces two handlers as a starting point: One for kernel
keyrings, and one for secretmem. Future places where such handlers could
be added would be for example drivers for crypto accelerators. 

The patch series applies on top of linux-next but should work on 7.0.0,
too. I've tested the patches on a Arduino uno Q (Qualcomm QRB2210)
embedded target.

Jan Sebastian Götte (4):
  of/kexec: fix typo in comment (usable-memory-range)
  kexec: add CRASH_ZEROIZE to wipe secrets before kdump
  mm/secretmem: zeroize secret pages before kdump
  security/keys: zeroize key payloads before kdump

 drivers/of/kexec.c                        |  2 +-
 include/linux/crash_core.h                |  5 +++
 include/linux/key-type.h                  |  9 ++++
 kernel/Kconfig.kexec                      |  8 ++++
 kernel/crash_core.c                       | 18 ++++++++
 mm/secretmem.c                            | 50 +++++++++++++++++++++++
 security/keys/big_key.c                   | 15 +++++++
 security/keys/encrypted-keys/encrypted.c  | 12 ++++++
 security/keys/key.c                       | 44 ++++++++++++++++++++
 security/keys/trusted-keys/trusted_core.c | 14 +++++++
 security/keys/user_defined.c              | 11 +++++
 11 files changed, 187 insertions(+), 1 deletion(-)

-- 
2.53.0