[PATCH v3 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[PATCH v3 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[Eric DeVolder]

When the kdump service is loaded, if a CPU or memory is hot
un/plugged, the crash elfcorehdr, which describes the CPUs and memory
in the system, must also be updated, else the resulting vmcore is
inaccurate (eg. missing either CPU context or memory regions).

The current solution utilizes udev (eg. RHEL /usr/lib/udev/rules.d/
98-kexec.rules) to initiate an unload-then-reload of the *entire* kdump
image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by
the userspace kexec utility. This occurrs just so the elfcorehdr can
be updated with the latest list of CPUs and memory regions. In a
previous post I have outlined the significant performance problems
related to offloading this activity to userspace.

With the Linux kernel 6.6 commit below, the kernel now has the ability
to directly modify the elfcorehdr, eliminating the need to
unload-then-reload the entire kdump image when CPU or memory is hot
un/plugged or on/offlined.

 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d6
8b4b6f307d155475cce541f2aee938032ed22e

This kexec-tools patch series is for supporting hotplug with the
kexec_load() syscall; the kernel directly supports hotplug for the
kexec_file_load() syscall, requiring no userspace help.

There are two basic obstacles/requirements for the kexec-tools to
overcome in order to support kernel hotplug rewriting of the
elfcorehdr.

First, the buffer containing the elfcorehdr must be excluded from the
purgatory checksum/digest, which is computed at load time. Otherwise
kernel run-time changes to the elfcorehdr, as a result of hot un/plug,
would result in the checksum failing (specifically in purgatory at
panic kernel boot time), and kdump capture kernel failing to start.
To let the kernel know it is okay to modify the elfcorehdr, kexec
sets the KEXEC_UPDATE_ELFCOREHDR flag.

NOTE: The kernel specifically does *NOT* attempt to recompute the
checksum/digest as that would ultimately require patching the in-
memory purgatory image with the updated checksum. As that purgatory
image is already fully linked, it is binary blob containing no ELF
information which would allow it to be re-linked or patched. Thus
excluding the elfcorehdr from the checksum/digests avoids all these
problems.

Second, the size of the elfcorehdr buffer must be large enough
to accomodate growth of the number of CPUs and/or memory regions.

To satisfy the first requirement, this patch series introduces the
--hotplug option to indicate to kexec-tools that kexec should exclude
the elfcorehdr buffer from the purgatory checksum/digest calculation
and set the KEXEC_UPDATE_ELFCOREHDR flag.

To satisfy the second requirement, the size is obtained from the
/sys/kernel/crash_elfcorehdr_size node (new with the kernel series
cited above).

To use this feature with kexec_load() syscall, invoke kexec with:

 kexec -c --hotplug ...

Thanks!
eric

---
v3: 27sep2023
 - Cite the merged Linux 6.6 commit that supports crash hotplug.
 - Removed the --elfcorehdrsz option, instead using the the
   /sys/kernel/crash_elfcorehdr_size node from the new kernel
   crash hotplug feature.

v2: 3may2023
 http://lists.infradead.org/pipermail/kexec/2023-May/027049.html
 - Setting KEXEC_UPDATE_ELFCOREHDR flag
 - Utilizing /sys/kernel/crash_elfcorehdr_size info.

v1: 20oct2022
 http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
 - Initial patch series

RFC:
 https://lore.kernel.org/lkml/b04ed259-dc5f-7f30-6661-c26f92d9096a@oracle.com/
 s/vmcoreinfo/elfcorehdr/g
---


Eric DeVolder (6):
  kexec: define KEXEC_UPDATE_ELFCOREHDR
  crashdump: introduce the hotplug command line options
  crashdump: setup general hotplug support
  crashdump: exclude elfcorehdr segment from digest for hotplug
  crashdump/x86: identify elfcorehdr segment for hotplug
  crashdump/x86: set the elfcorehdr segment size for hotplug

 kexec/arch/i386/crashdump-x86.c | 11 +++++++++++
 kexec/kexec-syscall.h           |  1 +
 kexec/kexec.8                   |  6 ++++++
 kexec/kexec.c                   | 32 ++++++++++++++++++++++++++++++++
 kexec/kexec.h                   |  8 +++++++-
 5 files changed, 57 insertions(+), 1 deletion(-)

-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 1/6] kexec: define KEXEC_UPDATE_ELFCOREHDR

[Eric DeVolder]

The Linux kernel defines this flag to indicate that the kexec_load()'ed
image is setup so that the kernel may directly modify the elfcorehdr
(and not cause the purgatory digest checksum to fail) in response to
CPU or memory hot un/plug and/or on/offline events.

Define this flag to match/mirror the kernel flag.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec-syscall.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kexec/kexec-syscall.h b/kexec/kexec-syscall.h
index 1e2d12f..2559bff 100644
--- a/kexec/kexec-syscall.h
+++ b/kexec/kexec-syscall.h
@@ -112,6 +112,7 @@ static inline long kexec_file_load(int kernel_fd, int initrd_fd,
 
 #define KEXEC_ON_CRASH		0x00000001
 #define KEXEC_PRESERVE_CONTEXT	0x00000002
+#define KEXEC_UPDATE_ELFCOREHDR	0x00000004
 #define KEXEC_ARCH_MASK		0xffff0000
 
 /* Flags for kexec file based system call */
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 2/6] crashdump: introduce the hotplug command line options

[Eric DeVolder]

Introducing the --hotplug command line option, which is used to
indicate to the kernel that the kdump image is setup to permit
the kernel to directly modify the elfcorehdr in response to CPU
and memory hotplug and/or online/offline events.

This option is only meaningful for kexec_load() syscall. For the
kexec_file_load() syscall, this option is a no-op as the kernel
handles all aspects of loading the kdump image.

This is the command line processing and documentation.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.8 | 6 ++++++
 kexec/kexec.c | 6 ++++++
 kexec/kexec.h | 7 ++++++-
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/kexec/kexec.8 b/kexec/kexec.8
index 3a344c5..4400baf 100644
--- a/kexec/kexec.8
+++ b/kexec/kexec.8
@@ -132,6 +132,12 @@ in one call.
 Open a help file for
 .BR kexec .
 .TP
+.B \-\-hotplug
+Setup for kernel modification of the elfcorehdr. This option performs
+the steps needed to support kernel updates to the elfcorehdr in the
+presence of hot un/plug and/or on/offline events. This option only
+useful for KEXEC_LOAD syscall.
+.TP
 .B \-i\ (\-\-no-checks)
 Fast reboot, no memory integrity checks.
 .TP
diff --git a/kexec/kexec.c b/kexec/kexec.c
index 1edbd34..d790748 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -58,6 +58,8 @@
 
 unsigned long long mem_min = 0;
 unsigned long long mem_max = ULONG_MAX;
+unsigned long elfcorehdrsz = 0;
+int do_hotplug = 0;
 static unsigned long kexec_flags = 0;
 /* Flags for kexec file (fd) based syscall */
 static unsigned long kexec_file_flags = 0;
@@ -1069,6 +1071,7 @@ void usage(void)
 	       "                      back to the compatibility syscall when file based\n"
 	       "                      syscall is not supported or the kernel did not\n"
 	       "                      understand the image (default)\n"
+	       " --hotplug            Setup for kernel modification of elfcorehdr.\n"
 	       " -d, --debug          Enable debugging to help spot a failure.\n"
 	       " -S, --status         Return 1 if the type (by default crash) is loaded,\n"
 	       "                      0 if not.\n"
@@ -1579,6 +1582,9 @@ int main(int argc, char *argv[])
 		case OPT_PRINT_CKR_SIZE:
 			print_crashkernel_region_size();
 			return 0;
+		case OPT_HOTPLUG:
+			do_hotplug = 1;
+			break;
 		default:
 			break;
 		}
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 0933389..487f707 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -232,7 +232,8 @@ extern int file_types;
 #define OPT_PRINT_CKR_SIZE	262
 #define OPT_LOAD_LIVE_UPDATE	263
 #define OPT_EXEC_LIVE_UPDATE	264
-#define OPT_MAX			265
+#define OPT_HOTPLUG		        265
+#define OPT_MAX		266
 #define KEXEC_OPTIONS \
 	{ "help",		0, 0, OPT_HELP }, \
 	{ "version",		0, 0, OPT_VERSION }, \
@@ -259,6 +260,7 @@ extern int file_types;
 	{ "debug",		0, 0, OPT_DEBUG }, \
 	{ "status",		0, 0, OPT_STATUS }, \
 	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
+	{ "hotplug",		    0, 0, OPT_HOTPLUG }, \
 
 #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
 
@@ -297,6 +299,9 @@ extern int ifdown(void);
 extern char purgatory[];
 extern size_t purgatory_size;
 
+extern unsigned long elfcorehdrsz;
+extern int do_hotplug;
+
 #define BOOTLOADER "kexec"
 #define BOOTLOADER_VERSION PACKAGE_VERSION
 
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 3/6] crashdump: setup general hotplug support

[Eric DeVolder]

To allow direct modification of the elfcorehdr by the kernel, in
response to CPU and memory hot un/plug and/or online/offline events,
the following conditions must occur:

 - the elfcorehdr buffer must be excluded from the purgatory
   checksum/digest, and
 - the elfcorehdr segment must be large enough, and
 - the kernel must be notified that it can modify the elfcorehdr

Excluding the elfcorehdr buffer from the digest occurs in patch
"crashdump: exclude elfcorehdr segment from digest for hotplug".
If this is not done, a change to the elfcorehdr will cause the
purgatory check at panic time to fail, and kdump capture kernel
does not start.

For hotplug, the size of the elfcorehdr segment is obtained from the
kernel via the /sys/kernel/crash_elforehdr_size node.

The KEXEC_UPDATE_ELFCOREHDR flag indicates to the kernel that it can
make direct modifications to the elfcorehdr.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/kexec/kexec.c b/kexec/kexec.c
index d790748..0207608 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -1631,6 +1631,24 @@ int main(int argc, char *argv[])
 		die("--load-live-update can only be used with xen\n");
 	}
 
+	/* NOTE: Xen KEXEC_LIVE_UPDATE and KEXEC_UPDATE_ELFCOREHDR collide */
+	if (do_hotplug) {
+		const char *ces = "/sys/kernel/crash_elfcorehdr_size";
+		char *buf, *endptr = NULL;
+		off_t nread = 0;
+		buf = slurp_file_len(ces, sizeof(buf)-1, &nread);
+		if (buf) {
+			if (buf[nread-1] == '\n')
+				buf[nread-1] = '\0';
+			elfcorehdrsz = strtoul(buf, &endptr, 0);
+		}
+		if (!elfcorehdrsz || (endptr && *endptr != '\0'))
+			die("Path %s does not exist, the kernel needs CONFIG_CRASH_HOTPLUG\n", ces);
+		dbgprintf("ELFCOREHDR_SIZE %lu\n", elfcorehdrsz);
+		/* Indicate to the kernel it is ok to modify the elfcorehdr */
+		kexec_flags |= KEXEC_UPDATE_ELFCOREHDR;
+	}
+
 	fileind = optind;
 	/* Reset getopt for the next pass; called in other source modules */
 	opterr = 1;
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 4/6] crashdump: exclude elfcorehdr segment from digest for hotplug

[Eric DeVolder]

To allow direct modification of the elfcorehdr by the kernel, in
response to CPU and memory hot un/plug and/or online/offline events,
the buffer containing the elfcorehdr must be excluded from the
purgatory checksum/digest.

If the elfcorehdr is not excluded from the purgatory checksum/digest,
then at panic time, the checksum/digest check fails (due to the
elfcorehdr having been modified), and the kdump capture kernel does
not start.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.c | 8 ++++++++
 kexec/kexec.h | 1 +
 2 files changed, 9 insertions(+)

diff --git a/kexec/kexec.c b/kexec/kexec.c
index 0207608..fdb4c98 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -689,6 +689,14 @@ static void update_purgatory(struct kexec_info *info)
 		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
 			continue;
 		}
+
+		/* Don't include elfcorehdr in the checksum, if hotplug
+		 * support enabled.
+		 */
+		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
+			continue;
+		}
+
 		sha256_update(&ctx, info->segment[i].buf,
 			      info->segment[i].bufsz);
 		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 487f707..1004aff 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -170,6 +170,7 @@ struct kexec_info {
 	int command_line_len;
 
 	int skip_checks;
+	unsigned long elfcorehdr;
 };
 
 struct arch_map_entry {
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 5/6] crashdump/x86: identify elfcorehdr segment for hotplug

[Eric DeVolder]

Identify the segment containing the elfcorehdr buffer so that
it can be excluded from the purgatory checksum/digest, if hotplug
support is in effect.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/arch/i386/crashdump-x86.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index df1f24c..cb86ca7 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -956,6 +956,9 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
 	} else {
 		memsz = bufsz;
 	}
+
+	/* Record the location of the elfcorehdr for hotplug handling */
+	info->elfcorehdr =
 	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
 							max_addr, -1);
 	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v3 6/6] crashdump/x86: set the elfcorehdr segment size for hotplug

[Eric DeVolder]

For hotplug, the elfcorehdr segment must be sized appropriately
to allow a growing number of CPUs or memory regions. Use the size
reported by the kernel via /sys/kernel/crash_elfcorehdr_sz.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/arch/i386/crashdump-x86.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index cb86ca7..a01031e 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -957,6 +957,14 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
 		memsz = bufsz;
 	}
 
+	/* For hotplug support, override the minimum necessary size just
+	 * computed with the value from /sys/kernel/crash_elfcorehdr_size.
+	 * Properly align the size as well.
+	 */
+	if (do_hotplug) {
+		memsz = _ALIGN(elfcorehdrsz, align);
+	}
+
 	/* Record the location of the elfcorehdr for hotplug handling */
 	info->elfcorehdr =
 	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
-- 
2.39.3


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v3 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[Simon Horman]

On Wed, Sep 27, 2023 at 02:11:30PM -0400, Eric DeVolder wrote:
> When the kdump service is loaded, if a CPU or memory is hot
> un/plugged, the crash elfcorehdr, which describes the CPUs and memory
> in the system, must also be updated, else the resulting vmcore is
> inaccurate (eg. missing either CPU context or memory regions).
> 
> The current solution utilizes udev (eg. RHEL /usr/lib/udev/rules.d/
> 98-kexec.rules) to initiate an unload-then-reload of the *entire* kdump
> image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by
> the userspace kexec utility. This occurrs just so the elfcorehdr can
> be updated with the latest list of CPUs and memory regions. In a
> previous post I have outlined the significant performance problems
> related to offloading this activity to userspace.
> 
> With the Linux kernel 6.6 commit below, the kernel now has the ability
> to directly modify the elfcorehdr, eliminating the need to
> unload-then-reload the entire kdump image when CPU or memory is hot
> un/plugged or on/offlined.
> 
>  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d6
> 8b4b6f307d155475cce541f2aee938032ed22e
> 
> This kexec-tools patch series is for supporting hotplug with the
> kexec_load() syscall; the kernel directly supports hotplug for the
> kexec_file_load() syscall, requiring no userspace help.
> 
> There are two basic obstacles/requirements for the kexec-tools to
> overcome in order to support kernel hotplug rewriting of the
> elfcorehdr.
> 
> First, the buffer containing the elfcorehdr must be excluded from the
> purgatory checksum/digest, which is computed at load time. Otherwise
> kernel run-time changes to the elfcorehdr, as a result of hot un/plug,
> would result in the checksum failing (specifically in purgatory at
> panic kernel boot time), and kdump capture kernel failing to start.
> To let the kernel know it is okay to modify the elfcorehdr, kexec
> sets the KEXEC_UPDATE_ELFCOREHDR flag.
> 
> NOTE: The kernel specifically does *NOT* attempt to recompute the
> checksum/digest as that would ultimately require patching the in-
> memory purgatory image with the updated checksum. As that purgatory
> image is already fully linked, it is binary blob containing no ELF
> information which would allow it to be re-linked or patched. Thus
> excluding the elfcorehdr from the checksum/digests avoids all these
> problems.
> 
> Second, the size of the elfcorehdr buffer must be large enough
> to accomodate growth of the number of CPUs and/or memory regions.
> 
> To satisfy the first requirement, this patch series introduces the
> --hotplug option to indicate to kexec-tools that kexec should exclude
> the elfcorehdr buffer from the purgatory checksum/digest calculation
> and set the KEXEC_UPDATE_ELFCOREHDR flag.
> 
> To satisfy the second requirement, the size is obtained from the
> /sys/kernel/crash_elfcorehdr_size node (new with the kernel series
> cited above).
> 
> To use this feature with kexec_load() syscall, invoke kexec with:
> 
>  kexec -c --hotplug ...
> 
> Thanks!
> eric

Thanks Eric,

applied.

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v3 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[Eric DeVolder]

On 10/4/23 07:08, Simon Horman wrote:
> On Wed, Sep 27, 2023 at 02:11:30PM -0400, Eric DeVolder wrote:
>> When the kdump service is loaded, if a CPU or memory is hot
>> un/plugged, the crash elfcorehdr, which describes the CPUs and memory
>> in the system, must also be updated, else the resulting vmcore is
>> inaccurate (eg. missing either CPU context or memory regions).
>>
>> The current solution utilizes udev (eg. RHEL /usr/lib/udev/rules.d/
>> 98-kexec.rules) to initiate an unload-then-reload of the *entire* kdump
>> image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by
>> the userspace kexec utility. This occurrs just so the elfcorehdr can
>> be updated with the latest list of CPUs and memory regions. In a
>> previous post I have outlined the significant performance problems
>> related to offloading this activity to userspace.
>>
>> With the Linux kernel 6.6 commit below, the kernel now has the ability
>> to directly modify the elfcorehdr, eliminating the need to
>> unload-then-reload the entire kdump image when CPU or memory is hot
>> un/plugged or on/offlined.
>>
>>   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d6
>> 8b4b6f307d155475cce541f2aee938032ed22e
>>
>> This kexec-tools patch series is for supporting hotplug with the
>> kexec_load() syscall; the kernel directly supports hotplug for the
>> kexec_file_load() syscall, requiring no userspace help.
>>
>> There are two basic obstacles/requirements for the kexec-tools to
>> overcome in order to support kernel hotplug rewriting of the
>> elfcorehdr.
>>
>> First, the buffer containing the elfcorehdr must be excluded from the
>> purgatory checksum/digest, which is computed at load time. Otherwise
>> kernel run-time changes to the elfcorehdr, as a result of hot un/plug,
>> would result in the checksum failing (specifically in purgatory at
>> panic kernel boot time), and kdump capture kernel failing to start.
>> To let the kernel know it is okay to modify the elfcorehdr, kexec
>> sets the KEXEC_UPDATE_ELFCOREHDR flag.
>>
>> NOTE: The kernel specifically does *NOT* attempt to recompute the
>> checksum/digest as that would ultimately require patching the in-
>> memory purgatory image with the updated checksum. As that purgatory
>> image is already fully linked, it is binary blob containing no ELF
>> information which would allow it to be re-linked or patched. Thus
>> excluding the elfcorehdr from the checksum/digests avoids all these
>> problems.
>>
>> Second, the size of the elfcorehdr buffer must be large enough
>> to accomodate growth of the number of CPUs and/or memory regions.
>>
>> To satisfy the first requirement, this patch series introduces the
>> --hotplug option to indicate to kexec-tools that kexec should exclude
>> the elfcorehdr buffer from the purgatory checksum/digest calculation
>> and set the KEXEC_UPDATE_ELFCOREHDR flag.
>>
>> To satisfy the second requirement, the size is obtained from the
>> /sys/kernel/crash_elfcorehdr_size node (new with the kernel series
>> cited above).
>>
>> To use this feature with kexec_load() syscall, invoke kexec with:
>>
>>   kexec -c --hotplug ...
>>
>> Thanks!
>> eric
> 
> Thanks Eric,
> 
> applied.

Excellent, thank you!
eric

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec