[PATCH v2 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[PATCH v2 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[Eric DeVolder]

When the kdump service is loaded, if a CPU or memory is hot
un/plugged, the crash elfcorehdr, which describes the CPUs and memory
in the system, must also be updated, else the resulting vmcore is
inaccurate (eg. missing either CPU context or memory regions).

The current solution (eg. RHEL /usr/lib/udev/rules.d/98-kexec.rules)
utilizes udev to initiate an unload-then-reload of the *entire* kdump
image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by
the userspace kexec utility. In a previous kernel patch post I have
outlined the significant performance problems related to offloading
this activity to userspace.

As such, I've been working to provide the ability for the Linux kernel
to directly modify the elfcorehdr in response to hotplug changes.

 https://lore.kernel.org/lkml/20230404180326.6890-1-eric.devolder@oracle.com/

The series listed above is v21, and the v22 contains changes that
work in concert with the v2 changes cited within. (I'm posting the
kexec-tools changes first so I can reference them in the kernel v22
posting.)

I believe this work to be nearing the finish line. As such, I'd like
to start posting the kexec-tools userspace changes for review in order
to minimize the time to adoption.

This kexec-tools patch series is for supporting the kexec_load
syscall only. The kernel patch series cited above is self-contained
for the kexec_file_load syscall, requiring no userspace help.

There are two basic obstacles/requirements for the kexec-tools to
overcome in order to support kernel hotplug rewriting of the
elfcorehdr.

First, the buffer containing the elfcorehdr must be excluded from the
purgatory checksum/digest, which is computed at load time. Otherwise
kernel run-time changes to the elfcorehdr, as a result of hot un/plug,
would result in the checksum failing (specifically in purgatory at
panic kernel boot time), and kdump capture kernel failing to start.
To let the kernel know it is okay to modify the elfcorehdr, kexec
sets the KEXEC_UPDATE_ELFCOREHDR flag.

NOTE: The kernel specifically does *NOT* attempt to recompute the
checksum/digest as that would ultimately require patching the in-
memory purgatory image with the updated checksum. As that purgatory
image is already fully linked, it is binary blob containing no ELF
information which would allow it to be re-linked or patched. Thus
excluding the elfcorehdr from the checksum/digests avoids all these
problems.

Second, the size of the elfcorehdr buffer must be large enough
to accomodate growth of the number of CPUs and/or memory regions.

To satisfy the first requirement, this patch series introduces the
--hotplug option to indicate to kexec-tools that kexec should exclude
the elfcorehdr buffer from the purgatory checksum/digest calculation
and set the KEXEC_UPDATE_ELFCOREHDR flag.

To satisfy the second requirement, the size is obtained from the
(proposed in the kernel series above)
/sys/kernel/crash_elfcorehdr_size node, or it can be specified
manually with new --elfcorehdrsz= option.

I am intentionally posting this series before the kernel changes
have been merged. I'm hoping to facilitate discussion as to how
kexec-tools wants to handle the soon-to-be new kernel feature.

Discussion items:

- It is worth noting, that deploying kexec-tools, with this series
  included, on kernels that do NOT have the kernel hotplug series
  cited above, is safe to do. The result of running a kernel without
  hotplug elfcorehdr support with kexec-tools and the --hotplug option
  simply removes the elfcorehdr buffer from the digest. This does not
  prevent kdump from operating; the only risk being a slight chance of
  corruption of the elfcorehdr, as it now not covered by the checksum.
  Using the --elfcorehdrsz option on a kernel without hotplug
  elfcorehdr support simply results in a possibly oversized buffer for
  the elfcorehdr, there is no harm in that.

- While I currently have the --hotplug as an option, the option could
  be eliminated (or reversed polarity) it would be safe to *always*
  omit the elfcorehdr from the checksum/digest for purgatory.
  If this were the case, then distros would not have to make any
  changes to kdump scripts to pass the --hotplug option. Then, when
  their kernel does include the kernel patch series cited above,
  kdump and hotplug would "just work".

- I'm unsure if these options should be kept as common/global
  kexec options, or moved to arch options.

- I'm only showing x86 support (and testing) at this time, but
  it would be straight forward to provide similar support for the
  other architectures in a future patch revision.

Thanks!
eric

---
v2: 3may2023
 - Setting KEXEC_UPDATE_ELFCOREHDR flag
 - Utilizing /sys/kernel/crash_elfcorehdr_size info.

v1: 20oct2022
 http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
 - Initial patch series

RFC:
 https://lore.kernel.org/lkml/b04ed259-dc5f-7f30-6661-c26f92d9096a@oracle.com/
 s/vmcoreinfo/elfcorehdr/g
---


Eric DeVolder (6):
  kexec: define KEXEC_UPDATE_ELFCOREHDR
  crashdump: introduce the hotplug command line options
  crashdump: setup hotplug support
  crashdump: exclude elfcorehdr segment from digest for hotplug
  crashdump/x86: identify elfcorehdr segment for hotplug
  crashdump/x86: set the elfcorehdr segment size for hotplug

 kexec/arch/i386/crashdump-x86.c |  8 ++++++
 kexec/kexec-syscall.h           |  1 +
 kexec/kexec.c                   | 45 +++++++++++++++++++++++++++++++++
 kexec/kexec.h                   | 10 +++++++-
 4 files changed, 63 insertions(+), 1 deletion(-)

-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 1/6] kexec: define KEXEC_UPDATE_ELFCOREHDR

[Eric DeVolder]

The Linux kernel defines this flag to indicate that the kexec_load()'ed
image is setup so that the kernel may directly modify the elfcorehdr
(and not cause the purgatory digest checksum to fail).

Define this flag to match/mirror the kernel flag.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec-syscall.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kexec/kexec-syscall.h b/kexec/kexec-syscall.h
index 1e2d12f..2559bff 100644
--- a/kexec/kexec-syscall.h
+++ b/kexec/kexec-syscall.h
@@ -112,6 +112,7 @@ static inline long kexec_file_load(int kernel_fd, int initrd_fd,
 
 #define KEXEC_ON_CRASH		0x00000001
 #define KEXEC_PRESERVE_CONTEXT	0x00000002
+#define KEXEC_UPDATE_ELFCOREHDR	0x00000004
 #define KEXEC_ARCH_MASK		0xffff0000
 
 /* Flags for kexec file based system call */
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 2/6] crashdump: introduce the hotplug command line options

[Eric DeVolder]

Introducing the --hotplug and --elfcorehdrsz command line options,
which are used to indicate to the kernel that the kdump image
is setup to permit the kernel to directly modify the elfcorehdr in
response to CPU and memory hotplug and/or online/offline events.

This option is only meaningful for kexec_load() syscall. For the
kexec_file_load() syscall, this option is a no-op as the kernel
handles all aspects of loading the kdump image.

This change is just the command line processing portion; the handling
of hotplug is in the subsequent patches.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.c | 15 +++++++++++++++
 kexec/kexec.h |  9 ++++++++-
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/kexec/kexec.c b/kexec/kexec.c
index 36bb2ad..4a64e53 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -58,6 +58,8 @@
 
 unsigned long long mem_min = 0;
 unsigned long long mem_max = ULONG_MAX;
+unsigned long elfcorehdrsz = 0;
+int do_hotplug = 0;
 static unsigned long kexec_flags = 0;
 /* Flags for kexec file (fd) based syscall */
 static unsigned long kexec_file_flags = 0;
@@ -1565,6 +1567,19 @@ int main(int argc, char *argv[])
 		case OPT_PRINT_CKR_SIZE:
 			print_crashkernel_region_size();
 			return 0;
+		case OPT_ELFCOREHDRSZ:
+			elfcorehdrsz = strtoul(optarg, &endptr, 0);
+			if (*endptr) {
+				fprintf(stderr,
+					"Bad option value in --elfcorehdrsz=%s\n",
+					optarg);
+				usage();
+				return 1;
+			}
+			break;
+		case OPT_HOTPLUG:
+			do_hotplug = 1;
+			break;
 		default:
 			break;
 		}
diff --git a/kexec/kexec.h b/kexec/kexec.h
index 0d820ad..fc24b67 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -231,7 +231,9 @@ extern int file_types;
 #define OPT_PRINT_CKR_SIZE	262
 #define OPT_LOAD_LIVE_UPDATE	263
 #define OPT_EXEC_LIVE_UPDATE	264
-#define OPT_MAX			265
+#define OPT_ELFCOREHDRSZ		265
+#define OPT_HOTPLUG		        266
+#define OPT_MAX		267
 #define KEXEC_OPTIONS \
 	{ "help",		0, 0, OPT_HELP }, \
 	{ "version",		0, 0, OPT_VERSION }, \
@@ -258,6 +260,8 @@ extern int file_types;
 	{ "debug",		0, 0, OPT_DEBUG }, \
 	{ "status",		0, 0, OPT_STATUS }, \
 	{ "print-ckr-size",     0, 0, OPT_PRINT_CKR_SIZE }, \
+	{ "elfcorehdrsz",		1, 0, OPT_ELFCOREHDRSZ }, \
+	{ "hotplug",		    0, 0, OPT_HOTPLUG }, \
 
 #define KEXEC_OPT_STR "h?vdfixyluet:pscaS"
 
@@ -295,6 +299,9 @@ extern int ifdown(void);
 extern char purgatory[];
 extern size_t purgatory_size;
 
+extern unsigned long elfcorehdrsz;
+extern int do_hotplug;
+
 #define BOOTLOADER "kexec"
 #define BOOTLOADER_VERSION PACKAGE_VERSION
 
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 3/6] crashdump: setup hotplug support

[Eric DeVolder]

To allow direct modification of the elfcorehdr by the kernel, in
response to CPU and memory hot un/plug and/or online/offline events,
the buffer containing the:
 - elfcorehdr must be excluded from the purgatory checksum/digest
   (see "crashdump: exclude elfcorehdr segment from digest for
   hotplug")
 - the elfcorehdr segment must be large enough, and
 - the kernel must be permitted to modify the elfcorehdr

For hotplug, the size of the elfcorehdr segment is obtained from the
kernel, or from the command line option.

The KEXEC_UPDATE_ELFCOREHDR flag indicates to the kernel that it can
make direct modifications to the elfcorehdr.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/kexec/kexec.c b/kexec/kexec.c
index 4a64e53..c59e795 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -1626,6 +1626,28 @@ int main(int argc, char *argv[])
 		die("--load-live-update can only be used with xen\n");
 	}
 
+    /* NOTE: Xen KEXEC_LIVE_UPDATE and KEXEC_UPDATE_ELFCOREHDR collide */
+	if (do_hotplug) {
+		const char *ces = "/sys/kernel/crash_elfcorehdr_size";
+		if (!elfcorehdrsz) {
+			unsigned long ul;
+			char *buf;
+			off_t nread = 0;
+			buf = slurp_file_len(ces, sizeof(buf)-1, &nread);
+			if (buf) {
+			    ul = strtoul(buf, NULL, 0);
+				elfcorehdrsz = ul;
+			}
+		}
+		if (!elfcorehdrsz) {
+			die("If path %s does not exist, must specify the\n"
+				"--elfcorehdrsz= option\n", ces);
+		}
+		dbgprintf("ELFCOREHDR_SIZE %lu\n", elfcorehdrsz);
+		/* Indicate to the kernel it is ok to modify the elfcorehdr */
+		kexec_flags |= KEXEC_UPDATE_ELFCOREHDR;
+	}
+
 	fileind = optind;
 	/* Reset getopt for the next pass; called in other source modules */
 	opterr = 1;
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 4/6] crashdump: exclude elfcorehdr segment from digest for hotplug

[Eric DeVolder]

To allow direct modification of the elfcorehdr by the kernel, in
response to CPU and memory hot un/plug and/or online/offline events,
the buffer containing the elfcorehdr must be excluded from the
purgatory checksum/digest.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/kexec.c | 8 ++++++++
 kexec/kexec.h | 1 +
 2 files changed, 9 insertions(+)

diff --git a/kexec/kexec.c b/kexec/kexec.c
index c59e795..9d0d6cb 100644
--- a/kexec/kexec.c
+++ b/kexec/kexec.c
@@ -674,6 +674,14 @@ static void update_purgatory(struct kexec_info *info)
 		if (info->segment[i].mem == (void *)info->rhdr.rel_addr) {
 			continue;
 		}
+
+		/* Don't include elfcorehdr in the checksum, if hotplug
+		 * support enabled.
+		 */
+		if (do_hotplug && (info->segment[i].mem == (void *)info->elfcorehdr)) {
+			continue;
+		}
+
 		sha256_update(&ctx, info->segment[i].buf,
 			      info->segment[i].bufsz);
 		nullsz = info->segment[i].memsz - info->segment[i].bufsz;
diff --git a/kexec/kexec.h b/kexec/kexec.h
index fc24b67..8acbdc1 100644
--- a/kexec/kexec.h
+++ b/kexec/kexec.h
@@ -169,6 +169,7 @@ struct kexec_info {
 	int command_line_len;
 
 	int skip_checks;
+	unsigned long elfcorehdr;
 };
 
 struct arch_map_entry {
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 5/6] crashdump/x86: identify elfcorehdr segment for hotplug

[Eric DeVolder]

Identify the segment containing the elfcorehdr buffer so that
it can be excluded from the purgatory checksum/digest, if hotplug
support is in effect.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/arch/i386/crashdump-x86.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index df1f24c..cb86ca7 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -956,6 +956,9 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
 	} else {
 		memsz = bufsz;
 	}
+
+	/* Record the location of the elfcorehdr for hotplug handling */
+	info->elfcorehdr =
 	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
 							max_addr, -1);
 	dbgprintf("Created elf header segment at 0x%lx\n", elfcorehdr);
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH v2 6/6] crashdump/x86: set the elfcorehdr segment size for hotplug

[Eric DeVolder]

For hotplug, the elfcorehdr segment must be oversized to allow a
growing number of CPUs or memory regions.

Signed-off-by: Eric DeVolder <eric.devolder@oracle.com>
---
 kexec/arch/i386/crashdump-x86.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kexec/arch/i386/crashdump-x86.c b/kexec/arch/i386/crashdump-x86.c
index cb86ca7..18f745e 100644
--- a/kexec/arch/i386/crashdump-x86.c
+++ b/kexec/arch/i386/crashdump-x86.c
@@ -957,6 +957,11 @@ int load_crashdump_segments(struct kexec_info *info, char* mod_cmdline,
 		memsz = bufsz;
 	}
 
+	/* Handle override in order to support hotplug */
+	if (do_hotplug) {
+		memsz = _ALIGN(elfcorehdrsz, align);
+	}
+
 	/* Record the location of the elfcorehdr for hotplug handling */
 	info->elfcorehdr =
 	elfcorehdr = add_buffer(info, tmp, bufsz, memsz, align, min_base,
-- 
2.31.1


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH v2 0/6] crashdump: Kernel handling of CPU and memory hot un/plug

[Hari Bathini]

On 04/05/23 3:46 am, Eric DeVolder wrote:
> When the kdump service is loaded, if a CPU or memory is hot
> un/plugged, the crash elfcorehdr, which describes the CPUs and memory
> in the system, must also be updated, else the resulting vmcore is
> inaccurate (eg. missing either CPU context or memory regions).
> 
> The current solution (eg. RHEL /usr/lib/udev/rules.d/98-kexec.rules)
> utilizes udev to initiate an unload-then-reload of the *entire* kdump
> image (eg. kernel, initrd, boot_params, purgatory and elfcorehdr) by
> the userspace kexec utility. In a previous kernel patch post I have
> outlined the significant performance problems related to offloading
> this activity to userspace.
> 
> As such, I've been working to provide the ability for the Linux kernel
> to directly modify the elfcorehdr in response to hotplug changes.
> 
>   https://lore.kernel.org/lkml/20230404180326.6890-1-eric.devolder@oracle.com/
> 
> The series listed above is v21, and the v22 contains changes that
> work in concert with the v2 changes cited within. (I'm posting the
> kexec-tools changes first so I can reference them in the kernel v22
> posting.)
> 
> I believe this work to be nearing the finish line. As such, I'd like
> to start posting the kexec-tools userspace changes for review in order
> to minimize the time to adoption.
> 
> This kexec-tools patch series is for supporting the kexec_load
> syscall only. The kernel patch series cited above is self-contained
> for the kexec_file_load syscall, requiring no userspace help.
> 
> There are two basic obstacles/requirements for the kexec-tools to
> overcome in order to support kernel hotplug rewriting of the
> elfcorehdr.
> 
> First, the buffer containing the elfcorehdr must be excluded from the
> purgatory checksum/digest, which is computed at load time. Otherwise
> kernel run-time changes to the elfcorehdr, as a result of hot un/plug,
> would result in the checksum failing (specifically in purgatory at
> panic kernel boot time), and kdump capture kernel failing to start.
> To let the kernel know it is okay to modify the elfcorehdr, kexec
> sets the KEXEC_UPDATE_ELFCOREHDR flag.
> 
> NOTE: The kernel specifically does *NOT* attempt to recompute the
> checksum/digest as that would ultimately require patching the in-
> memory purgatory image with the updated checksum. As that purgatory
> image is already fully linked, it is binary blob containing no ELF
> information which would allow it to be re-linked or patched. Thus
> excluding the elfcorehdr from the checksum/digests avoids all these
> problems.
> 
> Second, the size of the elfcorehdr buffer must be large enough
> to accomodate growth of the number of CPUs and/or memory regions.
> 
> To satisfy the first requirement, this patch series introduces the
> --hotplug option to indicate to kexec-tools that kexec should exclude
> the elfcorehdr buffer from the purgatory checksum/digest calculation
> and set the KEXEC_UPDATE_ELFCOREHDR flag.
> 
> To satisfy the second requirement, the size is obtained from the
> (proposed in the kernel series above)
> /sys/kernel/crash_elfcorehdr_size node, or it can be specified
> manually with new --elfcorehdrsz= option.
> 
> I am intentionally posting this series before the kernel changes
> have been merged. I'm hoping to facilitate discussion as to how
> kexec-tools wants to handle the soon-to-be new kernel feature.
> 
> Discussion items:
> 
> - It is worth noting, that deploying kexec-tools, with this series
>    included, on kernels that do NOT have the kernel hotplug series
>    cited above, is safe to do. The result of running a kernel without
>    hotplug elfcorehdr support with kexec-tools and the --hotplug option
>    simply removes the elfcorehdr buffer from the digest. This does not
>    prevent kdump from operating; the only risk being a slight chance of
>    corruption of the elfcorehdr, as it now not covered by the checksum.
>    Using the --elfcorehdrsz option on a kernel without hotplug
>    elfcorehdr support simply results in a possibly oversized buffer for
>    the elfcorehdr, there is no harm in that.
> 
> - While I currently have the --hotplug as an option, the option could
>    be eliminated (or reversed polarity) it would be safe to *always*
>    omit the elfcorehdr from the checksum/digest for purgatory.
>    If this were the case, then distros would not have to make any
>    changes to kdump scripts to pass the --hotplug option. Then, when
>    their kernel does include the kernel patch series cited above,
>    kdump and hotplug would "just work".
> 
> - I'm unsure if these options should be kept as common/global
>    kexec options, or moved to arch options.
> 
> - I'm only showing x86 support (and testing) at this time, but
>    it would be straight forward to provide similar support for the
>    other architectures in a future patch revision.

True. Should be straightforward to add similar support for other
architectures. For example, powerpc would need another flag
KEXEC_UPDATE_FDT on top of the flag to update elfcorehdr.

Looks good to me. For the series..

Acked-by: Hari Bathini <hbathini@linux.ibm.com>

> Thanks!
> eric
> 
> ---
> v2: 3may2023
>   - Setting KEXEC_UPDATE_ELFCOREHDR flag
>   - Utilizing /sys/kernel/crash_elfcorehdr_size info.
> 
> v1: 20oct2022
>   http://lists.infradead.org/pipermail/kexec/2022-October/026032.html
>   - Initial patch series
> 
> RFC:
>   https://lore.kernel.org/lkml/b04ed259-dc5f-7f30-6661-c26f92d9096a@oracle.com/
>   s/vmcoreinfo/elfcorehdr/g
> ---
> 
> 
> Eric DeVolder (6):
>    kexec: define KEXEC_UPDATE_ELFCOREHDR
>    crashdump: introduce the hotplug command line options
>    crashdump: setup hotplug support
>    crashdump: exclude elfcorehdr segment from digest for hotplug
>    crashdump/x86: identify elfcorehdr segment for hotplug
>    crashdump/x86: set the elfcorehdr segment size for hotplug
> 
>   kexec/arch/i386/crashdump-x86.c |  8 ++++++
>   kexec/kexec-syscall.h           |  1 +
>   kexec/kexec.c                   | 45 +++++++++++++++++++++++++++++++++
>   kexec/kexec.h                   | 10 +++++++-
>   4 files changed, 63 insertions(+), 1 deletion(-)
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec