[PATCH] makedumpfile: search for a debug vmlinux

[PATCH] makedumpfile: search for a debug vmlinux

[Cliff Wickman]

From: Cliff Wickman <cpw@sgi.com>


makedumpfile needs the debug info from either /proc/vmcore or in vmlinux
to know how to find free pages using the buddy method.

The distro procedures do not pass the vmlinux (with the -x option), so
add a search for a debug vmlinux in the usual locations.
It's not needed if the page info is in vmcore.  But warn if it is found
in neither place.

Diffed against makedumpfile-1.5.4
Signed-off-by: Cliff Wickman <cpw@sgi.com>
---
 makedumpfile.c |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

Index: makedumpfile-1.5.4/makedumpfile.c
===================================================================
--- makedumpfile-1.5.4.orig/makedumpfile.c
+++ makedumpfile-1.5.4/makedumpfile.c
@@ -23,6 +23,7 @@
 #include <stddef.h>
 #include <ctype.h>
 #include <sys/time.h>
+#include <sys/stat.h>
 
 struct symbol_table	symbol_table;
 struct size_table	size_table;
@@ -33,6 +34,7 @@ struct srcfile_table	srcfile_table;
 
 struct vm_table		vt = { 0 };
 struct DumpInfo		*info = NULL;
+struct utsname		utsname;
 
 char filename_stdout[] = FILENAME_STDOUT;
 
@@ -62,6 +64,7 @@ unsigned long long num_dumped;
 int retcd = FAILED;	/* return code */
 int aflag = 0;
 int total_cycles = 0;
+static int no_vmlinux;
 
 #define INITIALIZE_LONG_TABLE(table, value) \
 do { \
@@ -2987,6 +2990,9 @@ initial(void)
 		if (!read_vmcoreinfo_from_vmcore(offset, size, FALSE))
 			return FALSE;
 		debug_info = TRUE;
+		if ((OFFSET(page.private) == NOT_FOUND_STRUCTURE) &&
+		    no_vmlinux)
+			PROGRESS_MSG("No vmlinux and no page info in vmcore\n");
 	}
 
 out:
@@ -7799,6 +7805,7 @@ create_dumpfile(void)
 		return FALSE;
 
 	if (!info->flag_refiltering && !info->flag_sadump) {
+		/* cpw: we get debug info from /proc/vmcore here */
 		if (!get_elf_info(info->fd_memory, info->name_memory))
 			return FALSE;
 	}
@@ -8740,6 +8747,91 @@ static struct option longopts[] = {
 	{0, 0, 0, 0}
 };
 
+/*
+ * Look for a debug vmlinux in the usual places.
+ */
+void
+find_vmlinux()
+{
+	int ret;
+	char pathname[200];
+	struct stat stat_buf;
+
+	ret = uname(&utsname);
+	if (ret < 0) {
+		fprintf(stderr, "uname failed; errno %d", errno);
+	}
+
+	/* these may work if the crash kernel is in multi-user mode */
+	sprintf(pathname, "/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/boot/vmlinux-%s", utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+
+	/*
+	 * the crash kernel normally runs with the root device mounted
+	 * as /root or /mnt
+	 */
+	sprintf(pathname, "/root/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/root/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/root/boot/vmlinux-%s", utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/mnt/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/mnt/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+	sprintf(pathname, "/mnt/boot/vmlinux-%s", utsname.release);
+	if (stat(pathname, &stat_buf) == 0) {
+		info->name_vmlinux = pathname;
+		PROGRESS_MSG("Using %s.\n", pathname);
+		return;
+	}
+
+	no_vmlinux = 1;
+}
+
 int
 main(int argc, char *argv[])
 {
@@ -8888,6 +8980,11 @@ main(int argc, char *argv[])
 	if (flag_debug)
 		message_level |= ML_PRINT_DEBUG_MSG;
 
+	if (!info->flag_read_vmcoreinfo && !info->name_vmlinux) {
+		/* -x not specified, so look in standard places */
+		find_vmlinux();
+	}
+
 	if (info->flag_show_usage) {
 		print_usage();
 		return COMPLETED;

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH] makedumpfile: search for a debug vmlinux

[HATAYAMA Daisuke]

(2013/08/29 7:08), Cliff Wickman wrote:
> From: Cliff Wickman <cpw@sgi.com>
> 
> 
> makedumpfile needs the debug info from either /proc/vmcore or in vmlinux
> to know how to find free pages using the buddy method.
> 
> The distro procedures do not pass the vmlinux (with the -x option), so
> add a search for a debug vmlinux in the usual locations.
> It's not needed if the page info is in vmcore.  But warn if it is found
> in neither place.
> 

makedumpfile is designed to get necessary debug information from VMCOREINFO note
available from /proc/vmcore. Why do you need vmlinux?

<cut>

> @@ -8740,6 +8747,91 @@ static struct option longopts[] = {
>   	{0, 0, 0, 0}
>   };
>   
> +/*
> + * Look for a debug vmlinux in the usual places.
> + */
> +void
> +find_vmlinux()
> +{
> +	int ret;
> +	char pathname[200];
> +	struct stat stat_buf;
> +
> +	ret = uname(&utsname);
> +	if (ret < 0) {
> +		fprintf(stderr, "uname failed; errno %d", errno);
> +	}
> +
> +	/* these may work if the crash kernel is in multi-user mode */
> +	sprintf(pathname, "/usr/lib/debug/lib/modules/%s/vmlinux",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/usr/lib/debug/boot/vmlinux-%s.debug",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/boot/vmlinux-%s", utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +
> +	/*
> +	 * the crash kernel normally runs with the root device mounted
> +	 * as /root or /mnt
> +	 */

This depends on distributions and their own kdump/kexec userland configuration
tools. I don't think it a good idea. It is the distribution tools that know
where debuginfo files are located and so they should search for debuginfo files
and they should specify them with -x option.

Basically, makedumpfile should not assume root device in the 2nd kernel since
it could be corrupted due to the same crash causing kdump to be triggered.
Also, debug informaiton is very large file over 100MiB and so should not be
included in kdump initramfs. VMCOREINFO was invented so to suppress amount
of debug informaiton as much as possible.

> +	sprintf(pathname, "/root/usr/lib/debug/lib/modules/%s/vmlinux",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/root/usr/lib/debug/boot/vmlinux-%s.debug",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/root/boot/vmlinux-%s", utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/mnt/usr/lib/debug/lib/modules/%s/vmlinux",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/mnt/usr/lib/debug/boot/vmlinux-%s.debug",
> +		utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +	sprintf(pathname, "/mnt/boot/vmlinux-%s", utsname.release);
> +	if (stat(pathname, &stat_buf) == 0) {
> +		info->name_vmlinux = pathname;
> +		PROGRESS_MSG("Using %s.\n", pathname);
> +		return;
> +	}
> +
> +	no_vmlinux = 1;
> +}
> +


-- 
Thanks.
HATAYAMA, Daisuke


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH] makedumpfile: search for a debug vmlinux

[Cliff Wickman]

On Fri, Aug 30, 2013 at 10:11:09AM +0900, HATAYAMA Daisuke wrote:
> (2013/08/29 7:08), Cliff Wickman wrote:
> > From: Cliff Wickman <cpw@sgi.com>
> > 
> > 
> > makedumpfile needs the debug info from either /proc/vmcore or in vmlinux
> > to know how to find free pages using the buddy method.
> > 
> > The distro procedures do not pass the vmlinux (with the -x option), so
> > add a search for a debug vmlinux in the usual locations.
> > It's not needed if the page info is in vmcore.  But warn if it is found
> > in neither place.
> > 
> 
> makedumpfile is designed to get necessary debug information from VMCOREINFO note
> available from /proc/vmcore. Why do you need vmlinux?
 
The vmcore doesn't always provide the page structure information.
Specifically, the offset of 'private' and '_mapcount' in the page
structure, which are needed for identification of user huge pages.
(I've seen this to be true on a sles11s2 and rhel65 (3.0.80 2.6.32-410))
I'm using the huge page patch from Petr Tesarik. The one that I think you
said Kumagai-san is reworking.

I did improve my patch, as provided below.
The below version does not try to replace the debuginfo from vmcore, but
only to extract the few bits of page structure information needed for 
the Tesarik patch.

You may not see the need for the additional (page) info on current kernels.
So I understand that you may not be interested in this patch, as it may
not be needed for kernels that are current when the huge page
identification patch is ready.

-Cliff




makedumpfile needs some debug info from either /proc/vmcore or in vmlinux
to know how to find free pages using the buddy method.

The distro procedures do not pass the vmlinux (with the -x option), so
add a search for a debug vmlinux in the usual locations.
It's not needed if the page info is in vmcore.
Get it from a debug vmlinux if one is found.
Warn if it is found in neither place.

-cpw
---

Index: makedumpfile-1.5.4/makedumpfile.c
===================================================================
--- makedumpfile-1.5.4.orig/makedumpfile.c
+++ makedumpfile-1.5.4/makedumpfile.c
@@ -23,6 +23,7 @@
 #include <stddef.h>
 #include <ctype.h>
 #include <sys/time.h>
+#include <sys/stat.h>
 
 struct symbol_table	symbol_table;
 struct size_table	size_table;
@@ -33,6 +34,7 @@ struct srcfile_table	srcfile_table;
 
 struct vm_table		vt = { 0 };
 struct DumpInfo		*info = NULL;
+struct utsname		utsname;
 
 char filename_stdout[] = FILENAME_STDOUT;
 
@@ -62,6 +64,7 @@ unsigned long long num_dumped;
 int retcd = FAILED;	/* return code */
 int aflag = 0;
 int total_cycles = 0;
+static char vmpathname[200];
 
 #define INITIALIZE_LONG_TABLE(table, value) \
 do { \
@@ -2857,6 +2860,84 @@ initialize_bitmap_memory(void)
 }
 
 int
+get_page_info(char *vmpathname)
+{
+	int fd;
+
+	if (vmpathname) {
+		if ((fd = open(vmpathname, O_RDONLY)) < 0) {
+			ERRMSG("Can't open the kernel file(%s). %s\n",
+			    vmpathname, strerror(errno));
+			return FALSE;
+		}
+		info->fd_vmlinux = fd;
+	}
+	set_dwarf_debuginfo("vmlinux", NULL, vmpathname, info->fd_vmlinux);
+	OFFSET_INIT(page.private, "page", "private");
+	ENUM_NUMBER_INIT(PG_buddy, "PG_buddy");
+	OFFSET_INIT(page._mapcount, "page", "_mapcount");
+	return TRUE;
+}
+
+/*
+ * Look for a debug vmlinux in the usual places.
+ */
+int
+find_vmlinux()
+{
+	int ret;
+	struct stat stat_buf;
+
+	ret = uname(&utsname);
+	if (ret < 0) {
+		fprintf(stderr, "uname failed; errno %d", errno);
+		return 0;
+	}
+
+	/* these may work if the crash kernel is in multi-user mode */
+	sprintf(vmpathname, "/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/boot/vmlinux-%s", utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+
+	/*
+	 * the crash kernel normally runs with the root device mounted
+	 * as /root or /mnt
+	 */
+	sprintf(vmpathname, "/root/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/root/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/root/boot/vmlinux-%s", utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/mnt/usr/lib/debug/lib/modules/%s/vmlinux",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/mnt/usr/lib/debug/boot/vmlinux-%s.debug",
+		utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+	sprintf(vmpathname, "/mnt/boot/vmlinux-%s", utsname.release);
+	if (!stat(vmpathname, &stat_buf))
+		return 1;
+
+	return 0;
+}
+
+int
 initial(void)
 {
 	off_t offset;
@@ -3007,10 +3088,25 @@ initial(void)
 	 *       than -x/-i option.
 	 */
 	if (has_vmcoreinfo()) {
+		PROGRESS_MSG("Getting debug information from /proc/vmcore\n");
 		get_vmcoreinfo(&offset, &size);
 		if (!read_vmcoreinfo_from_vmcore(offset, size, FALSE))
 			return FALSE;
 		debug_info = TRUE;
+		if (OFFSET(page.private) == NOT_FOUND_STRUCTURE) {
+			/* before giving up on identifying page flags, try
+ 			   any found vmlinux */
+			if (find_vmlinux()) {
+				PROGRESS_MSG("Getting page info from %s\n",
+					vmpathname);
+				if (!get_page_info(vmpathname)) {
+					PROGRESS_MSG("No page info in %s\n",
+						vmpathname);
+				}
+			} else
+				PROGRESS_MSG(
+				"No vmlinux and no page info in vmcore\n");
+		}
 	}
 
 out:

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH] makedumpfile: search for a debug vmlinux

[HATAYAMA Daisuke]

(2013/09/04 3:48), Cliff Wickman wrote:
> On Fri, Aug 30, 2013 at 10:11:09AM +0900, HATAYAMA Daisuke wrote:
>> (2013/08/29 7:08), Cliff Wickman wrote:
>>> From: Cliff Wickman <cpw@sgi.com>
>>>
>>>
>>> makedumpfile needs the debug info from either /proc/vmcore or in vmlinux
>>> to know how to find free pages using the buddy method.
>>>
>>> The distro procedures do not pass the vmlinux (with the -x option), so
>>> add a search for a debug vmlinux in the usual locations.
>>> It's not needed if the page info is in vmcore.  But warn if it is found
>>> in neither place.
>>>
>>
>> makedumpfile is designed to get necessary debug information from VMCOREINFO note
>> available from /proc/vmcore. Why do you need vmlinux?
>
> The vmcore doesn't always provide the page structure information.
> Specifically, the offset of 'private' and '_mapcount' in the page
> structure, which are needed for identification of user huge pages.
> (I've seen this to be true on a sles11s2 and rhel65 (3.0.80 2.6.32-410))
> I'm using the huge page patch from Petr Tesarik. The one that I think you
> said Kumagai-san is reworking.
>

I don't know your situation in each distribution, but simply, you should post
patch to distribution sides, not upstream makedumpfile. Try to extend VMCOREINFO in
kernels or add the workaround of this patch series in makedumpfile on each
distributions.

Also, I think automatic search is not absolutely necessary. It's enough to
specify vmlinux path manually in configuration file. Using vmlinux file itself is
possible without any problem. This workaround doesn't need makedumpfile change.

If you still want automatic search, it's enough to prepare a script that does that
and add it to initramfs by extra_bins directive and then specify it in core_collector
directive. This is way on RHEL. I don't way on SLES, but it's SLES issue.

> I did improve my patch, as provided below.
> The below version does not try to replace the debuginfo from vmcore, but
> only to extract the few bits of page structure information needed for
> the Tesarik patch.
>
> You may not see the need for the additional (page) info on current kernels.
> So I understand that you may not be interested in this patch, as it may
> not be needed for kernels that are current when the huge page
> identification patch is ready.
>
> -Cliff

kdump framework is designed carefully, primarily focusing on how to keep
reliability. Not assuming root device in the 2nd kernel is one of them.
Keeping reliability is more important. We should not break that by this kind of
workaround.

-- 
Thanks.
HATAYAMA, Daisuke


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec