makedumpfile-1.1.5: Xen support

makedumpfile-1.1.5: Xen support

[Ken'ichi Ohmichi]

Hi,

makedumpfile version 1.1.5 is released.
Please test it. Your opinions/comments are welcome.

Explanation of makedumpfile:
  To shorten the size of the dumpfile and the time of creating the
  dumpfile, makedumpfile copies only the necessary pages for analysis
  to the dumpfile from /proc/vmcore. You can specify the kind of
  unnecessary pages with dump_level. If you want to shorten the size
  further, enable the compression of the page data.


Changelog:
o Add a new feature that extracting the part of xen hypervisor and
  domain-0 from /proc/vmcore. In most cases under xen environment,
  it is enough to analyze a cause of system crash.
  The patch was provided by Itsuro Oda.
  Usage:
    # makedumpfile -E --xen-syms xen-syms /proc/vmcore dumpfile

o Add a new option '-f' to overwrite existing dumpfile.
  The patch was provided by Bernhard Walle.

o Add a new option '--message-level "message-level"'.
  A user can specify multiple message types by setting the sum of each
  message type for "message-level". The default value is '7' (Print
  the progress indicator, common messages, and error messages).
   0: Print nothing.
   1: Print the progress indicator.
   2: Print common messages.
   4: Print error messages.
   8: Print debugging messages.

o Fix the problem that makedumpfile complains about overlapping memory
  segments. On some machines (ia64, discontigmem, and specific machine),
  one pglist_data has overlapping memory segments with other nodes.
  The existing makedumpfile complains about them, and it fails.
  To fix the problem, overlapping memory segments is separated by
  node_memblk. The idea was provided by Bernhard Walle.

o Rename makedumpfile's config file to "vmcoreinfo file".
  http://lists.infradead.org/pipermail/kexec/2007-July/000349.html

o Change the method for getting the kernel's OSRELEASE if generating
  a vmcoreinfo file. A new makedumpfile gets it from a vmlinux file.
  http://lists.infradead.org/pipermail/kexec/2007-July/000319.html

o Some cleanups
  - Cleanup the linux-version checking method.
    The patch was provided by Neil Horman.
  - Cleanup README for the building problem.
  - Cleanup the method for getting the offset of page.mapping.
  - Cleanup the method for reading a vmcoreinfo file.


You can download the latest makedumpfile(ver. 1.1.5) from the following URL.
Details of the change are written on the CVS page of the following site.
https://sourceforge.net/projects/makedumpfile/

Method of installation:
  You can compile the makedumpfile command as follows;
  1. "tar -zxvf makedumpfile-1.1.5.tar.gz"
  2. "cd makedumpfile"
  3. "make; make install"

Usage:
  makedumpfile [-c] [-E] [-d dump_level] [-x vmlinux] dump_mem dump_file

Example:
  If you want to exclude pages filled by zero, cache pages, user pages
  and free pages and to enable compression, please execute the following
  command.

  # makedumpfile -c -d 31 -x vmlinux /proc/vmcore dumpfile


Thanks
Ken'ichi Ohmichi

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Neil Horman]

On Fri, Jul 27, 2007 at 06:12:43PM +0900, Ken'ichi Ohmichi wrote:
> 
> Hi,
> 
> makedumpfile version 1.1.5 is released.
> Please test it. Your opinions/comments are welcome.
> 
> Explanation of makedumpfile:
>   To shorten the size of the dumpfile and the time of creating the
>   dumpfile, makedumpfile copies only the necessary pages for analysis
>   to the dumpfile from /proc/vmcore. You can specify the kind of
>   unnecessary pages with dump_level. If you want to shorten the size
>   further, enable the compression of the page data.
> 
Hey, this is too late to take into 5.1, but I'll hapily take it for 5.2
Neil



> 
> Changelog:
> o Add a new feature that extracting the part of xen hypervisor and
>   domain-0 from /proc/vmcore. In most cases under xen environment,
>   it is enough to analyze a cause of system crash.
>   The patch was provided by Itsuro Oda.
>   Usage:
>     # makedumpfile -E --xen-syms xen-syms /proc/vmcore dumpfile
> 
> o Add a new option '-f' to overwrite existing dumpfile.
>   The patch was provided by Bernhard Walle.
> 
> o Add a new option '--message-level "message-level"'.
>   A user can specify multiple message types by setting the sum of each
>   message type for "message-level". The default value is '7' (Print
>   the progress indicator, common messages, and error messages).
>    0: Print nothing.
>    1: Print the progress indicator.
>    2: Print common messages.
>    4: Print error messages.
>    8: Print debugging messages.
> 
> o Fix the problem that makedumpfile complains about overlapping memory
>   segments. On some machines (ia64, discontigmem, and specific machine),
>   one pglist_data has overlapping memory segments with other nodes.
>   The existing makedumpfile complains about them, and it fails.
>   To fix the problem, overlapping memory segments is separated by
>   node_memblk. The idea was provided by Bernhard Walle.
> 
> o Rename makedumpfile's config file to "vmcoreinfo file".
>   http://lists.infradead.org/pipermail/kexec/2007-July/000349.html
> 
> o Change the method for getting the kernel's OSRELEASE if generating
>   a vmcoreinfo file. A new makedumpfile gets it from a vmlinux file.
>   http://lists.infradead.org/pipermail/kexec/2007-July/000319.html
> 
> o Some cleanups
>   - Cleanup the linux-version checking method.
>     The patch was provided by Neil Horman.
>   - Cleanup README for the building problem.
>   - Cleanup the method for getting the offset of page.mapping.
>   - Cleanup the method for reading a vmcoreinfo file.
> 
> 
> You can download the latest makedumpfile(ver. 1.1.5) from the following URL.
> Details of the change are written on the CVS page of the following site.
> https://sourceforge.net/projects/makedumpfile/
> 
> Method of installation:
>   You can compile the makedumpfile command as follows;
>   1. "tar -zxvf makedumpfile-1.1.5.tar.gz"
>   2. "cd makedumpfile"
>   3. "make; make install"
> 
> Usage:
>   makedumpfile [-c] [-E] [-d dump_level] [-x vmlinux] dump_mem dump_file
> 
> Example:
>   If you want to exclude pages filled by zero, cache pages, user pages
>   and free pages and to enable compression, please execute the following
>   command.
> 
>   # makedumpfile -c -d 31 -x vmlinux /proc/vmcore dumpfile
> 
> 
> Thanks
> Ken'ichi Ohmichi

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Dave Anderson]

> Changelog:
> o Add a new feature that extracting the part of xen hypervisor and
>   domain-0 from /proc/vmcore. In most cases under xen environment,
>   it is enough to analyze a cause of system crash.
>   The patch was provided by Itsuro Oda.
>   Usage:
>     # makedumpfile -E --xen-syms xen-syms /proc/vmcore dumpfile

Can you verify that the dumpfile created above is usable as both a
hypervisor dumpfile as well as a dom0 vmlinux dumpfile?  In other
words, can both of the following sessions be accomplished with the
resultant dumpfile?

  # crash xen-syms dumpfile

and:

  # crash vmlinux dumpfile

Thanks,
   Dave



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Itsuro ODA]

Hi Dave,

>> Changelog:
>> o Add a new feature that extracting the part of xen hypervisor and
>>   domain-0 from /proc/vmcore. In most cases under xen environment,
>>   it is enough to analyze a cause of system crash.
>>   The patch was provided by Itsuro Oda.
>>   Usage:
>>     # makedumpfile -E --xen-syms xen-syms /proc/vmcore dumpfile
>
> Can you verify that the dumpfile created above is usable as both a
> hypervisor dumpfile as well as a dom0 vmlinux dumpfile?  In other
> words, can both of the following sessions be accomplished with the
> resultant dumpfile?
>
>   # crash xen-syms dumpfile
>
> and:
>
>   # crash vmlinux dumpfile
>

Yes.

Thanks.
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Dave Anderson]

Itsuro ODA wrote:
> Hi Dave,
> 
> 
>>>Changelog:
>>>o Add a new feature that extracting the part of xen hypervisor and
>>>  domain-0 from /proc/vmcore. In most cases under xen environment,
>>>  it is enough to analyze a cause of system crash.
>>>  The patch was provided by Itsuro Oda.
>>>  Usage:
>>>    # makedumpfile -E --xen-syms xen-syms /proc/vmcore dumpfile
>>
>>Can you verify that the dumpfile created above is usable as both a
>>hypervisor dumpfile as well as a dom0 vmlinux dumpfile?  In other
>>words, can both of the following sessions be accomplished with the
>>resultant dumpfile?
>>
>>  # crash xen-syms dumpfile
>>
>>and:
>>
>>  # crash vmlinux dumpfile
>>
> 
> 
> Yes.
> 
> Thanks.

Great -- thanks.

Is the "-d" dump_level option completely ignored, or can it
(without vmlinux awareness) recognize and skip zero-filled
pages?

Dave



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Itsuro ODA]

Hi,

> Is the "-d" dump_level option completely ignored, or can it
> (without vmlinux awareness) recognize and skip zero-filled
> pages?

Yes, "-d" is ignored.

Hmm, adding to skip zero-filled pages is easy but I wonder
how effective it is ...

Thanks.
-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Itsuro ODA]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Hi,

> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...

I found it is sometimes effective.

The attached patch enables the "-d" option with --xen-syms.
Note that only the level 1 (skip zero-filled pages) is supported.

ex. # makedumpfile -E -d 1 --xen-syms xen-syms.debug vmcore output

Thanks.
Itsuro Oda

On Tue, 31 Jul 2007 07:19:48 +0900
Itsuro ODA <oda@valinux.co.jp> wrote:

> Hi,
> 
> > Is the "-d" dump_level option completely ignored, or can it
> > (without vmlinux awareness) recognize and skip zero-filled
> > pages?
> 
> Yes, "-d" is ignored.
> 
> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...
> 
> Thanks.
> -- 
> Itsuro ODA <oda@valinux.co.jp>
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

-- 
Itsuro ODA <oda@valinux.co.jp>

[-- Attachment #2: patch-zero-070810 --]
[-- Type: application/octet-stream, Size: 2219 bytes --]

--- makedumpfile.c.org	2007-08-10 15:14:22.000000000 +0900
+++ makedumpfile.c	2007-08-10 15:51:16.000000000 +0900
@@ -5195,6 +5195,9 @@
 	int i;
 	struct pt_load_segment *pls;
 	int ret = FALSE;
+	unsigned long long paddr;
+	off_t offset_page;
+	unsigned char *buf = NULL;
 
 	/*
 	 * NOTE: the first half of bitmap is not used for Xen extraction
@@ -5211,6 +5214,11 @@
 		    strerror(errno));
 		goto out;
 	}
+	if ((buf = malloc(info->page_size)) == NULL) {
+		ERRMSG("Can't allocate memory for the page. %s\n",
+		    strerror(errno));
+		goto out;
+	}
 
 	pfn = 0;
 	for (i = 0; i < info->num_load_memory; i++) {
@@ -5255,11 +5263,35 @@
 			 *  - xen heap area, or
 			 *  - selected domain page
 			 */
-			if (_domain == 0 ||
+			if (!(_domain == 0 ||
 				(info->xen_heap_start <= pfn && pfn < info->xen_heap_end) ||
-				((count_info & 0xffff) && is_select_domain(info, _domain))) {
-				set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
+				((count_info & 0xffff) && is_select_domain(info, _domain)))) {
+				continue;
+			}
+			if (info->dump_level & DL_EXCLUDE_ZERO) {
+				paddr = (unsigned long long)pfn * info->page_size;
+				offset_page = paddr_to_offset(info, paddr);
+				if (!offset_page) {
+					ERRMSG("Can't convert physaddr(%llx) to a offset.\n",
+					    paddr);
+					goto out;
+				}
+				if (lseek(info->fd_memory, offset_page,
+				    SEEK_SET) == (off_t)-1) {
+					ERRMSG("Can't seek the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (read(info->fd_memory, buf, info->page_size)
+				    != info->page_size) {
+					ERRMSG("Can't read the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (is_zero_page(buf, info->page_size))
+					continue;
 			}
+			set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
 		}
 	}
 
@@ -5275,6 +5307,8 @@
 out:
 	if (bm2.buf != NULL)
 		free(bm2.buf);
+	if (buf != NULL)
+		free(buf);
 
 	return ret;
 }
@@ -5546,7 +5580,7 @@
 			MSG("-E must be specified with --xen-syms or --xen-vmcoreinfo.\n");
 			goto out;
 		}
-		info->dump_level = DL_EXCLUDE_XEN;
+		info->dump_level |= DL_EXCLUDE_XEN;
 		return handle_xen(info);
 
 	} else if (info->flag_rearrange) {

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Itsuro ODA]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

Hi,

> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...

I found it is sometimes effective.

The attached patch enables the "-d" option with --xen-syms.
Note that only the level 1 (skip zero-filled pages) is supported.

ex. # makedumpfile -E -d 1 --xen-syms xen-syms.debug vmcore output

Thanks.
Itsuro Oda

On Tue, 31 Jul 2007 07:19:48 +0900
Itsuro ODA <oda@valinux.co.jp> wrote:

> Hi,
> 
> > Is the "-d" dump_level option completely ignored, or can it
> > (without vmlinux awareness) recognize and skip zero-filled
> > pages?
> 
> Yes, "-d" is ignored.
> 
> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...
> 
> Thanks.
> -- 
> Itsuro ODA <oda@valinux.co.jp>
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexe
-- 
Itsuro ODA <oda@valinux.co.jp>

[-- Attachment #2: patch-zero-070810 --]
[-- Type: application/octet-stream, Size: 2219 bytes --]

--- makedumpfile.c.org	2007-08-10 15:14:22.000000000 +0900
+++ makedumpfile.c	2007-08-10 15:51:16.000000000 +0900
@@ -5195,6 +5195,9 @@
 	int i;
 	struct pt_load_segment *pls;
 	int ret = FALSE;
+	unsigned long long paddr;
+	off_t offset_page;
+	unsigned char *buf = NULL;
 
 	/*
 	 * NOTE: the first half of bitmap is not used for Xen extraction
@@ -5211,6 +5214,11 @@
 		    strerror(errno));
 		goto out;
 	}
+	if ((buf = malloc(info->page_size)) == NULL) {
+		ERRMSG("Can't allocate memory for the page. %s\n",
+		    strerror(errno));
+		goto out;
+	}
 
 	pfn = 0;
 	for (i = 0; i < info->num_load_memory; i++) {
@@ -5255,11 +5263,35 @@
 			 *  - xen heap area, or
 			 *  - selected domain page
 			 */
-			if (_domain == 0 ||
+			if (!(_domain == 0 ||
 				(info->xen_heap_start <= pfn && pfn < info->xen_heap_end) ||
-				((count_info & 0xffff) && is_select_domain(info, _domain))) {
-				set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
+				((count_info & 0xffff) && is_select_domain(info, _domain)))) {
+				continue;
+			}
+			if (info->dump_level & DL_EXCLUDE_ZERO) {
+				paddr = (unsigned long long)pfn * info->page_size;
+				offset_page = paddr_to_offset(info, paddr);
+				if (!offset_page) {
+					ERRMSG("Can't convert physaddr(%llx) to a offset.\n",
+					    paddr);
+					goto out;
+				}
+				if (lseek(info->fd_memory, offset_page,
+				    SEEK_SET) == (off_t)-1) {
+					ERRMSG("Can't seek the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (read(info->fd_memory, buf, info->page_size)
+				    != info->page_size) {
+					ERRMSG("Can't read the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (is_zero_page(buf, info->page_size))
+					continue;
 			}
+			set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
 		}
 	}
 
@@ -5275,6 +5307,8 @@
 out:
 	if (bm2.buf != NULL)
 		free(bm2.buf);
+	if (buf != NULL)
+		free(buf);
 
 	return ret;
 }
@@ -5546,7 +5580,7 @@
 			MSG("-E must be specified with --xen-syms or --xen-vmcoreinfo.\n");
 			goto out;
 		}
-		info->dump_level = DL_EXCLUDE_XEN;
+		info->dump_level |= DL_EXCLUDE_XEN;
 		return handle_xen(info);
 
 	} else if (info->flag_rearrange) {

[-- Attachment #3: Type: text/plain, Size: 143 bytes --]

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Itsuro ODA]

Hi,

> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...

I found it is sometimes effective.

The attached patch enables the "-d" option with --xen-syms.
Note that only the level 1 (skip zero-filled pages) is supported.

ex. # makedumpfile -E -d 1 --xen-syms xen-syms.debug vmcore output

Thanks.
Itsuro Oda

On Tue, 31 Jul 2007 07:19:48 +0900
Itsuro ODA <oda@valinux.co.jp> wrote:

> Hi,
> 
> > Is the "-d" dump_level option completely ignored, or can it
> > (without vmlinux awareness) recognize and skip zero-filled
> > pages?
> 
> Yes, "-d" is ignored.
> 
> Hmm, adding to skip zero-filled pages is easy but I wonder
> how effective it is ...
> 
> Thanks.
> -- 
> Itsuro ODA <oda@valinux.co.jp>
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexe
-- 
--- makedumpfile.c.org	2007-08-10 15:14:22.000000000 +0900
+++ makedumpfile.c	2007-08-10 15:51:16.000000000 +0900
@@ -5195,6 +5195,9 @@
 	int i;
 	struct pt_load_segment *pls;
 	int ret = FALSE;
+	unsigned long long paddr;
+	off_t offset_page;
+	unsigned char *buf = NULL;
 
 	/*
 	 * NOTE: the first half of bitmap is not used for Xen extraction
@@ -5211,6 +5214,11 @@
 		    strerror(errno));
 		goto out;
 	}
+	if ((buf = malloc(info->page_size)) == NULL) {
+		ERRMSG("Can't allocate memory for the page. %s\n",
+		    strerror(errno));
+		goto out;
+	}
 
 	pfn = 0;
 	for (i = 0; i < info->num_load_memory; i++) {
@@ -5255,11 +5263,35 @@
 			 *  - xen heap area, or
 			 *  - selected domain page
 			 */
-			if (_domain == 0 ||
+			if (!(_domain == 0 ||
 				(info->xen_heap_start <= pfn && pfn < info->xen_heap_end) ||
-				((count_info & 0xffff) && is_select_domain(info, _domain))) {
-				set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
+				((count_info & 0xffff) && is_select_domain(info, _domain)))) {
+				continue;
+			}
+			if (info->dump_level & DL_EXCLUDE_ZERO) {
+				paddr = (unsigned long long)pfn * info->page_size;
+				offset_page = paddr_to_offset(info, paddr);
+				if (!offset_page) {
+					ERRMSG("Can't convert physaddr(%llx) to a offset.\n",
+					    paddr);
+					goto out;
+				}
+				if (lseek(info->fd_memory, offset_page,
+				    SEEK_SET) == (off_t)-1) {
+					ERRMSG("Can't seek the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (read(info->fd_memory, buf, info->page_size)
+				    != info->page_size) {
+					ERRMSG("Can't read the dump memory(%s). %s\n",
+					    info->name_memory, strerror(errno));
+					goto out;
+				}
+				if (is_zero_page(buf, info->page_size))
+					continue;
 			}
+			set_bitmap(bm2.buf, pfn%PFN_BUFBITMAP, 1);
 		}
 	}
 
@@ -5275,6 +5307,8 @@
 out:
 	if (bm2.buf != NULL)
 		free(bm2.buf);
+	if (buf != NULL)
+		free(buf);
 
 	return ret;
 }
@@ -5546,7 +5580,7 @@
 			MSG("-E must be specified with --xen-syms or --xen-vmcoreinfo.\n");
 			goto out;
 		}
-		info->dump_level = DL_EXCLUDE_XEN;
+		info->dump_level |= DL_EXCLUDE_XEN;
 		return handle_xen(info);
 
 	} else if (info->flag_rearrange) {


-- 
Itsuro ODA <oda@valinux.co.jp>


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.1.5: Xen support

[Ken'ichi Ohmichi]

Hi ODA-san,

2007/08/14 08:27:17 +0900, Itsuro ODA <oda@valinux.co.jp> wrote:
>Hi,
>
>> Hmm, adding to skip zero-filled pages is easy but I wonder
>> how effective it is ...
>
>I found it is sometimes effective.
>
>The attached patch enables the "-d" option with --xen-syms.
>Note that only the level 1 (skip zero-filled pages) is supported.
>
>ex. # makedumpfile -E -d 1 --xen-syms xen-syms.debug vmcore output

Thank you for a good patch.
I will release a new version with your patch.


Thanks
Ken'ichi Ohmichi

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec