makedumpfile-1.5.9: Introduce parallel compression processing.

makedumpfile-1.5.9: Introduce parallel compression processing.

[Atsushi Kumagai]

Hello,

makedumpfile version 1.5.9 is released.
Your comments/patches are welcome.

Main new feature:
o Parallel compression
 - Add the new option "--num-threads" to enable parallel compression
   for faster processing. This option intends parallel compression
   while --split intends parallel i/o, they try to solve different
   bottlenecks.

o Change the way to choose the non-cyclic mode
 - Add the new option "--work-dir" to specify the directory used to
   store the bitmap. This is the alternative to the combination of
   --non-cyclic and TMPDIR.

o Support new kernels
 - The supported kernel is updated to 4.1 in this version.
   At first I'm going to extend the supported version to 4.2, but
   I found an issue that makedumpfile seems to exclude necessary pages
   by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
   the following message is shown.

      crash: page excluded: kernel virtual address: f3fe0000  type: "fill_task_struct"

   This will be fixed in the next version.

Changelog:
o New feature
    - [PATCH] Support ARM64. (by Pratyush Anand) 94d47d1
    - [PATCH 1/2] Enable compressed dump formats for Xen. (by Petr Tesarik) 349a0ed
    - [PATCH 2/2] Remove notes about ELF being the only available format for Xen dumps.
      (by Petr Tesarik) 5cab480
    - [PATCH v5 01/10] Add readpage_kdump_compressed_parallel. (by Qiao Nuohan) 5fab021
    - [PATCH v5 02/10] Add mappage_elf_parallel. (by Qiao Nuohan) edca232
    - [PATCH v5 03/10] Add readpage_elf_parallel. (by Qiao Nuohan) 7f15b41
    - [PATCH v5 04/10] Add read_pfn_parallel. (by Qiao Nuohan) a7243fe
    - [PATCH v5 05/10] Add function to initial bitmap for parallel use. (by Qiao Nuohan) 56a6bc4
    - [PATCH v5 06/10] Add filter_data_buffer_parallel. (by Qiao Nuohan) 2a7cb9f
    - [PATCH v5 07/10] Add write_kdump_pages_parallel to allow parallel process. (by Qiao Nuohan) c2595d9
    - [PATCH v5 08/10] Initial and free data used for parallel process. (by Qiao Nuohan) 4538848
    - [PATCH v5 09/10] Make makedumpfile available to read and compress pages parallelly.
      (by Qiao Nuohan) b265102
    - [PATCH v5 10/10] Add usage and manual about multiple threads process. (by Qiao Nuohan) b18a8a6

o Bugfix
    - [PATCH] Correct vmap_area_list support for i386 and ppc32. (by Atsushi Kumagai) 4194d7b
    - [PATCH] Free pages aren't removed in non-cyclic mode. (by Atsushi Kumagai) d64cf7f
    - [PATCH v2] Fix a bug in generating incomplete kdump core. (by Zhou Wenjian) 9cee495
    - [PATCH] Remove the double free of sph. (by Chao Fan) 72dec41
    - [PATCH 1/2] sadump: Change bit order. (by HATAYAMA Daisuke) 5f15256
    - [PATCH 2/2] sadump: Perform explicit zero page filtering. (by HATAYAMA Daisuke) 19b3a91

o Cleanup
    - [PATCH 01/13] Organize bitmap structure for cyclic logic. (by Atsushi Kumagai) 5fc24bf
    - [PATCH 02/13] Add option to specify working directory for the bitmap. (by Atsushi Kumagai) 5c5a8b8
    - [PATCH 03/13] Integrate the entry point of is_dumpable(). (by Atsushi Kumagai) 5c8c680
    - [PATCH 04/13] Integrate the main logic of writing kdump file. (by Atsushi Kumagai) d18796d
    - [PATCH 05/13] Communalize the function for creating 1st bitmap. (by Atsushi Kumagai) b00a7b0
    - [PATCH 06/13] Remove the old logic of writing kdump pages. (by Atsushi Kumagai) 4d55542
    - [PATCH 07/13] Integrate filtering process for ELF path. (by Atsushi Kumagai) 679a818
    - [PATCH 08/13] Remove the old logic of writing ELF pages. (by Atsushi Kumagai) 672354b
    - [PATCH 09/13] Adjust --mem-usage path to the new code. (by Atsushi Kumagai) 57153b0
    - [PATCH 10/13] Adjust --split/--reassemble path to the new code. (by Atsushi Kumagai) 95435de
    - [PATCH 11/13] Adjust refiltering path to the new code. (by Atsushi Kumagai) c7fb41c
    - [PATCH 12/13] Adjust sadump path to the new code. (by Atsushi Kumagai) edd2bf2
    - [PATCH 13/13] Remove --non-cyclic option. (by Atsushi Kumagai) a117ae9
    - [PATCH] Optimise the function reserve_diskspace. (by Zhou Wenjian) 3605073
    - [PATCH v2] Add description of pages lost by ENOSPACE in IMPLEMENTATION. (by Zhou Wenjian) 0e20f33
    - [PATCH 1/2] Enable compressed dump formats for Xen. (by Petr Tesarik) 349a0ed
    - [PATCH 2/2] Remove notes about ELF being the only available format for Xen dumps.
      (by Petr Tesarik) 5cab480
    - [PATCH v2] Code changes to satisfy the coverity scan. (by Chao Fan) 0c65e38
    - [PATCH] Improve performance for parallel compression with zlib. (by Atsushi Kumagai) 3ef6629

Explanation of makedumpfile:
  To shorten the size of the dumpfile and the time of creating the
  dumpfile, makedumpfile copies only the necessary pages for analysis
  to the dumpfile from /proc/vmcore. You can specify the kind of
  unnecessary pages with dump_level. If you want to shorten the size
  further, enable the compression of the page data.

Download:
  You can download the latest makedumpfile from the following URL.
  Details of the change are written on the git page of the following site.
  https://sourceforge.net/projects/makedumpfile/

Method of installation:
  You can compile the makedumpfile command as follows;
  1. "tar -zxvf makedumpfile-x.y.z.tar.gz"
  2. "cd makedumpfile-x.y.z"
  3. "make; make install"

Usage:
  makedumpfile [-c] [-E] [-d dump_level] [-x vmlinux] dump_mem dump_file

Example:
  If you want to exclude pages filled by zero, cache pages, user pages
  and free pages and to enable compression, please execute the following
  command.	

  # makedumpfile -c -d 31 /proc/vmcore dumpfile


Thanks
Atsushi Kumagai

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.5.9: Introduce parallel compression processing.

[Louis Bouchard]

Hello Kumagai-san,

Le 28/10/2015 07:31, Atsushi Kumagai a écrit :
> Hello,
> 
> makedumpfile version 1.5.9 is released.
> Your comments/patches are welcome.
> 
> Main new feature:
> o Parallel compression
>  - Add the new option "--num-threads" to enable parallel compression
>    for faster processing. This option intends parallel compression
>    while --split intends parallel i/o, they try to solve different
>    bottlenecks.
> 
> o Change the way to choose the non-cyclic mode
>  - Add the new option "--work-dir" to specify the directory used to
>    store the bitmap. This is the alternative to the combination of
>    --non-cyclic and TMPDIR.
> 
> o Support new kernels
>  - The supported kernel is updated to 4.1 in this version.
>    At first I'm going to extend the supported version to 4.2, but
>    I found an issue that makedumpfile seems to exclude necessary pages
>    by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
>    the following message is shown.
> 
>       crash: page excluded: kernel virtual address: f3fe0000  type: "fill_task_struct"
> 
>    This will be fixed in the next version.

Have you fixed the 4.2 kernel issue already or you intend to work on it in the
future ?

Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is acceptable I
would rather bring in a 1.5.9 + patch version.

Kind regards,


...Louis


-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: makedumpfile-1.5.9: Introduce parallel compression processing.

[Atsushi Kumagai]

Hello Louis,

>> o Support new kernels
>>  - The supported kernel is updated to 4.1 in this version.
>>    At first I'm going to extend the supported version to 4.2, but
>>    I found an issue that makedumpfile seems to exclude necessary pages
>>    by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
>>    the following message is shown.
>>
>>       crash: page excluded: kernel virtual address: f3fe0000  type: "fill_task_struct"
>>
>>    This will be fixed in the next version.
>
>Have you fixed the 4.2 kernel issue already or you intend to work on it in the
>future ?

I haven't done anything yet, I was going to work from now.

>Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is acceptable I
>would rather bring in a 1.5.9 + patch version.

So it will be helpful if you can do it.


Thanks,
Atsushi Kumagai
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.5.9: Introduce parallel compression processing.

[Louis Bouchard]

Hi,

Le 28/10/2015 10:07, Atsushi Kumagai a écrit :
> Hello Louis,
> 
>>> o Support new kernels
>>>  - The supported kernel is updated to 4.1 in this version.
>>>    At first I'm going to extend the supported version to 4.2, but
>>>    I found an issue that makedumpfile seems to exclude necessary pages
>>>    by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
>>>    the following message is shown.
>>>
>>>       crash: page excluded: kernel virtual address: f3fe0000  type: "fill_task_struct"
>>>
>>>    This will be fixed in the next version.
>>
>> Have you fixed the 4.2 kernel issue already or you intend to work on it in the
>> future ?
> 
> I haven't done anything yet, I was going to work from now.
> 
>> Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is acceptable I
>> would rather bring in a 1.5.9 + patch version.
> 
> So it will be helpful if you can do it.
> 
> 
> Thanks,
> Atsushi Kumagai
> 

That is perfect.  Just let me know once you have identified a fix and I will
include it in the new package.

Kind regards,

...Louis
-- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: makedumpfile-1.5.9: Introduce parallel compression processing.

[Atsushi Kumagai]

Hello Louis,

>>>> o Support new kernels
>>>>  - The supported kernel is updated to 4.1 in this version.
>>>>    At first I'm going to extend the supported version to 4.2, but
>>>>    I found an issue that makedumpfile seems to exclude necessary pages
>>>>    by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
>>>>    the following message is shown.
>>>>
>>>>       crash: page excluded: kernel virtual address: f3fe0000  type: "fill_task_struct"
>>>>
>>>>    This will be fixed in the next version.
>>>
>>> Have you fixed the 4.2 kernel issue already or you intend to work on it in the
>>> future ?
>>
>> I haven't done anything yet, I was going to work from now.
>>
>>> Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is acceptable I
>>> would rather bring in a 1.5.9 + patch version.
>>
>> So it will be helpful if you can do it.
>>
>>
>> Thanks,
>> Atsushi Kumagai
>>
>
>That is perfect.  Just let me know once you have identified a fix and I will
>include it in the new package.

I confirmed that there is no problem in makedumpfile, so you don't need to
update your makedumpfile packages.

That was a crash side issue and the problem is still under discussion.
The warning I reported first is going to be solved in the next crash version,
but "struct" and "task" commands with task_struct (or some corresponding fields)
can fail if a dumpfile of linux 4.2 is filtered.

You should see this thread for more details:

  https://www.redhat.com/archives/crash-utility/2015-November/msg00021.html


Thanks,
Atsushi Kumagai
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: makedumpfile-1.5.9: Introduce parallel compression processing.

[Louis Bouchard]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello Kumagai-san,

Le 19/11/2015 08:30, Atsushi Kumagai a écrit :
> Hello Louis,
> 
>>>>> o Support new kernels - The supported kernel is updated to 4.1 in
>>>>> this version. At first I'm going to extend the supported version to
>>>>> 4.2, but I found an issue that makedumpfile seems to exclude
>>>>> necessary pages by mistake on linux 4.2. When crash-7.1.3 reads
>>>>> that filtered dump file, the following message is shown.
>>>>> 
>>>>> crash: page excluded: kernel virtual address: f3fe0000  type:
>>>>> "fill_task_struct"
>>>>> 
>>>>> This will be fixed in the next version.
>>>> 
>>>> Have you fixed the 4.2 kernel issue already or you intend to work on
>>>> it in the future ?
>>> 
>>> I haven't done anything yet, I was going to work from now.
>>> 
>>>> Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is
>>>> acceptable I would rather bring in a 1.5.9 + patch version.
>>> 
>>> So it will be helpful if you can do it.
>>> 
>>> 
>>> Thanks, Atsushi Kumagai
>>> 
>> 
>> That is perfect.  Just let me know once you have identified a fix and I
>> will include it in the new package.
> 
> I confirmed that there is no problem in makedumpfile, so you don't need to 
> update your makedumpfile packages.
> 
> That was a crash side issue and the problem is still under discussion. The
> warning I reported first is going to be solved in the next crash version, 
> but "struct" and "task" commands with task_struct (or some corresponding
> fields) can fail if a dumpfile of linux 4.2 is filtered.
> 
> You should see this thread for more details:
> 
> https://www.redhat.com/archives/crash-utility/2015-November/msg00021.html
> 
> 
> Thanks, Atsushi Kumagai
> 

As I understand it, 1.5.9 will work fine on 4.2 kernels.  But as the released
source stand, 1.5.9 only supports up to 4.1.  Should I simply go ahead and
"manually" add support statement for 4.2 (in README & makedumpfile.h) or you
prefer to change it in your git tree ?

Kind regards,

...Louis

- -- 
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer                       Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63  B9C4 8B3D 867C 823E 7A61
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWTvR9AAoJEFZStL2qfwOthLEP/A042ORfYD6vfKTM7WWjW5Yh
upEkZQKms7I2HzGGWhC1x1TtJAmvWOK8G7Vou+ULpKMjH8byRMivP7ioTSqtw24J
aTiwLiC+2K2QleasjeYtY5pY3OOyhIsd7XOIxfcvuPa46ssmsGevnq6ZFB06s5nx
HrkfYQRuT6uLQRqpem61N8gykngABJ25b7REGYhjsSl0OZNwz1YBlpHtQqgEXtm2
Cr17q2yTyV8t05LgQ35JMEoIRbgw4bKobMhV2yP0HnZjHyXVoeEY4VhBwU9Aor4A
laycOk5u0bShqEIoeweBSdW+2qMld2kM/y3uGABOpByZgQSmNVtmBzXaMPPHELRd
XtghNxOPvRLa5ukw6ebBI+fRMuxxd6K/OpmEk2p2fVlfBxBwFHQSQy+lp9WBKF/r
ZDPiIXq1xYlNS5eGQcAXsT+48jaR656rxkuEv5RQoNNMv/KVS2l0KtE10umfTjJY
72A2g240gRzSIQzBwUaheKC/FDBRjL2NETTNvnU3iGWMdctAsKd3kck5CKIooPX5
CUVeMndqEenfWMrOjLGsfWJpwW+vHMI0Q0wGUn/RMk733r34rZYHN3m86Ph2cRAh
RLQYONyDQGp1ig4KKmTwkOgKIoR7vbeWqbOeg3jMaO8VNDJXUmRQYEGfzf3bamcK
g0txtgExdvXsvgoeAHri
=Dv4d
-----END PGP SIGNATURE-----

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: makedumpfile-1.5.9: Introduce parallel compression processing.

[Atsushi Kumagai]

>>>>>> o Support new kernels - The supported kernel is updated to 4.1 in
>>>>>> this version. At first I'm going to extend the supported version to
>>>>>> 4.2, but I found an issue that makedumpfile seems to exclude
>>>>>> necessary pages by mistake on linux 4.2. When crash-7.1.3 reads
>>>>>> that filtered dump file, the following message is shown.
>>>>>>
>>>>>> crash: page excluded: kernel virtual address: f3fe0000  type:
>>>>>> "fill_task_struct"
>>>>>>
>>>>>> This will be fixed in the next version.
>>>>>
>>>>> Have you fixed the 4.2 kernel issue already or you intend to work on
>>>>> it in the future ?
>>>>
>>>> I haven't done anything yet, I was going to work from now.
>>>>
>>>>> Both Debian and Ubuntu run 4.2 kernel so if waiting for a fix is
>>>>> acceptable I would rather bring in a 1.5.9 + patch version.
>>>>
>>>> So it will be helpful if you can do it.
>>>>
>>>>
>>>> Thanks, Atsushi Kumagai
>>>>
>>>
>>> That is perfect.  Just let me know once you have identified a fix and I
>>> will include it in the new package.
>>
>> I confirmed that there is no problem in makedumpfile, so you don't need to
>> update your makedumpfile packages.
>>
>> That was a crash side issue and the problem is still under discussion. The
>> warning I reported first is going to be solved in the next crash version,
>> but "struct" and "task" commands with task_struct (or some corresponding
>> fields) can fail if a dumpfile of linux 4.2 is filtered.
>>
>> You should see this thread for more details:
>>
>> https://www.redhat.com/archives/crash-utility/2015-November/msg00021.html
>>
>>
>> Thanks, Atsushi Kumagai
>>
>
>As I understand it, 1.5.9 will work fine on 4.2 kernels.  But as the released
>source stand, 1.5.9 only supports up to 4.1.  Should I simply go ahead and
>"manually" add support statement for 4.2 (in README & makedumpfile.h) or you
>prefer to change it in your git tree ?

I've updated the git tree, you can pick up the commit:

commit 1c01d4bfb383c10a97e256f4a6cbf1d92ed133f6
Author: Atsushi Kumagai <ats-kumagai@wm.jp.nec.com>
Date:   Tue Nov 24 10:50:07 2015 +0900

    [PATCH] Support newer kernels.

    A new makedumpfile supports newer kernels:

          - 4.2    (x86 FLATMEM)
          - 4.2    (x86 SPARSEMEM)
          - 4.2    (x86_64 SPARSEMEM)

    Signed-off-by: Atsushi Kumagai <ats-kumagai@wm.jp.nec.com>


Thanks,
Atsushi Kumagai
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec