[REQUEST] makedumpfile: stream compress flat ELF format with libz

[REQUEST] makedumpfile: stream compress flat ELF format with libz

[Daniel Walker (danielwa)]

Hi,

I created an enhancement request for makedumpfile here,

https://github.com/makedumpfile/makedumpfile/issues/1

I found that compressing a flat core with gzip significantly reduces the size of
the core. Here were my findings,

32G flat elf core -E -F -d 0
33G kdump core -d 0
16G kdump compressed -c -d 0
1.9G flat elf core stream compressed with gzip -E -F -d 0

My feature request was to implement an option inside makedumpfile to gzip
compress the core output. This can already be accomplished by piping the core
thru the gzip tool, however, because makedumpfile already links against libz
having the option for makedumpfile to do it allows the gzip tools to be removed
from the crash kernels initramfs there by reducing the size.

Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do
this.

So a resulting command line might looks like this,

makedumpfile -C -F -E -d 31 /proc/vmcore core.gz

Thanks,
Daniel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[John Donnelly]

> On May 12, 2020, at 12:25 PM, Daniel Walker (danielwa) <danielwa@cisco.com> wrote:
> 
> 
> Hi,
> 
> I created an enhancement request for makedumpfile here,
> 
> https://urldefense.com/v3/__https://github.com/makedumpfile/makedumpfile/issues/1__;!!GqivPVa7Brio!LJIWfQ8ged-9RjjV00zqmBGbL2-UU0baDJmxqVXo5AxYcofHzP8oxfWHZZ4ijk6O9N9I$ 
> 
> I found that compressing a flat core with gzip significantly reduces the size of
> the core. Here were my findings,
> 
> 32G flat elf core -E -F -d 0
> 33G kdump core -d 0
> 16G kdump compressed -c -d 0
> 1.9G flat elf core stream compressed with gzip -E -F -d 0
> 
> My feature request was to implement an option inside makedumpfile to gzip
> compress the core output. This can already be accomplished by piping the core
> thru the gzip tool, however, because makedumpfile already links against libz
> having the option for makedumpfile to do it allows the gzip tools to be removed
> from the crash kernels initramfs there by reducing the size.
> 
> Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do
> this.
> 


  Hi Daniel. 

   -z happens to be used by tar and rsync to indicate compression ;-) . 





> So a resulting command line might looks like this,
> 
> makedumpfile -C -F -E -d 31 /proc/vmcore core.gz
> 
> Thanks,
> Daniel
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> https://urldefense.com/v3/__http://lists.infradead.org/mailman/listinfo/kexec__;!!GqivPVa7Brio!LJIWfQ8ged-9RjjV00zqmBGbL2-UU0baDJmxqVXo5AxYcofHzP8oxfWHZZ4ijsPXKz5A$ 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[Daniel Walker (danielwa)]

On Tue, May 12, 2020 at 12:38:42PM -0500, John Donnelly wrote:
 
> 
>   Hi Daniel. 
> 
>    -z happens to be used by tar and rsync to indicate compression ;-) . 
> 
> 

This would be fine with me,

makedumpfile -z -F -E -d 31 /proc/vmcore core.gz


Daniel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[Guilherme Piccoli]

It's a quite interesting feature Daniel, thanks for the effort!
I'm curious about memory usage - did you somehow measure the
consumption with your patches vs. the regular kdump compression?

Thanks,


Guilherme

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[Daniel Walker (danielwa)]

On Tue, May 12, 2020 at 03:14:49PM -0300, Guilherme Piccoli wrote:
> It's a quite interesting feature Daniel, thanks for the effort!
> I'm curious about memory usage - did you somehow measure the
> consumption with your patches vs. the regular kdump compression?

It's not really a memory consumption issue, it's a core size issue. The core
size is much smaller when compressed with gzip. I measured the core size with
different formats and gzip compressed.

Daniel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[Guilherme Piccoli]

Sure, I understood that. I'm curious though about the memory
consumption of the compression operation, or put in other words: do we
need to increase crashkernel reserved memory if we plan to use your
gzip approach?
It's good to be sure about this, to evaluate the trade-off of core
file size vs. necessary reserved memory.

Thanks!

On Tue, May 12, 2020 at 3:27 PM Daniel Walker (danielwa)
<danielwa@cisco.com> wrote:
>
> On Tue, May 12, 2020 at 03:14:49PM -0300, Guilherme Piccoli wrote:
> > It's a quite interesting feature Daniel, thanks for the effort!
> > I'm curious about memory usage - did you somehow measure the
> > consumption with your patches vs. the regular kdump compression?
>
> It's not really a memory consumption issue, it's a core size issue. The core
> size is much smaller when compressed with gzip. I measured the core size with
> different formats and gzip compressed.
>
> Daniel

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[Daniel Walker (danielwa)]

On Tue, May 12, 2020 at 03:32:25PM -0300, Guilherme Piccoli wrote:
> Sure, I understood that. I'm curious though about the memory
> consumption of the compression operation, or put in other words: do we
> need to increase crashkernel reserved memory if we plan to use your
> gzip approach?
> It's good to be sure about this, to evaluate the trade-off of core
> file size vs. necessary reserved memory.

I'm not sure. I didn't measure this, but from my experience makedumpfile uses
more memory during dumping than gzip uses for compression. If our dump kernels
run out of memory it's usually something inside makedumpfile when it's removing
pages or something along those lines.

Daniel
_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

RE: [REQUEST] makedumpfile: stream compress flat ELF format with libz

[HAGIO KAZUHITO(萩尾 一仁)]

> -----Original Message-----
>
> Hi,
>
> I created an enhancement request for makedumpfile here,
>
> https://github.com/makedumpfile/makedumpfile/issues/1
>
> I found that compressing a flat core with gzip significantly reduces the size of
> the core. Here were my findings,
>
> 32G flat elf core -E -F -d 0
> 33G kdump core -d 0
> 16G kdump compressed -c -d 0
> 1.9G flat elf core stream compressed with gzip -E -F -d 0

The kdump-compressed format does per-page compression with compression level 1,
so depending on data in vmcore, gzip (with -6 by default) can compress one much
smaller like this.  Although I don't have a vmcore that shows such a drop in size
with gzip as far as I tested.

>
> My feature request was to implement an option inside makedumpfile to gzip
> compress the core output. This can already be accomplished by piping the core
> thru the gzip tool, however, because makedumpfile already links against libz
> having the option for makedumpfile to do it allows the gzip tools to be removed
> from the crash kernels initramfs there by reducing the size.

(personally I prefer piping it through gzip, but apparently there are cases where
the initramfs size needs to be minimized..)

>
> Kazuhito Hagio had suggest adding the -C option instead of repurposing -c to do
> this.
>
> So a resulting command line might looks like this,
>
> makedumpfile -C -F -E -d 31 /proc/vmcore core.gz

Yes, the existing -c option is for per-page compression and creates a dumpfile that
can be opened directly with crash utility, but stream compression is different.
So I'd like to separate it from the -c.

(as John commented, -z also might be good, but I wonder if someone might start to
want the same function with lzo or snappy..)

Anyway, I'll wait for patches for this enhancement.

Thanks,
Kazu

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec