Reducing the size of the dump file/speeding up collection

Reducing the size of the dump file/speeding up collection

[Nikolay Borisov]

Hello,

I've been using makedumpfile as the crash collector with the -d31
parameter. The machine this is being run on usually have 128-256GB of
ram and the resulting crash dumps are in the range of 14-20gb which is
very bug for the type of analysis I'm usually performing on crashed
machine. I was wondering whether there is a way  to further reduce the
size and the time to take the dump (now it takes around 25 minutes to
collect such a dump). I've seen reports where people with TBs of ram
take that long, meaning for a machine with 256gb it should be even
faster. I've been running this configuration on kernels 3.12.28 and 4.1
where mmap for the vmcore file is supported.

Please advise

Regards,
Nikolay

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[qiaonuohan]

On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
> Hello,
>
> I've been using makedumpfile as the crash collector with the -d31
> parameter. The machine this is being run on usually have 128-256GB of
> ram and the resulting crash dumps are in the range of 14-20gb which is
> very bug for the type of analysis I'm usually performing on crashed
> machine. I was wondering whether there is a way  to further reduce the
> size and the time to take the dump (now it takes around 25 minutes to
> collect such a dump). I've seen reports where people with TBs of ram
> take that long, meaning for a machine with 256gb it should be even
> faster. I've been running this configuration on kernels 3.12.28 and 4.1
> where mmap for the vmcore file is supported.
>
> Please advise

Hi nikolay,

Yes, this issue is what we are concerning a lot.
About the current situation, try --split, it will save time.


And lzo/snappy instead of zlib, these two compression format are faster
but need more space to save. Or if you still want zlib (to save space),
try multiple threads, check the following site, it will help you:

https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html


-- 
Regards
Qiao Nuohan

>
> Regards,
> Nikolay
>
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec
>



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[Nikolay Borisov]

Hi Qiao,

Thanks for the reply. So far I haven't been using the the compression
feature of makedumpfile. But I want to ask if anything wouldn't
compression make the dump process slower since in addition to having to
write the dump to disk it also has to compress it which would put more
strain on the cpu. Also, which part of the dump process is the bottleneck:

- Reading from /proc/vmcore - that has mmap support so should be fairly
fast?
- Discarding unnecessary pages as memory is being scanned?
- Writing/compressing content to disk?

Regards,
Nikolay

On 09/17/2015 06:27 AM, qiaonuohan wrote:
> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
>> Hello,
>>
>> I've been using makedumpfile as the crash collector with the -d31
>> parameter. The machine this is being run on usually have 128-256GB of
>> ram and the resulting crash dumps are in the range of 14-20gb which is
>> very bug for the type of analysis I'm usually performing on crashed
>> machine. I was wondering whether there is a way  to further reduce the
>> size and the time to take the dump (now it takes around 25 minutes to
>> collect such a dump). I've seen reports where people with TBs of ram
>> take that long, meaning for a machine with 256gb it should be even
>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
>> where mmap for the vmcore file is supported.
>>
>> Please advise
> 
> Hi nikolay,
> 
> Yes, this issue is what we are concerning a lot.
> About the current situation, try --split, it will save time.
> 
> 
> And lzo/snappy instead of zlib, these two compression format are faster
> but need more space to save. Or if you still want zlib (to save space),
> try multiple threads, check the following site, it will help you:
> 
> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
> 
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[Nikolay Borisov]

Just to follow up: which version of kexec/makedumpfile are required to
have the parallel dump feature? The thread you referenced adds
documentation how to use that and I grepped the source of makedumpfile
1.5.8 and couldn't find any references to the num-threads option? So
what version of the code do I need to test that ?

On 09/17/2015 09:32 AM, Nikolay Borisov wrote:
> Hi Qiao,
> 
> Thanks for the reply. So far I haven't been using the the compression
> feature of makedumpfile. But I want to ask if anything wouldn't
> compression make the dump process slower since in addition to having to
> write the dump to disk it also has to compress it which would put more
> strain on the cpu. Also, which part of the dump process is the bottleneck:
> 
> - Reading from /proc/vmcore - that has mmap support so should be fairly
> fast?
> - Discarding unnecessary pages as memory is being scanned?
> - Writing/compressing content to disk?
> 
> Regards,
> Nikolay
> 
> On 09/17/2015 06:27 AM, qiaonuohan wrote:
>> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
>>> Hello,
>>>
>>> I've been using makedumpfile as the crash collector with the -d31
>>> parameter. The machine this is being run on usually have 128-256GB of
>>> ram and the resulting crash dumps are in the range of 14-20gb which is
>>> very bug for the type of analysis I'm usually performing on crashed
>>> machine. I was wondering whether there is a way  to further reduce the
>>> size and the time to take the dump (now it takes around 25 minutes to
>>> collect such a dump). I've seen reports where people with TBs of ram
>>> take that long, meaning for a machine with 256gb it should be even
>>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
>>> where mmap for the vmcore file is supported.
>>>
>>> Please advise
>>
>> Hi nikolay,
>>
>> Yes, this issue is what we are concerning a lot.
>> About the current situation, try --split, it will save time.
>>
>>
>> And lzo/snappy instead of zlib, these two compression format are faster
>> but need more space to save. Or if you still want zlib (to save space),
>> try multiple threads, check the following site, it will help you:
>>
>> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
>>
>>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[qiaonuohan]

On 09/17/2015 02:32 PM, Nikolay Borisov wrote:
> Hi Qiao,
>
> Thanks for the reply. So far I haven't been using the the compression
> feature of makedumpfile. But I want to ask if anything wouldn't
> compression make the dump process slower since in addition to having to
> write the dump to disk it also has to compress it which would put more
> strain on the cpu. Also, which part of the dump process is the bottleneck:
>
> - Reading from /proc/vmcore - that has mmap support so should be fairly
> fast?
> - Discarding unnecessary pages as memory is being scanned?
> - Writing/compressing content to disk?

I cannot recall percentage of each part. But writing/compression takes most
of the time

1. mmap is used for reading faster
2. --split is used to split the dump task into several processes, so compressing
    and writing will be speeded up.
3. multiple-thread is another option for speeding up compressing, it is a recently
    committed patch, so you cannot find it in the master branch, checkout devel branch
    or find it here:

http://sourceforge.net/p/makedumpfile/code/commit_browser

Make makedumpfile available to read and compress pages parallelly.

>
> Regards,
> Nikolay
>
> On 09/17/2015 06:27 AM, qiaonuohan wrote:
>> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
>>> Hello,
>>>
>>> I've been using makedumpfile as the crash collector with the -d31
>>> parameter. The machine this is being run on usually have 128-256GB of
>>> ram and the resulting crash dumps are in the range of 14-20gb which is
>>> very bug for the type of analysis I'm usually performing on crashed
>>> machine. I was wondering whether there is a way  to further reduce the
>>> size and the time to take the dump (now it takes around 25 minutes to
>>> collect such a dump). I've seen reports where people with TBs of ram
>>> take that long, meaning for a machine with 256gb it should be even
>>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
>>> where mmap for the vmcore file is supported.
>>>
>>> Please advise
>>
>> Hi nikolay,
>>
>> Yes, this issue is what we are concerning a lot.
>> About the current situation, try --split, it will save time.
>>
>>
>> And lzo/snappy instead of zlib, these two compression format are faster
>> but need more space to save. Or if you still want zlib (to save space),
>> try multiple threads, check the following site, it will help you:
>>
>> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
>>
>>
> .
>


-- 
Regards
Qiao Nuohan

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[Nikolay Borisov]

Yeah, I did see the commit browser. But in my case I haven't even tested
the split option so I guess there are things to try. Am I correct in my
understanding as to how --split is supposed to work ( i tried that to no
avail though):

My core_collector line is this:

core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2
dump3 dump4 dump5 dump6

And then in /etc/sysconfig/kdump I have:

KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices
cgroup_disable=memory mce=off"

(the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical
cores in total). Do I need to do something else to utilize the --split
option?

On 09/18/2015 05:38 AM, qiaonuohan wrote:
> On 09/17/2015 02:32 PM, Nikolay Borisov wrote:
>> Hi Qiao,
>>
>> Thanks for the reply. So far I haven't been using the the compression
>> feature of makedumpfile. But I want to ask if anything wouldn't
>> compression make the dump process slower since in addition to having to
>> write the dump to disk it also has to compress it which would put more
>> strain on the cpu. Also, which part of the dump process is the
>> bottleneck:
>>
>> - Reading from /proc/vmcore - that has mmap support so should be fairly
>> fast?
>> - Discarding unnecessary pages as memory is being scanned?
>> - Writing/compressing content to disk?
> 
> I cannot recall percentage of each part. But writing/compression takes most
> of the time
> 
> 1. mmap is used for reading faster
> 2. --split is used to split the dump task into several processes, so
> compressing
>    and writing will be speeded up.
> 3. multiple-thread is another option for speeding up compressing, it is
> a recently
>    committed patch, so you cannot find it in the master branch, checkout
> devel branch
>    or find it here:
> 
> http://sourceforge.net/p/makedumpfile/code/commit_browser
> 
> Make makedumpfile available to read and compress pages parallelly.
> 
>>
>> Regards,
>> Nikolay
>>
>> On 09/17/2015 06:27 AM, qiaonuohan wrote:
>>> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
>>>> Hello,
>>>>
>>>> I've been using makedumpfile as the crash collector with the -d31
>>>> parameter. The machine this is being run on usually have 128-256GB of
>>>> ram and the resulting crash dumps are in the range of 14-20gb which is
>>>> very bug for the type of analysis I'm usually performing on crashed
>>>> machine. I was wondering whether there is a way  to further reduce the
>>>> size and the time to take the dump (now it takes around 25 minutes to
>>>> collect such a dump). I've seen reports where people with TBs of ram
>>>> take that long, meaning for a machine with 256gb it should be even
>>>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
>>>> where mmap for the vmcore file is supported.
>>>>
>>>> Please advise
>>>
>>> Hi nikolay,
>>>
>>> Yes, this issue is what we are concerning a lot.
>>> About the current situation, try --split, it will save time.
>>>
>>>
>>> And lzo/snappy instead of zlib, these two compression format are faster
>>> but need more space to save. Or if you still want zlib (to save space),
>>> try multiple threads, check the following site, it will help you:
>>>
>>> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
>>>
>>>
>>>
>> .
>>
> 
> 

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[qiaonuohan]

On 09/18/2015 08:45 PM, Nikolay Borisov wrote:
> Yeah, I did see the commit browser. But in my case I haven't even tested
> the split option so I guess there are things to try. Am I correct in my
> understanding as to how --split is supposed to work ( i tried that to no
> avail though):
>
> My core_collector line is this:
>
> core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2
> dump3 dump4 dump5 dump6
>
> And then in /etc/sysconfig/kdump I have:
>
> KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices
> cgroup_disable=memory mce=off"
>
> (the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical
> cores in total). Do I need to do something else to utilize the --split
> option?

As I recall, it is OK to go on the test.

>
> On 09/18/2015 05:38 AM, qiaonuohan wrote:
>> On 09/17/2015 02:32 PM, Nikolay Borisov wrote:
>>> Hi Qiao,
>>>
>>> Thanks for the reply. So far I haven't been using the the compression
>>> feature of makedumpfile. But I want to ask if anything wouldn't
>>> compression make the dump process slower since in addition to having to
>>> write the dump to disk it also has to compress it which would put more
>>> strain on the cpu. Also, which part of the dump process is the
>>> bottleneck:
>>>
>>> - Reading from /proc/vmcore - that has mmap support so should be fairly
>>> fast?
>>> - Discarding unnecessary pages as memory is being scanned?
>>> - Writing/compressing content to disk?
>>
>> I cannot recall percentage of each part. But writing/compression takes most
>> of the time
>>
>> 1. mmap is used for reading faster
>> 2. --split is used to split the dump task into several processes, so
>> compressing
>>     and writing will be speeded up.
>> 3. multiple-thread is another option for speeding up compressing, it is
>> a recently
>>     committed patch, so you cannot find it in the master branch, checkout
>> devel branch
>>     or find it here:
>>
>> http://sourceforge.net/p/makedumpfile/code/commit_browser
>>
>> Make makedumpfile available to read and compress pages parallelly.
>>
>>>
>>> Regards,
>>> Nikolay
>>>
>>> On 09/17/2015 06:27 AM, qiaonuohan wrote:
>>>> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
>>>>> Hello,
>>>>>
>>>>> I've been using makedumpfile as the crash collector with the -d31
>>>>> parameter. The machine this is being run on usually have 128-256GB of
>>>>> ram and the resulting crash dumps are in the range of 14-20gb which is
>>>>> very bug for the type of analysis I'm usually performing on crashed
>>>>> machine. I was wondering whether there is a way  to further reduce the
>>>>> size and the time to take the dump (now it takes around 25 minutes to
>>>>> collect such a dump). I've seen reports where people with TBs of ram
>>>>> take that long, meaning for a machine with 256gb it should be even
>>>>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
>>>>> where mmap for the vmcore file is supported.
>>>>>
>>>>> Please advise
>>>>
>>>> Hi nikolay,
>>>>
>>>> Yes, this issue is what we are concerning a lot.
>>>> About the current situation, try --split, it will save time.
>>>>
>>>>
>>>> And lzo/snappy instead of zlib, these two compression format are faster
>>>> but need more space to save. Or if you still want zlib (to save space),
>>>> try multiple threads, check the following site, it will help you:
>>>>
>>>> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
>>>>
>>>>
>>>>
>>> .
>>>
>>
>>
> .
>


-- 
Regards
Qiao Nuohan

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Reducing the size of the dump file/speeding up collection

[Baoquan He]

On 09/18/15 at 03:45pm, Nikolay Borisov wrote:
> Yeah, I did see the commit browser. But in my case I haven't even tested
> the split option so I guess there are things to try. Am I correct in my
> understanding as to how --split is supposed to work ( i tried that to no
> avail though):
> 
> My core_collector line is this:
> 
> core_collector makedumpfile --message-level 1 -d 3 --split dump1 dump2
> dump3 dump4 dump5 dump6
> 
> And then in /etc/sysconfig/kdump I have:
> 
> KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=6 reset_devices
> cgroup_disable=memory mce=off"

disable_cpu_apicid need be specified if nr_cpus > 1. Otherwise it could
be hang.

> 
> (the machine I'm testing on has 4 cores x2 hyperthreads so 8 logical
> cores in total). Do I need to do something else to utilize the --split
> option?
> 
> On 09/18/2015 05:38 AM, qiaonuohan wrote:
> > On 09/17/2015 02:32 PM, Nikolay Borisov wrote:
> >> Hi Qiao,
> >>
> >> Thanks for the reply. So far I haven't been using the the compression
> >> feature of makedumpfile. But I want to ask if anything wouldn't
> >> compression make the dump process slower since in addition to having to
> >> write the dump to disk it also has to compress it which would put more
> >> strain on the cpu. Also, which part of the dump process is the
> >> bottleneck:
> >>
> >> - Reading from /proc/vmcore - that has mmap support so should be fairly
> >> fast?
> >> - Discarding unnecessary pages as memory is being scanned?
> >> - Writing/compressing content to disk?
> > 
> > I cannot recall percentage of each part. But writing/compression takes most
> > of the time
> > 
> > 1. mmap is used for reading faster
> > 2. --split is used to split the dump task into several processes, so
> > compressing
> >    and writing will be speeded up.
> > 3. multiple-thread is another option for speeding up compressing, it is
> > a recently
> >    committed patch, so you cannot find it in the master branch, checkout
> > devel branch
> >    or find it here:
> > 
> > http://sourceforge.net/p/makedumpfile/code/commit_browser
> > 
> > Make makedumpfile available to read and compress pages parallelly.
> > 
> >>
> >> Regards,
> >> Nikolay
> >>
> >> On 09/17/2015 06:27 AM, qiaonuohan wrote:
> >>> On 09/16/2015 04:30 PM, Nikolay Borisov wrote:
> >>>> Hello,
> >>>>
> >>>> I've been using makedumpfile as the crash collector with the -d31
> >>>> parameter. The machine this is being run on usually have 128-256GB of
> >>>> ram and the resulting crash dumps are in the range of 14-20gb which is
> >>>> very bug for the type of analysis I'm usually performing on crashed
> >>>> machine. I was wondering whether there is a way  to further reduce the
> >>>> size and the time to take the dump (now it takes around 25 minutes to
> >>>> collect such a dump). I've seen reports where people with TBs of ram
> >>>> take that long, meaning for a machine with 256gb it should be even
> >>>> faster. I've been running this configuration on kernels 3.12.28 and 4.1
> >>>> where mmap for the vmcore file is supported.
> >>>>
> >>>> Please advise
> >>>
> >>> Hi nikolay,
> >>>
> >>> Yes, this issue is what we are concerning a lot.
> >>> About the current situation, try --split, it will save time.
> >>>
> >>>
> >>> And lzo/snappy instead of zlib, these two compression format are faster
> >>> but need more space to save. Or if you still want zlib (to save space),
> >>> try multiple threads, check the following site, it will help you:
> >>>
> >>> https://lists.fedoraproject.org/pipermail/kexec/2015-September/002322.html
> >>>
> >>>
> >>>
> >> .
> >>
> > 
> > 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec