how to enable linux box to dump crash image on local disk

how to enable linux box to dump crash image on local disk

[amit mehta]

I've a Linux box which is crashing under a particular test scenario.
To debug this issue I was thinking of using the crash (man 8 crash)
utility (available by default on recent releases of RHEL and SuSE) .
This tool requires the kernel crash dump (vmcore) and the debugging symbols.
On my machine, I don't see any crash under /var/crash, Hence please
let me know the
configuration changes which i need to do to enable my machine to dump
the kernel image
locally (do i need to apply diskdump patches and rebuild the kernel??) whenever
there is a kernel crash. Kdump seem to be another tool for this job
but that requires a lot of
setup. Is there a *shortcut* to enable kernel crash dump facility ?

The debugging symbols can be found here:
http://ftp.riken.go.jp/Linux/slc/slc6X/updates/x86_64/debug/
http://ftp.riken.go.jp/Linux/slc/slc6X/updates/i386/debug/


-Amit

how to enable linux box to dump crash image on local disk

[amit mehta]

On Wed, Aug 17, 2011 at 3:04 PM, amit mehta <gmate.amit@gmail.com> wrote:
> I've a Linux box which is crashing under a particular test scenario.
> To debug this issue I was thinking of using the crash (man 8 crash)
> utility (available by default on recent releases of RHEL and SuSE) .
> This tool requires the kernel crash dump (vmcore) and the debugging symbols.
> On my machine, I don't see any crash under /var/crash, Hence please
> let me know the
Here
> configuration changes which i need to do to enable my machine to dump
> the kernel image
Here
>
It seems my gmail setup is randomly adding newline character when i
draft a mail.
Which i'm sure is annoying to those who'll read my mail on this
mailing list. Can you please
help me resolve this issue.

-Amit

how to enable linux box to dump crash image on local disk

[Leonid V. Fedorenchik]

On Wed, 17 Aug 2011 15:12:22 +0530
amit mehta <gmate.amit@gmail.com> wrote:

> On Wed, Aug 17, 2011 at 3:04 PM, amit mehta <gmate.amit@gmail.com> wrote:
> > I've a Linux box which is crashing under a particular test scenario.
> > To debug this issue I was thinking of using the crash (man 8 crash)
> > utility (available by default on recent releases of RHEL and SuSE) .
> > This tool requires the kernel crash dump (vmcore) and the debugging symbols.
> > On my machine, I don't see any crash under /var/crash, Hence please
> > let me know the
> Here
> > configuration changes which i need to do to enable my machine to dump
> > the kernel image
> Here
> >
> It seems my gmail setup is randomly adding newline character when i
> draft a mail.
> Which i'm sure is annoying to those who'll read my mail on this
> mailing list. Can you please
> help me resolve this issue.

Read Documentation/email-clients.txt
Hope this helps.

> 
> -Amit
> 
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Leonid V. Fedorenchik

how to enable linux box to dump crash image on local disk

[sumeet gandhare]

hi Amit,

On Wed, Aug 17, 2011 at 3:04 PM, amit mehta <gmate.amit@gmail.com> wrote:

> I've a Linux box which is crashing under a particular test scenario.
> To debug this issue I was thinking of using the crash (man 8 crash)
> utility (available by default on recent releases of RHEL and SuSE) .
> This tool requires the kernel crash dump (vmcore) and the debugging
> symbols.
> On my machine, I don't see any crash under /var/crash, Hence please
> let me know the
> configuration changes which i need to do to enable my machine to dump
> the kernel image
> locally (do i need to apply diskdump patches and rebuild the kernel??)
> whenever
> there is a kernel crash. Kdump seem to be another tool for this job
> but that requires a lot of
> setup. Is there a *shortcut* to enable kernel crash dump facility ?
>
> The debugging symbols can be found here:
> http://ftp.riken.go.jp/Linux/slc/slc6X/updates/x86_64/debug/
> http://ftp.riken.go.jp/Linux/slc/slc6X/updates/i386/debug/
>
>
>
 You need to configure kdump  on  your RHEL system to enable it to dump the
vmcore file when the panic happens... Kdump configuration is explained
below.. There is a crashkernel boot parameter to be passed to kernel to
specify the memory reserved for the crash kernel.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html

http://www.mindtwist.de/main/linux/11-red-hat/27-how-to-configure-kernel-dumps-on-rhel-5.html

http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaai/crashdump/liaaicrashdumpconfigkdump.htm


Thanks and Regards,
Sumeet


> -Amit
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110817/4a7630c0/attachment-0001.html 

how to enable linux box to dump crash image on local disk

[amit mehta]

> ?You need to configure kdump? on? your RHEL system to enable it to dump the
> vmcore file when the panic happens... Kdump configuration is explained
> below.. There is a crashkernel boot parameter to be passed to kernel to
> specify the memory reserved for the crash kernel.
>
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html
>
> http://www.mindtwist.de/main/linux/11-red-hat/27-how-to-configure-kernel-dumps-on-rhel-5.html
>
> http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaai/crashdump/liaaicrashdumpconfigkdump.htm
>
>
> Thanks and Regards,
> Sumeet


Thank you Sumeet for the link, I've enabled kdump on my machine by
following the steps mentioned
on the link above(http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html)
But it seems that there is some setup problem on my machine as
invoking "crash" is giving me an error.
see the logs below.

<<<snip from /etc/kdump.cfg>>>
#raw /dev/sda5
#ext4 /dev/sda2
#ext4 LABEL=/boot
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
#net my.server.com:/export/tmp
#net user at my.server.com
path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
#core_collector cp --sparse=always
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
default halt
<<<snip from /etc/kdump.cfg>>>


[root at rhel6 boot]# mkinitrd /boot/initrd-2.6.32-71.el6.x86_64kdump.img
2.6.32-71.el6.x86_64

[root at rhel6 ~]# service kdump restart
Stopping kdump:                                            [  OK  ]
Your running kernel is using more than 70% of the amount of[WARNING]u
reserved for kdump, you should consider increasing your crashkernel
reservation
Starting kdump:                                            [  OK  ]

[root at rhel6 ~]# service kdump status
Kdump is operational

I hit the crash again and this time there was a vmcore under
/var/crash directory

[root at rhel6 2011-08-17-17:50]# pwd
/var/crash/2011-08-17-17:50

[root at rhel6 2011-08-17-17:50]# du -sh vmcore
43M	vmcore

These are the two debug rpms that i've installed for kernel symbols:
[root at rhel6 ~]# rpm -qa|grep -i debug
kernel-debuginfo-common-x86_64-2.6.32-71.7.1.el6.x86_64
kernel-debug-debuginfo-2.6.32-71.7.1.el6.x86_64

my production kernel, from which i've booted:
[root at rhel6 ~]# uname -r
2.6.32-71.el6.x86_64

[root at rhel6 ~]# crash -s
/usr/lib/debug/lib/modules/2.6.32-71.7.1.el6.x86_64.debug/vmlinux
/var/crash/2011-08-17-17\:50/vmcore
crash: invalid kernel virtual address: 7180  type: "possible"
WARNING: cannot read cpu_possible_map < - ----------------------------
crash: seek error: kernel virtual address: ffffffff8208e980  type: "xtime"


-Amit

how to enable linux box to dump crash image on local disk

[sumeet gandhare]

On Wed, Aug 17, 2011 at 6:16 PM, amit mehta <gmate.amit@gmail.com> wrote:

> >  You need to configure kdump  on  your RHEL system to enable it to dump
> the
> > vmcore file when the panic happens... Kdump configuration is explained
> > below.. There is a crashkernel boot parameter to be passed to kernel to
> > specify the memory reserved for the crash kernel.
> >
> >
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html
> >
> >
> http://www.mindtwist.de/main/linux/11-red-hat/27-how-to-configure-kernel-dumps-on-rhel-5.html
> >
> >
> http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaai/crashdump/liaaicrashdumpconfigkdump.htm
> >
> >
> > Thanks and Regards,
> > Sumeet
>
>
> Thank you Sumeet for the link, I've enabled kdump on my machine by
> following the steps mentioned
> on the link above(
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/ch-kdump.html
> )
> But it seems that there is some setup problem on my machine as
> invoking "crash" is giving me an error.
> see the logs below.
>
> <<<snip from /etc/kdump.cfg>>>
> #raw /dev/sda5
> #ext4 /dev/sda2
> #ext4 LABEL=/boot
> #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
> #net my.server.com:/export/tmp
> #net user at my.server.com
> path /var/crash
> core_collector makedumpfile -c --message-level 1 -d 31
> #core_collector cp --sparse=always
> #link_delay 60
> #kdump_post /var/crash/scripts/kdump-post.sh
> #extra_bins /usr/bin/lftp
> #disk_timeout 30
> #extra_modules gfs2
> #options modulename options
> default halt
> <<<snip from /etc/kdump.cfg>>>
>
>
> [root at rhel6 boot]# mkinitrd /boot/initrd-2.6.32-71.el6.x86_64kdump.img
> 2.6.32-71.el6.x86_64
>
> [root at rhel6 ~]# service kdump restart
> Stopping kdump:                                            [  OK  ]
> Your running kernel is using more than 70% of the amount of[WARNING]u
> reserved for kdump, you should consider increasing your crashkernel
> reservation
> Starting kdump:                                            [  OK  ]
>
> [root at rhel6 ~]# service kdump status
> Kdump is operational
>
> I hit the crash again and this time there was a vmcore under
> /var/crash directory
>
> [root at rhel6 2011-08-17-17:50]# pwd
> /var/crash/2011-08-17-17:50
>
> [root at rhel6 2011-08-17-17:50]# du -sh vmcore
> 43M     vmcore
>
> These are the two debug rpms that i've installed for kernel symbols:
> [root at rhel6 ~]# rpm -qa|grep -i debug
> kernel-debuginfo-common-x86_64-2.6.32-71.7.1.el6.x86_64
> kernel-debug-debuginfo-2.6.32-71.7.1.el6.x86_64
>
> my production kernel, from which i've booted:
> [root at rhel6 ~]# uname -r
> 2.6.32-71.el6.x86_64
>
> [root at rhel6 ~]# crash -s
> /usr/lib/debug/lib/modules/2.6.32-71.7.1.el6.x86_64.debug/vmlinux
> /var/crash/2011-08-17-17\:50/vmcore
> crash: invalid kernel virtual address: 7180  type: "possible"
> WARNING: cannot read cpu_possible_map < - ----------------------------
> crash: seek error: kernel virtual address: ffffffff8208e980  type: "xtime"
>
>
> Hi Amit,
  It seems there is a bugzilla from Red Hat on this issue(680864) in RHEL
6.0 kernel. Please could you try this issue on RHEL 6.1 kernel ?
 Also as mentioned in the https://bugzilla.redhat.com/show_bug.cgi?id=680864,
which is similar to your issue, you should be able to see the output of bt
command and log command in the crash sesion. Please have a look at and let
us know what you see.

Regards,
Sumeet

> -Amit
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110817/a37aa23b/attachment.html 

how to enable linux box to dump crash image on local disk

[amit mehta]

> ? It seems there is a bugzilla from Red Hat on this issue(680864) in RHEL
> 6.0 kernel. Please could you try this issue on RHEL 6.1 kernel ?
> ?Also as mentioned in the
> https://bugzilla.redhat.com/show_bug.cgi?id=680864, which is similar to your
> issue, you should be able to see the output of bt command and log command in
> the crash sesion. Please have a look at and let us know what you see.
>
> Regards,
> Sumeet


It seems that it was a memory issue(not enough memory) to enable crash dump.
Today, after migrating my machine to RHEL 6.1 and bumping up the memory,
i was able to get the kernel crash dump and could analyze it using crash
utility.

Thanks again for your help.
<<<snip from sample forced crash>>
crash> bt
PID: 19208  TASK: ffff8801540200c0  CPU: 3   COMMAND: "bash"
 #0 [ffff88003778d9e0] machine_kexec at ffffffff810310db
 #1 [ffff88003778da40] kimage_free_pages at ffffffff810b63b2
 #2 [ffff88003778dac8] hvc_write_room at ffffffff8131abc6
 #3 [ffff88003778db10] setjmp_pre_handler at ffffffff814dec50
 #4 [ffff88003778db40] no_context at ffffffff81040cdb
 #5 [ffff88003778db90] __bad_area_nosemaphore at ffffffff81040f65
 #6 [ffff88003778dbe0] bad_area_access_error at ffffffff8104108e
 #7 [ffff88003778dc10] __do_page_fault at ffffffff810417b3
 #8 [ffff88003778dd30] __unregister_kprobe_top at ffffffff814e0c3e
 #9 [ffff88003778dd60] oops_end at ffffffff814ddfe5
#10 [ffff88003778dde8] hvc_write_room at ffffffff8131abc6
#11 [ffff88003778de20] destroy_hvc_struct at ffffffff8131ae82
#12 [ffff88003778de70] hvc_hangup at ffffffff8131af3e
#13 [ffff88003778dea0] proc_reg_unlocked_ioctl at ffffffff811d4ffe
#14 [ffff88003778def0] default_llseek at ffffffff81172718
#15 [ffff88003778df00] destroy_watch at ffffffff810d1b62
#16 [ffff88003778df30] sys_pread64 at ffffffff81173151
    RIP: 0000003b158d95e0  RSP: 00007fff4e780008  RFLAGS: 00000202
    RAX: 0000000000000001  RBX: ffffffff8100b172  RCX: 00007fff4e780098
    RDX: 0000000000000002  RSI: 00007f13cb5d7000  RDI: 0000000000000001
    RBP: 00007f13cb5d7000   R8: 000000000000000a   R9: 00007f13cb5c8700
    R10: 00000000ffffffff  R11: 0000000000000246  R12: 0000000000000002
    R13: 0000003b15b8b780  R14: 0000000000000002  R15: 0000003b15b8b780
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
<<<snip from sample forced crash>>

-Amit