[OOT]: crash / kdump query

[gmate.amit@gmail.com (Kumar Mehta)]

> 
> linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
> /boot/vmlinuz-2.6.32.12-crash-crash
> 
> crash 5.0.1
> Copyright (C) 2002-2010  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
> crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
> 
> Usage:
>   crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
> 
> 


Adil, If you have a RHEL 6.1 then kdump installation and setup is a
piece of cake, Luckily I have RHEL 6.1 on my machine and had setup kdump
by following the instructions in the link below:
http://people.redhat.com/anderson/crash_whitepaper/

To use crash utility for analyzing kernel crashes here's a sample
invocation of this command on my machine.

[root at rhel 127.0.0.1-2011-11-24-12:57:47]# cat /etc/issue
Red Hat Enterprise Linux Server release 6.1 (Santiago)
Kernel \r on an \m

[root at rhel 127.0.0.1-2011-11-24-12:57:47]# rpm -qa|grep -i debug
kernel-debuginfo-2.6.32-131.0.15.el6.x86_64
kernel-debuginfo-common-x86_64-2.6.32-131.0.15.el6.x86_64

[root at rhel 127.0.0.1-2011-11-24-12:57:47]# pwd
/var/crash/127.0.0.1-2011-11-24-12:57:47

[root at rhel 127.0.0.1-2011-11-24-12:57:47]# ls
vmcore

[root at lancer-evt03 127.0.0.1-2011-11-24-12:57:47]# crash -s
/usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux vmcore
crash: page excluded: kernel virtual address: ffffffffffffffff  type:
"possible"
WARNING: cannot read cpu_possible_map
crash: page excluded: kernel virtual address: ffffffffffffffff  type:
"present"
WARNING: cannot read cpu_present_map
crash: page excluded: kernel virtual address: ffffffffffffffff  type:
"online"
WARNING: cannot read cpu_online_map
WARNING: kernels compiled by different gcc versions:
  /usr/lib/debug/lib/modules/2.6.32-131.0.15.el6.x86_64/vmlinux: 4.4.5
  vmcore kernel: 4.4.4

crash: page excluded: kernel virtual address: ffffffffffffffff  type:
"cpu_present_map"
crash: page excluded: kernel virtual address: ffffffffffffffff  type:
"cpu_present_map"
crash> log
Pid: 10679, comm: bash Tainted: P        W  ---------------- T
2.6.32-131.0.15.el6.x86_64 #1 S5000PAL
RIP: 0010:[<ffffffff8115a1d4>]  [<ffffffff8115a1d4>]
cache_alloc_refill+0x1e4/0x240
RSP: 0018:ffff880157997bd8  EFLAGS: 00010046
RAX: 0000000000000009 RBX: ffff88015fc10080 RCX: 000000000000001e
RDX: ffff8801586ef000 RSI: ffff88015fc224c0 RDI: ffff880157fa9000
RBP: ffff880157997c38 R08: ffff8801586ef000 R09: 0000000070616d5f
R10: 0000000013131313 R11: 0000000000000000 R12: ffff88015ac54400
R13: ffff88015fc224c0 R14: 0000000000000009 R15: ffff880157fa9000
FS:  00007f31a13ab700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001f084a8 CR3: 00000001559d5000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 10679, threadinfo ffff880157996000, task
ffff8801575c0a80)
Stack:
 ffff880158ef2cc0 0000000058ef2cc0 ffff88015fc22500 000492d0a028816e
<0> ffff88015fc224e0 ffff88015fc224d0 0000000000000020 0000000000000044
<0> 0000000000000010 00000000000080d0 ffff88015fc10080 ffffffffa027fd07
Call Trace:
 [<ffffffffa027fd07>] ? ext4_htree_store_dirent+0x37/0x140 [ext4]
 [<ffffffff8115b079>] __kmalloc+0x1a9/0x220
 [<ffffffffa027fd07>] ext4_htree_store_dirent+0x37/0x140 [ext4]
 [<ffffffffa028d338>] htree_dirblock_to_tree+0x128/0x190 [ext4]
 [<ffffffffa028ddaa>] ext4_htree_fill_tree+0x16a/0x260 [ext4]
 [<ffffffffa027fb75>] ext4_readdir+0x4f5/0x650 [ext4]
 [<ffffffff81185f90>] ? filldir+0x0/0xe0
 [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff81185f90>] ? filldir+0x0/0xe0
 [<ffffffff81186210>] vfs_readdir+0xc0/0xe0
 [<ffffffff81186399>] sys_getdents+0x89/0xf0
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: 89 ff e8 a0 8c 11 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00
00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b
66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31
RIP  [<ffffffff8115a1d4>] cache_alloc_refill+0x1e4/0x240
 RSP <ffff880157997bd8>
crash>

This crash was invoked by a module to delibrately cause kernel crash.

-Amit