Re: [Fastboot] Documentation feedback request.

Re: [Fastboot] Documentation feedback request.

[Michael Ellerman]

[-- Attachment #1: Type: text/plain, Size: 11643 bytes --]

Hi David,

Nice work. Few comments below ...

On Wed, 2006-04-26 at 10:49 -0700, David Wilder wrote:
> Attached is a updated  Documentation/kdump/kdump.txt.  Please provide 
> comments.  I will incorporate your feedback and send up to Linus.
> 
> plain text document attachment (kdump-final-20060425.txt)
> ================================================================
> Documentation for Kdump - The kexec-based Crash Dumping Solution
> ================================================================
> 
> This document includes overview, setup and installation, and analysis
> information.
> 
> Overview
> ========
> 
> Kdump uses kexec to quickly boot to a dump-capture kernel whenever a
> dump of the system kernel's memory needs to be taken (for example, when
> the system panics). The system kernel's memory image is preserved across
> the reboot and is accessible to the dump-capture kernel.
> 
> You can use common Linux commands, such as cp and scp, to copy the
> memory image to a dump file on the local disk, or across the network to
> a remote system.
> 
> Kdump and kexec are currently supported on the x86, x86_64, and ppc64
> architectures.

s/ppc64/64-bit powerpc/ ?

> When the system kernel boots, it reserves a small section of memory for
> the dump-capture kernel. This ensures that ongoing Direct Memory Access
> (DMA) from the system kernel does not corrupt the dump-capture kernel. 
> The kexec -p command loads the dump-capture kernel into this reserved
> memory.

Well hopefully ;)

> On x86 machines, the first 640 KB of physical memory is needed to boot,
> regardless of where the kernel loads. Therefore, kexec backs up this
> region just before rebooting into the dump-capture kernel.
> 
> All of the necessary information about the system kernel's core image is
> encoded in the ELF format, and stored in a reserved area of memory
> before a crash. The physical address of the start of the ELF header is
> passed to the dump-capture kernel through the elfcorehdr= boot
> parameter.
> 
> With the dump-capture kernel, you can access the memory image, or "old
> memory," in two ways:
> 
> - Through a /dev/oldmem device interface. A capture utility can read the
>   device file and write out the memory in raw format. This is a raw dump
>   of memory. Analysis and capture tools must be intelligent enough to
>   determine where to look for the right information.
> 
> - Through /proc/vmcore. This exports the dump as an ELF-format file that
>   you can write out using file copy commands such as cp or scp. Further,
>   you can use analysis tools such as the GNU Debugger (GDB) and the Crash
>   tool to debug the dump file. This method ensures that the dump pages are
>   correctly ordered.
> 
> 
> Setup and Installation
> ======================
> 
> Install kexec-tools and the Kdump patch
> ---------------------------------------
> 
> 1) Login as the root user.
> 
> 2) Download the kexec-tools user-space package from the following URL:
> 
>    http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz
> 
> 3) Unpack the tarball with the tar command, as follows:
> 
>    tar xvpzf kexec-tools-1.101.tar.gz
> 
> 4) Download the latest consolidated Kdump patch from the following URL:
> 
>    http://lse.sourceforge.net/kdump/
>    
>    (This location is being used until all the user-space Kdump patches
>    are integrated with the kexec-tools package.)
>    
> 5) Change to the kexec-tools-1.101 directory, as follows:
> 
>    cd kexec-tools-1.101
> 
> 6) Apply the consolidated patch to the kexec-tools-1.101 source tree
>    with the patch command, as follows. (Modify the path to the downloaded
>    patch as necessary.)
>    
>    patch -p1 < /path-to-kdump-patch/kexec-tools-1.101-kdump.patch
> 
> 7) Configure the package, as follows:
> 
>    ./configure
> 
> 8) Compile the package, as follows:
> 
>    make
> 
> 9) Install the package, as follows:
> 
>    make install
> 
> 
> Download and build the system and dump-capture kernels
> ------------------------------------------------------
> 
> Download the mainline (vanilla) kernel source code (2.6.13-rc1 or newer)
> from http://www.kernel.org. Two kernels must be built: a system kernel
> and a dump-capture kernel. Use the following steps to configure these
> kernels with the necessary kexec and Kdump features:
> 
> System kernel
> -------------
> 
> 1) Enable "kexec system call" in "Processor type and features."
> 
>    CONFIG_KEXEC=y
> 
> 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo
>    filesystems." This is usually enabled by default.
> 
>    CONFIG_SYSFS=y
> 
>    Note that "sysfs file system support" might not appear in the "Pseudo
>    filesystems" menu if "Configure standard kernel features (for small
>    systems)" is not enabled in "General Setup." In this case, check the
>    .config file itself to ensure that sysfs is turned on, as follows:
> 
>    grep 'CONFIG_SYSFS' .config

Is there a particular requirement for sysfs?

> 3) Enable "Compile the kernel with debug info" in "Kernel hacking."
> 
>    CONFIG_DEBUG_INFO=Y
> 
>    This causes the kernel to be built with debug symbols. The dump
>    analysis tools require a vmlinux with debug symbols in order to read
>    and analyze a dump file.
>            
> 4) Make and install the kernel and its modules. Update the boot loader
>    (such as grub, yaboot, or lilo) configuration files as necessary.
> 
> 5) Boot the system kernel with the boot parameter "crashkernel=Y@X",
>    where Y specifies how much memory to reserve for the dump-capture kernel
>    and X specifies the beginning of this reserved memory. For example,
>    "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory
>    starting at physical address 0x01000000 for the dump-capture kernel.

Most of this doesn't apply to powerpc.

>    On x86 and x86_64, use "crashkernel=64M@16M".
> 
>    On ppc64, use "crashkernel=128M@32M".

No just use "crashkernel=128M".

> The dump-capture kernel
> -----------------------
> 
> 1) Under "General setup," append "-kdump" to the current string in
>    "Local version."
> 
> 2) On x86, enable high memory support under "Processor type and
>    features":
> 
>    CONFIG_HIGHMEM=y
> 
> 3) On x86 and x86_64, disable symmetric multi-processing support
>    under "Processor type and features":
>    
>    CONFIG_SMP=n
> 
> 4) On ppc64, disable NUMA support and enable EMBEDDED support:
>    
>    CONFIG_NUMA=n
>    CONFIG_EMBEDDED=y
>    CONFIG_EEH=N for the dump-capture kernel

Why are we disabling NUMA? AFAIK we work on more systems with NUMA than
without?
And why are we turning off EMBEDDED and EEH?

> 5) Enable "kernel crash dumps" support under "Processor type and
>    features":
> 
>    CONFIG_CRASH_DUMP=y
> 
> 6) Use a suitable value for "Physical address where the kernel is
>    loaded" (under "Processor type and features"). This only appears when
>    "kernel crash dumps" is enabled. By default this value is 0x1000000
>    (16MB). It should be the same as X in the "crashkernel=Y@X" boot
>    parameter discussed above.
> 
>    On x86 and x86_64, use "CONFIG_PHYSICAL_START=0x1000000".
>       
>    On ppc64 the value is automatically set at 32MB when
>    CONFIG_CRASH_DUMP is set.

This whole step should start "On x86 ..."

> 6) Optionally enable "/proc/vmcore support" under "Filesystems" ->
>    "Pseudo filesystems".

6 + 1 = 6 :D

>    CONFIG_PROC_VMCORE=y
>       
> 7) Make and install the kernel and its modules. DO NOT add this kernel
>    to the boot loader configuration files.
>    
> 
> Load the Dump-capture Kernel
> ============================
> 
> After booting to the system kernel, load the dump-capture kernel using
> the following command:
> 
>    kexec -p <dump-capture-kernel> \
>    --initrd=<initrd-for-dump-capture-kernel> --args-linux \
>    --append="root=<root-dev> init 1 irqpoll"

I've never tested irqpoll on powerpc, I'm not sure we want to recommend
it, has someone tested it?

> Notes on loading the dump-capture kernel:
> 
> * <dump-capture-kernel> must be a vmlinux image (that is, an
>   uncompressed ELF image). bzImage does not work at this time.
> 
> * By default, the ELF headers are stored in ELF64 format to support
>   systems with more than 4GB memory. The --elf32-core-headers option can
>   be used to force the generation of ELF32 headers. This is necessary
>   because GDB currently cannot open vmcore files with ELF64 headers on
>   32-bit systems. ELF32 headers can be used on non-PAE systems (that is,
>   less than 4GB of memory).
> 		 
> * The "irqpoll" boot parameter reduces driver initialization failures
>   due to shared interrupts in the dump-capture kernel.
> 		  
> * You must specify <root-dev> in the format corresponding to the root
>   device name in the output of mount command.

> * "init 1" boots the dump-capture kernel into single-user mode without
>   networking. If you want networking, use "init 3."
> 
> 
> Kernel Panic
> ============
> 
> After successfully loading the dump-capture kernel as previously
> described, the system will reboot into the dump-capture kernel if a
> panic occurs. You can write a module to force the panic, or use
> "ALT-SysRq-c" to initiate a crash dump for testing purposes.
> 
> 
> Write Out the Dump File
> =======================
> 
> After the dump-capture kernel is booted, write out the dump file with
> the following command:
> 
>    cp /proc/vmcore <dump-file>
> 
> You can also access dumped memory as a /dev/oldmem device for a linear
> and raw view. To create the device, use the following command:
> 
>     mknod /dev/oldmem c 1 12
> 
> Use the dd command with suitable options for count, bs, and skip to
> access specific portions of the dump.
> 
> To see the entire memory, use the following command:
>    
>    dd if=/dev/oldmem of=oldmem.001
> 
> 
> Analysis
> ========
> 
> Before analyzing the dump image, you should reboot into a stable kernel.
> 
> You can do limited analysis using GDB on the dump file copied out of
> /proc/vmcore. Use the debug vmlinux built with -g and run the following
> command:
> 
>    gdb vmlinux <dump-file>
> 
> Stack trace for the task on processor 0, register display, and memory
> display work fine.
> 
> Note: GDB cannot analyze core files generated in ELF64 format for x86. 
> On systems with a maximum of 4GB of memory, you can generate
> ELF32-format headers using the --elf32-core-headers kernel option on the
> dump kernel.
> 
> You can also use the Crash utility to analyze dump files in Kdump
> format. Crash is available on Dave Anderson's site at the following URL:
> 
>    http://people.redhat.com/~anderson/
> 
> 
> To Do
> =====
> 
> 1) Provide a kernel pages filtering mechanism, so core file size is not
>    extreme on systems with huge memory banks.
> 
> 2) Relocatable kernel can help in maintaining multiple kernels for
>    crash_dump, and the same kernel as the system kernel can be used to
>    capture the dump.
> 
> 
> Contact
> =======
> 
> Vivek Goyal (vgoyal@in.ibm.com)
> Maneesh Soni (maneesh@in.ibm.com)

fastboot@lists.osdl.org


cheers

-- 
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: [Fastboot] Documentation feedback request.

[Mohan Kumar M]

On Thu, Apr 27, 2006 at 09:44:31AM +1000, Michael Ellerman wrote:
> Hi David,
> 
> Nice work. Few comments below ...
> 
> On Wed, 2006-04-26 at 10:49 -0700, David Wilder wrote:
> > Attached is a updated  Documentation/kdump/kdump.txt.  Please provide 
> > comments.  I will incorporate your feedback and send up to Linus.
> > 
> > plain text document attachment (kdump-final-20060425.txt)
> > ================================================================
> > Documentation for Kdump - The kexec-based Crash Dumping Solution
> > ================================================================
> > 
> > This document includes overview, setup and installation, and analysis
> > information.
> > 
> > Overview
> > ========
> > 
> > Kdump uses kexec to quickly boot to a dump-capture kernel whenever a
> > dump of the system kernel's memory needs to be taken (for example, when
> > the system panics). The system kernel's memory image is preserved across
> > the reboot and is accessible to the dump-capture kernel.
> > 
> > You can use common Linux commands, such as cp and scp, to copy the
> > memory image to a dump file on the local disk, or across the network to
> > a remote system.
> > 
> > Kdump and kexec are currently supported on the x86, x86_64, and ppc64
> > architectures.
> 
> s/ppc64/64-bit powerpc/ ?
> 
> > When the system kernel boots, it reserves a small section of memory for
> > the dump-capture kernel. This ensures that ongoing Direct Memory Access
> > (DMA) from the system kernel does not corrupt the dump-capture kernel. 
> > The kexec -p command loads the dump-capture kernel into this reserved
> > memory.
> 
> Well hopefully ;)
> 
> > On x86 machines, the first 640 KB of physical memory is needed to boot,
> > regardless of where the kernel loads. Therefore, kexec backs up this
> > region just before rebooting into the dump-capture kernel.
> > 
> > All of the necessary information about the system kernel's core image is
> > encoded in the ELF format, and stored in a reserved area of memory
> > before a crash. The physical address of the start of the ELF header is
> > passed to the dump-capture kernel through the elfcorehdr= boot
> > parameter.
> > 
> > With the dump-capture kernel, you can access the memory image, or "old
> > memory," in two ways:
> > 
> > - Through a /dev/oldmem device interface. A capture utility can read the
> >   device file and write out the memory in raw format. This is a raw dump
> >   of memory. Analysis and capture tools must be intelligent enough to
> >   determine where to look for the right information.
> > 
> > - Through /proc/vmcore. This exports the dump as an ELF-format file that
> >   you can write out using file copy commands such as cp or scp. Further,
> >   you can use analysis tools such as the GNU Debugger (GDB) and the Crash
> >   tool to debug the dump file. This method ensures that the dump pages are
> >   correctly ordered.
> > 
> > 
> > Setup and Installation
> > ======================
> > 
> > Install kexec-tools and the Kdump patch
> > ---------------------------------------
> > 
> > 1) Login as the root user.
> > 
> > 2) Download the kexec-tools user-space package from the following URL:
> > 
> >    http://www.xmission.com/~ebiederm/files/kexec/kexec-tools-1.101.tar.gz
> > 
> > 3) Unpack the tarball with the tar command, as follows:
> > 
> >    tar xvpzf kexec-tools-1.101.tar.gz
> > 
> > 4) Download the latest consolidated Kdump patch from the following URL:
> > 
> >    http://lse.sourceforge.net/kdump/
> >    
> >    (This location is being used until all the user-space Kdump patches
> >    are integrated with the kexec-tools package.)
> >    
> > 5) Change to the kexec-tools-1.101 directory, as follows:
> > 
> >    cd kexec-tools-1.101
> > 
> > 6) Apply the consolidated patch to the kexec-tools-1.101 source tree
> >    with the patch command, as follows. (Modify the path to the downloaded
> >    patch as necessary.)
> >    
> >    patch -p1 < /path-to-kdump-patch/kexec-tools-1.101-kdump.patch
> > 
> > 7) Configure the package, as follows:
> > 
> >    ./configure
> > 
> > 8) Compile the package, as follows:
> > 
> >    make
> > 
> > 9) Install the package, as follows:
> > 
> >    make install
> > 
> > 
> > Download and build the system and dump-capture kernels
> > ------------------------------------------------------
> > 
> > Download the mainline (vanilla) kernel source code (2.6.13-rc1 or newer)
> > from http://www.kernel.org. Two kernels must be built: a system kernel
> > and a dump-capture kernel. Use the following steps to configure these
> > kernels with the necessary kexec and Kdump features:
> > 
> > System kernel
> > -------------
> > 
> > 1) Enable "kexec system call" in "Processor type and features."
> > 
> >    CONFIG_KEXEC=y
> > 
> > 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo
> >    filesystems." This is usually enabled by default.
> > 
> >    CONFIG_SYSFS=y
> > 
> >    Note that "sysfs file system support" might not appear in the "Pseudo
> >    filesystems" menu if "Configure standard kernel features (for small
> >    systems)" is not enabled in "General Setup." In this case, check the
> >    .config file itself to ensure that sysfs is turned on, as follows:
> > 
> >    grep 'CONFIG_SYSFS' .config
> 
> Is there a particular requirement for sysfs?
> 
> > 3) Enable "Compile the kernel with debug info" in "Kernel hacking."
> > 
> >    CONFIG_DEBUG_INFO=Y
> > 
> >    This causes the kernel to be built with debug symbols. The dump
> >    analysis tools require a vmlinux with debug symbols in order to read
> >    and analyze a dump file.
> >            
> > 4) Make and install the kernel and its modules. Update the boot loader
> >    (such as grub, yaboot, or lilo) configuration files as necessary.
> > 
> > 5) Boot the system kernel with the boot parameter "crashkernel=Y@X",
> >    where Y specifies how much memory to reserve for the dump-capture kernel
> >    and X specifies the beginning of this reserved memory. For example,
> >    "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory
> >    starting at physical address 0x01000000 for the dump-capture kernel.
> 
> Most of this doesn't apply to powerpc.
> 
> >    On x86 and x86_64, use "crashkernel=64M@16M".
> > 
> >    On ppc64, use "crashkernel=128M@32M".
> 
> No just use "crashkernel=128M".
> 
May be this is to follow the same convention in other architectures.

> > The dump-capture kernel
> > -----------------------
> > 
> > 1) Under "General setup," append "-kdump" to the current string in
> >    "Local version."
> > 
> > 2) On x86, enable high memory support under "Processor type and
> >    features":
> > 
> >    CONFIG_HIGHMEM=y
> > 
> > 3) On x86 and x86_64, disable symmetric multi-processing support
> >    under "Processor type and features":
> >    
> >    CONFIG_SMP=n
> > 
> > 4) On ppc64, disable NUMA support and enable EMBEDDED support:
> >    
> >    CONFIG_NUMA=n
> >    CONFIG_EMBEDDED=y
> >    CONFIG_EEH=N for the dump-capture kernel
> 
> Why are we disabling NUMA? AFAIK we work on more systems with NUMA than
> without?
> And why are we turning off EMBEDDED and EEH?
> 
In some systems kdump with NUMA panics. Also EEH gives some oops while kdump booting. To disable EEH, we need to enable CONFIG_EMBEDDED

> > 5) Enable "kernel crash dumps" support under "Processor type and
> >    features":
> > 
> >    CONFIG_CRASH_DUMP=y
> > 
> > 6) Use a suitable value for "Physical address where the kernel is
> >    loaded" (under "Processor type and features"). This only appears when
> >    "kernel crash dumps" is enabled. By default this value is 0x1000000
> >    (16MB). It should be the same as X in the "crashkernel=Y@X" boot
> >    parameter discussed above.
> > 
> >    On x86 and x86_64, use "CONFIG_PHYSICAL_START=0x1000000".
> >       
> >    On ppc64 the value is automatically set at 32MB when
> >    CONFIG_CRASH_DUMP is set.
> 
> This whole step should start "On x86 ..."
> 
> > 6) Optionally enable "/proc/vmcore support" under "Filesystems" ->
> >    "Pseudo filesystems".
> 
> 6 + 1 = 6 :D
> 
> >    CONFIG_PROC_VMCORE=y
> >       
> > 7) Make and install the kernel and its modules. DO NOT add this kernel
> >    to the boot loader configuration files.
> >    
> > 
> > Load the Dump-capture Kernel
> > ============================
> > 
> > After booting to the system kernel, load the dump-capture kernel using
> > the following command:
> > 
> >    kexec -p <dump-capture-kernel> \
> >    --initrd=<initrd-for-dump-capture-kernel> --args-linux \
> >    --append="root=<root-dev> init 1 irqpoll"
> 
> I've never tested irqpoll on powerpc, I'm not sure we want to recommend
> it, has someone tested it?
> 
> > Notes on loading the dump-capture kernel:
> > 
> > * <dump-capture-kernel> must be a vmlinux image (that is, an
> >   uncompressed ELF image). bzImage does not work at this time.
> > 
> > * By default, the ELF headers are stored in ELF64 format to support
> >   systems with more than 4GB memory. The --elf32-core-headers option can
> >   be used to force the generation of ELF32 headers. This is necessary
> >   because GDB currently cannot open vmcore files with ELF64 headers on
> >   32-bit systems. ELF32 headers can be used on non-PAE systems (that is,
> >   less than 4GB of memory).
> > 		 
> > * The "irqpoll" boot parameter reduces driver initialization failures
> >   due to shared interrupts in the dump-capture kernel.
> > 		  
> > * You must specify <root-dev> in the format corresponding to the root
> >   device name in the output of mount command.
> 
> > * "init 1" boots the dump-capture kernel into single-user mode without
> >   networking. If you want networking, use "init 3."
> > 
> > 
> > Kernel Panic
> > ============
> > 
> > After successfully loading the dump-capture kernel as previously
> > described, the system will reboot into the dump-capture kernel if a
> > panic occurs. You can write a module to force the panic, or use
> > "ALT-SysRq-c" to initiate a crash dump for testing purposes.
> > 
> > 
> > Write Out the Dump File
> > =======================
> > 
> > After the dump-capture kernel is booted, write out the dump file with
> > the following command:
> > 
> >    cp /proc/vmcore <dump-file>
> > 
> > You can also access dumped memory as a /dev/oldmem device for a linear
> > and raw view. To create the device, use the following command:
> > 
> >     mknod /dev/oldmem c 1 12
> > 
> > Use the dd command with suitable options for count, bs, and skip to
> > access specific portions of the dump.
> > 
> > To see the entire memory, use the following command:
> >    
> >    dd if=/dev/oldmem of=oldmem.001
> > 
> > 
> > Analysis
> > ========
> > 
> > Before analyzing the dump image, you should reboot into a stable kernel.
> > 
> > You can do limited analysis using GDB on the dump file copied out of
> > /proc/vmcore. Use the debug vmlinux built with -g and run the following
> > command:
> > 
> >    gdb vmlinux <dump-file>
> > 
> > Stack trace for the task on processor 0, register display, and memory
> > display work fine.
> > 
> > Note: GDB cannot analyze core files generated in ELF64 format for x86. 
> > On systems with a maximum of 4GB of memory, you can generate
> > ELF32-format headers using the --elf32-core-headers kernel option on the
> > dump kernel.
> > 
> > You can also use the Crash utility to analyze dump files in Kdump
> > format. Crash is available on Dave Anderson's site at the following URL:
> > 
> >    http://people.redhat.com/~anderson/
> > 
> > 
> > To Do
> > =====
> > 
> > 1) Provide a kernel pages filtering mechanism, so core file size is not
> >    extreme on systems with huge memory banks.
> > 
> > 2) Relocatable kernel can help in maintaining multiple kernels for
> >    crash_dump, and the same kernel as the system kernel can be used to
> >    capture the dump.
> > 
> > 
> > Contact
> > =======
> > 
> > Vivek Goyal (vgoyal@in.ibm.com)
> > Maneesh Soni (maneesh@in.ibm.com)
> 
> fastboot@lists.osdl.org
> 
> 
> cheers
> 
> -- 
> Michael Ellerman
> IBM OzLabs
> 
> wwweb: http://michael.ellerman.id.au
> phone: +61 2 6212 1183 (tie line 70 21183)
> 
> We do not inherit the earth from our ancestors,
> we borrow it from our children. - S.M.A.R.T Person



> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev

Re: [Fastboot] Documentation feedback request.

[Vivek Goyal]

On Thu, Apr 27, 2006 at 09:44:31AM +1000, Michael Ellerman wrote:
> > Load the Dump-capture Kernel
> > ============================
> > 
> > After booting to the system kernel, load the dump-capture kernel using
> > the following command:
> > 
> >    kexec -p <dump-capture-kernel> \
> >    --initrd=<initrd-for-dump-capture-kernel> --args-linux \
> >    --append="root=<root-dev> init 1 irqpoll"
> 
> I've never tested irqpoll on powerpc, I'm not sure we want to recommend
> it, has someone tested it?
> 

We have tested it on x86 & x86_64 and it works fine. Not sure about powerpc.
Dave, Mohan and Mike have been doing some testing on powerpc. Guys any
inputs?

-vivek