From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Elder Subject: Kernel Crash Dumps Date: Wed, 21 Mar 2012 16:09:11 -0500 Message-ID: <4F6A4377.6020908@dreamhost.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from mail.hq.newdream.net ([66.33.206.127]:36101 "EHLO mail.hq.newdream.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750969Ab2CUVJF (ORCPT ); Wed, 21 Mar 2012 17:09:05 -0400 Received: from mail.hq.newdream.net (localhost [127.0.0.1]) by mail.hq.newdream.net (Postfix) with ESMTP id D2F27243EF for ; Wed, 21 Mar 2012 14:10:15 -0700 (PDT) Received: from [172.22.22.4] (c-71-195-31-37.hsd1.mn.comcast.net [71.195.31.37]) by mail.hq.newdream.net (Postfix) with ESMTPSA id 88CD2243BE for ; Wed, 21 Mar 2012 14:10:15 -0700 (PDT) Sender: ceph-devel-owner@vger.kernel.org List-ID: To: ceph-devel@vger.kernel.org I've been working on getting kernel crash dumps to be generated and saved to a remote server system for later analysis. I have managed to wade through the maze of old, new, and conflicting information out there and have come up with a mechanism that reliably saves a kernel core dump in the event of a crash. I have also written an init script and accompanying shell script that takes a processed crash dump and re-packages it along with supplementary information, then copies that entire result to a remote server machine. I did all my development work on a local machine I have physical access to. I have begun the process of doing the same things on QA systems but have bumped into some obstacles. Tv suggested I document what I've got and post it here for others to see, and so he (or others) might be able to help a bit. So the following documents the process I was trying to get through when moving my work from my home machine to one of the plana systems. For the short term this stuff is on hold while I tend to a few other pressing issues, but I'd like to get this stuff going soon; kernel core dumps are an important resource for tracking down the cause of bugs in kernel code. -Alex ----------------- First, here's an overview of how it works: - kdump init script arranges for saving and processing a kernel core dump at boot time: - loads a crashkernel (using kexec) into reserved memory, which will be jumped to in the event of a crash and whose sole purpose is to save kernel memory to disk - runs /usr/share/apport/kernel_crashdump to process a crash dump file, leaving the result in a single file in /var/crash. - ceph-kdump-copy init script runs immediately thereafter, processing each file found in /var/crash that appears to have been created by the kdump init script, and for each, runs /usr/bin/ceph-kdump-copy on it to re-package and copy it to a designated remote host. Here's what that script does: - unpacks the processed file into its constituent parts - collects summary information into a file "summary.txt" - renames the core file to be "vmcore-" reflecting the kernel version - gathers the appropriate debug symbol file corresponding to the crashed kernel, along with the kernel configuration, system map, and other files that provide context about the crashed system. - Generates a short README file telling how to use "crash" on the collected files to debug the crash. - groups all of these files into a directory named with a date stamp based on the time they were collected. - copies the date-stamped directory to the remote server, under a subdirectory that contains all saved crashes from the crashed host. - removes all remnants of the crash once copying to the remote host is complete. Getting all this to work together has been some trouble though, and there are existing bugs that are worked around in the steps described below. ================================================ First, setting up the dump server is pretty straightforward. By default crashes are saved to /var/crash/remote on the server. You need to arrange for the crashing hosts to have write access using "scp" to that directory on the server, so you need to set up ssh keys accordingly. (I'm not going to describe that process here.) First, set up the directory to receive the dumps on the server: sudo bash < I'm having trouble figuring out where to get the dbgsym packages. They sort of magically showed up when I was setting this up on my home system. To boot that I made /etc/grub.d/01_ceph_kernel look like this: cat <", i.e.: cat < define reasonable values for KDUMP_HOST and KDUMP_HOST_USER sudo update-rc.d ceph-kdump-copy start 02 2 . We need to have a vmcoreinfo file in place that matches the currently-running kernel. I'm not sure where that comes from, but it may be generated automatically by the crash dump process. ==> Not sure about this. My machine at home had a separate /boot partition, and that ended up requiring some additional commands to ensure /boot on the underlying root filesystem was up-to-date. The plana systems don't have a separate boot partition so for now I'm going to repeat those instructions here. That's all I have (and at this point have not completed the process of moving this over to our QA systems so I have missed something). If a crash occurs, the whole sequence described earlier will ensue. If you wish to trigger a crash manually, do this (as root): echo c > /proc/sysrq-trigger I always preceded that command with a bunch of sync calls in order to try to keep my filesystems intact. ================================================