Re: [PATCH 1/8] pseries: phyp dump: Docmentation

[Nathan Lynch <ntl@pobox.com>]

Manish Ahuja wrote:
> +
> +                   Hypervisor-Assisted Dump
> +                   ------------------------
> +                       November 2007

Date is unneeded (and, uhm, dated :)


> +The goal of hypervisor-assisted dump is to enable the dump of
> +a crashed system, and to do so from a fully-reset system, and
> +to minimize the total elapsed time until the system is back
> +in production use.

Is it actually faster than kdump?


> +As compared to kdump or other strategies, hypervisor-assisted
> +dump offers several strong, practical advantages:
> +
> +-- Unlike kdump, the system has been reset, and loaded
> +   with a fresh copy of the kernel.  In particular,
> +   PCI and I/O devices have been reinitialized and are
> +   in a clean, consistent state.
> +-- As the dump is performed, the dumped memory becomes
> +   immediately available to the system for normal use.
> +-- After the dump is completed, no further reboots are
> +   required; the system will be fully usable, and running
> +   in it's normal, production mode on it normal kernel.
> +
> +The above can only be accomplished by coordination with,
> +and assistance from the hypervisor. The procedure is
> +as follows:
> +
> +-- When a system crashes, the hypervisor will save
> +   the low 256MB of RAM to a previously registered
> +   save region. It will also save system state, system
> +   registers, and hardware PTE's.
> +
> +-- After the low 256MB area has been saved, the
> +   hypervisor will reset PCI and other hardware state.
> +   It will *not* clear RAM. It will then launch the
> +   bootloader, as normal.
> +
> +-- The freshly booted kernel will notice that there
> +   is a new node (ibm,dump-kernel) in the device tree,
> +   indicating that there is crash data available from
> +   a previous boot. It will boot into only 256MB of RAM,
> +   reserving the rest of system memory.
> +
> +-- Userspace tools will parse /sys/kernel/release_region
> +   and read /proc/vmcore to obtain the contents of memory,
> +   which holds the previous crashed kernel. The userspace
> +   tools may copy this info to disk, or network, nas, san,
> +   iscsi, etc. as desired.
> +
> +   For Example: the values in /sys/kernel/release-region
> +   would look something like this (address-range pairs).
> +   CPU:0x177fee000-0x10000: HPTE:0x177ffe020-0x1000: /
> +   DUMP:0x177fff020-0x10000000, 0x10000000-0x16F1D370A
> +
> +-- As the userspace tools complete saving a portion of
> +   dump, they echo an offset and size to
> +   /sys/kernel/release_region to release the reserved
> +   memory back to general use.
> +
> +   An example of this is:
> +     "echo 0x40000000 0x10000000 > /sys/kernel/release_region"
> +   which will release 256MB at the 1GB boundary.

This violates the "one file, one value" rule of sysfs, but nobody
really takes that seriously, I guess.  In any case, consider
documenting this in Documentation/ABI.


> +
> +Please note that the hypervisor-assisted dump feature
> +is only available on Power6-based systems with recent
> +firmware versions.

This statement will of course become dated/incorrect so I recommend
removing it.


> +
> +Implementation details:
> +----------------------
> +In order for this scheme to work, memory needs to be reserved
> +quite early in the boot cycle. However, access to the device
> +tree this early in the boot cycle is difficult, and device-tree
> +access is needed to determine if there is a crash data waiting.

I don't think this bit about early device tree access is correct.  By
the time your code is reserving memory (from early_init_devtree(), I
think), RTAS has been instantiated and you are able to test for the
existence of /rtas/ibm,dump-kernel.


> +To work around this problem, all but 256MB of RAM is reserved
> +during early boot. A short while later in boot, a check is made
> +to determine if there is dump data waiting. If there isn't,
> +then the reserved memory is released to general kernel use.

So I think these gymnastics are unneeded -- unless I'm
misunderstanding something, you should be able to determine very early
whether to reserve that memory.


> +If there is dump data, then the /sys/kernel/release_region
> +file is created, and the reserved memory is held.
> +
> +If there is no waiting dump data, then all but 256MB of the
> +reserved ram will be released for general kernel use. The
> +highest 256 MB of RAM will *not* be released: this region
> +will be kept permanently reserved, so that it can act as
> +a receptacle for a copy of the low 256MB in the case a crash
> +does occur. See, however, "open issues" below, as to whether
> +such a reserved region is really needed.
> +
> +Currently the dump will be copied from /proc/vmcore to a
> +a new file upon user intervention. The starting address
> +to be read and the range for each data point in provided
                                               ^is

> +in /sys/kernel/release_region.
> +
> +The tools to examine the dump will be same as the ones
> +used for kdump.
> +
> +
> +General notes:
> +--------------
> +Security: please note that there are potential security issues
> +with any sort of dump mechanism. In particular, plaintext
> +(unencrypted) data, and possibly passwords, may be present in
> +the dump data. Userspace tools must take adequate precautions to
> +preserve security.
> +
> +Open issues/ToDo:
> +------------
> + o The various code paths that tell the hypervisor that a crash
> +   occurred, vs. it simply being a normal reboot, should be
> +   reviewed, and possibly clarified/fixed.
> +
> + o Instead of using /sys/kernel, should there be a /sys/dump
> +   instead? There is a dump_subsys being created by the s390 code,
> +   perhaps the pseries code should use a similar layout as well.

Well, it seems to me that there's little reason to duplicate the s390
layout unless we can actually share code.

FWIW, I've been thinking about making a /sys/firmware/phyp hierarchy
which could contain much of the System P-specific functions (DLPAR,
lparcfg, other crud in /proc/ppc64)... seems suited to this
platform-specific dump mechanism.