Re: Diskdump / kdump format history

[Stephen Brennan <stephen.s.brennan@oracle.com>]

Hi Petr,

Thank you for these insights, especially regarding the flattened format
and the lkdump + diskdump origin stories. I don't have any particular
follow-up questions, but I do intend to take a deeper look at the
different header files used by different projects and try to build some
documentation. I hope to make some content in a community wiki or
something which could summarize the high level ideas, and provide links
to all the different implementations, past and present. I hope to share
any and all of that with you and the rest of the mailing list as I have
updates.

Thanks again,
Stephen

Petr Tesařík <petr@tesarici.cz> writes:
> Hi Stephen,
>
> thanks for your trust. I've been around for some time, but I actually
> came late to the party. Below is my understanding of what happened.
> People who were really involved in all that may add a lot of
> corrections.
>
> On Thu, 12 Jan 2023 16:25:10 -0800
> Stephen Brennan <stephen.s.brennan@oracle.com> wrote:
>
>>[...]
>> First, regarding naming. I guess I always assumed that "diskdump"
>> referred to some older utility which performed the same function that
>> makedumpfile now is commonly used for. Is that the case?
>
> Not quite. It did not perform the same function...
>
> Let me make a short historical excursion.
>
> When I joined SUSE, SLES9 was the latest release and it was based
> on kernel 2.6.5. There was no kdump at that time, but a team at
> SGI worked on LKCD (Linux Kernel Crash Dumps), which was an
> out-of-tree patch to the Linux Kernel, some user-space tools for
> setting it up and the lcrash analysis tool (which was, as a matter
> of fact, simply horrible). Most importantly, the dumping itself
> was performed by the crashed kernel, using a special polling mode
> that was implemented by a few device drivers. LKCD offered two
> options: netdump and diskdump. This is most likely where the term
> "diskdump" was first coined.
>
>> Do you know the origin of the diskdump format?
>
> Yes. Fujitsu developed their own Linux kernel dump project called
> lkdump. It was conceptually similar to LKCD, as it also ran within
> the crashed kernel, but it did not use the LKCD file format. IIUC
> they first modified LKCD for their Mission Critical Linux (or
> MCLX), but they soon discovered that there were some issues with
> the format itself and designed their own format with the signature
> "DISKDUMP".
>
>> Do you know where to find the code for
>> this older diskdump tool, if it existed?
>
> Yes. Have a look here:
>
> https://sourceforge.net/projects/lkdump/
>
>> Second, I have some confusion regarding headers. I've seen headers for
>> files with the content "DISKDUMP" and "KDUMP   " -- and according to
>> src/kdumpfile/diskdump.c, you have too. diskdump_probe() shows that you
>> check for these two headers and set a different description depending on
>> which is seen. Do you know why or where these two headers are set, and
>> which one is chose in which situation?
>
> "DISKDUMP" is the signature of the original Fujitsu lkdump
> files. "KDUMP   " is the signature of the makedumpfile modification
> (that's the meaning of "_mod" in makedumpfile's diskdump_mod.h
> header file name). Yes, there was a header_version field, but that
> wasn't deemed sufficient. Since the lkdump project was not known
> to be dead for sure when makedumpfile was written, Ken'ichi
> Ohmichi (the original author) was afraid that lkdump might make
> conflicting extensions to their format and bump this field,
> leading to two incompatible file formats with the same signature.
> Note that makedumpfile was a project by NEC, not Fujitsu.
>
>> Third, sometimes I see an additional header with signature
>> "makedumpfile", and a few 64-bit integer fields, and then the
>> KDUMP/DISKDUMP header occurs at offset 4096/0x1000. I wonder if you're
>> familiar with that header, why or why not it occurs?
>
> This is the flattened format, which is used if makedumpfile writes
> to a file descriptor that cannot be seeked, e.g. a pipe. You
> should rearrange the file with "makedumpfile -R" (or the
> makedumpfile-R.pl script) to convert it to a standard diskdump
> file.
>
>> Finally, and this one is a long shot, do you know any good central
>> document describing the diskdump / kdump format? From everything I can
>> tell, it seems very much like a "living standard", one which lives in
>> code more than documentation.
>
> This is very much the case. For the "DISKDUMP" format, whatever is
> in makedumpfile's diskdump_mod.h is authoritative. The "KDUMP    "
> format is essentially dead.
>
> HTH
> Petr T
>
>> But I would be glad to be wrong here, if
>> there's a document which describes the format then I'd love to read it,
>> and if not, maybe I could be part of creating it to help everyone out.
>> 
>> Thanks,
>> Stephen