[Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

(Note: I'm not subscribed to either qemu-devel or the kexec list; please
keep me CC'd.)

QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.

The resultant vmcore is usually analyzed with the "crash" utility.

The original tool producing such files is kdump. Unlike the procedure
performed by QEMU, kdump runs from *within* the guest (under a kexec'd
kdump kernel), and has more information about the original guest kernel
state (which is being dumped) than QEMU. To QEMU, the guest kernel state
is opaque.

For this reason, the kdump preparation logic in QEMU hardcodes a number
of fields in the kdump header. The direct issue is the "phys_base"
field. Refer to dump.c, functions create_header32(), create_header64(),
and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
"0").

http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD

http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD

This works in most cases, because the guest Linux kernel indeed tends to
be loaded at guest-phys address 0. However, when the guest Linux kernel
is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
then the guest Linux kernel is loaded at 16MB, thereby getting out of
sync with the phys_base=0 setting visible in the KDUMP header.

This trips up the "crash" utility.

Dave worked around the issue in "crash" for ELF format dumps -- "crash"
can identify QEMU as the originator of the vmcore by finding the QEMU
notes in the ELF vmcore. If those are present, then "crash" employs a
heuristic, probing for a phys_base up to 32MB, in 1MB steps.

Alas, the QEMU notes are not present in the KDUMP-format vmcores that
QEMU produces (they cannot be), hence crash has no way to tell apart
such files from those generated by genuine kdump. As an end result,
"crash" cannot automatically find the phys_base of OVMF-based Linux vmcores.

Dave suggested that a new flag, or a special phys_base value (like ~0UL)
be introduced as a distinguishing mark for QEMU-produced kdumps.

Implementing this in QEMU wouldn't be hard. The big question is
compatibility -- whose analysis tools would be broken by a (phys_base ==
~0UL) setting, or by a new flag?

Note that this change would affect SeaBIOS-based vmcores too. QEMU can't
(and shouldn't) discriminate the vmcores it dumps based on guest
firmware. (If QEMU did that, then it might as well try to figure out the
real phys_base value, which is clearly out of scope for qemu. One of the
selling points of the paging=false dump is that it doesn't involve
parsing guest RAM.)

Thanks
Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Peter Maydell]

On 11 November 2014 11:22, Laszlo Ersek <lersek@redhat.com> wrote:
> For this reason, the kdump preparation logic in QEMU hardcodes a number
> of fields in the kdump header. The direct issue is the "phys_base"
> field. Refer to dump.c, functions create_header32(), create_header64(),
> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> "0").
>
> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>
> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>
> This works in most cases, because the guest Linux kernel indeed tends to
> be loaded at guest-phys address 0. However, when the guest Linux kernel
> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> sync with the phys_base=0 setting visible in the KDUMP header.

Presumably this is also not going to work for machines other
than the x86 PC, where physical memory may well not start at
address zero...

-- PMM

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Christopher Covington]

On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
> keep me CC'd.)
> 
> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
> 
> The resultant vmcore is usually analyzed with the "crash" utility.
> 
> The original tool producing such files is kdump. Unlike the procedure
> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> kdump kernel), and has more information about the original guest kernel
> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> is opaque.
> 
> For this reason, the kdump preparation logic in QEMU hardcodes a number
> of fields in the kdump header. The direct issue is the "phys_base"
> field. Refer to dump.c, functions create_header32(), create_header64(),
> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> "0").
> 
> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
> 
> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
> 
> This works in most cases, because the guest Linux kernel indeed tends to
> be loaded at guest-phys address 0. However, when the guest Linux kernel
> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> sync with the phys_base=0 setting visible in the KDUMP header.
> 
> This trips up the "crash" utility.
> 
> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
> can identify QEMU as the originator of the vmcore by finding the QEMU
> notes in the ELF vmcore. If those are present, then "crash" employs a
> heuristic, probing for a phys_base up to 32MB, in 1MB steps.

What advantages does KDUMP have over ELF?

Thanks,
Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Petr Tesarik]

On Tue, 11 Nov 2014 12:27:44 -0500
Christopher Covington <cov@codeaurora.org> wrote:

> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
> > (Note: I'm not subscribed to either qemu-devel or the kexec list; please
> > keep me CC'd.)
> > 
> > QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> > kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
> > 
> > The resultant vmcore is usually analyzed with the "crash" utility.
> > 
> > The original tool producing such files is kdump. Unlike the procedure
> > performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> > kdump kernel), and has more information about the original guest kernel
> > state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> > is opaque.
> > 
> > For this reason, the kdump preparation logic in QEMU hardcodes a number
> > of fields in the kdump header. The direct issue is the "phys_base"
> > field. Refer to dump.c, functions create_header32(), create_header64(),
> > and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> > "0").
> > 
> > http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
> > 
> > http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
> > 
> > This works in most cases, because the guest Linux kernel indeed tends to
> > be loaded at guest-phys address 0. However, when the guest Linux kernel
> > is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> > then the guest Linux kernel is loaded at 16MB, thereby getting out of
> > sync with the phys_base=0 setting visible in the KDUMP header.
> > 
> > This trips up the "crash" utility.
> > 
> > Dave worked around the issue in "crash" for ELF format dumps -- "crash"
> > can identify QEMU as the originator of the vmcore by finding the QEMU
> > notes in the ELF vmcore. If those are present, then "crash" employs a
> > heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> 
> What advantages does KDUMP have over ELF?

It's smaller (data is compressed), and it contains a header with some
useful information (e.g. the crashed kernel's version and release).

HTH,
Petr T

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Christopher Covington]

On 11/12/2014 03:05 AM, Petr Tesarik wrote:
> On Tue, 11 Nov 2014 12:27:44 -0500
> Christopher Covington <cov@codeaurora.org> wrote:
> 
>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>> keep me CC'd.)
>>>
>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>
>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>
>>> The original tool producing such files is kdump. Unlike the procedure
>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>> kdump kernel), and has more information about the original guest kernel
>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>> is opaque.
>>>
>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>> of fields in the kdump header. The direct issue is the "phys_base"
>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>> "0").
>>>
>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>
>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>
>>> This works in most cases, because the guest Linux kernel indeed tends to
>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>
>>> This trips up the "crash" utility.
>>>
>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>
>> What advantages does KDUMP have over ELF?
> 
> It's smaller (data is compressed), and it contains a header with some
> useful information (e.g. the crashed kernel's version and release).

What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

How does QEMU figure out the kernel version information?

Thanks,
Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Petr Tesarik]

On Wed, 12 Nov 2014 08:18:04 -0500
Christopher Covington <cov@codeaurora.org> wrote:

> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
> > On Tue, 11 Nov 2014 12:27:44 -0500
> > Christopher Covington <cov@codeaurora.org> wrote:
> > 
> >> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
> >>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
> >>> keep me CC'd.)
> >>>
> >>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> >>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
> >>>
> >>> The resultant vmcore is usually analyzed with the "crash" utility.
> >>>
> >>> The original tool producing such files is kdump. Unlike the procedure
> >>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> >>> kdump kernel), and has more information about the original guest kernel
> >>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> >>> is opaque.
> >>>
> >>> For this reason, the kdump preparation logic in QEMU hardcodes a number
> >>> of fields in the kdump header. The direct issue is the "phys_base"
> >>> field. Refer to dump.c, functions create_header32(), create_header64(),
> >>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> >>> "0").
> >>>
> >>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
> >>>
> >>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
> >>>
> >>> This works in most cases, because the guest Linux kernel indeed tends to
> >>> be loaded at guest-phys address 0. However, when the guest Linux kernel
> >>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> >>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> >>> sync with the phys_base=0 setting visible in the KDUMP header.
> >>>
> >>> This trips up the "crash" utility.
> >>>
> >>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
> >>> can identify QEMU as the originator of the vmcore by finding the QEMU
> >>> notes in the ELF vmcore. If those are present, then "crash" employs a
> >>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> >>
> >> What advantages does KDUMP have over ELF?
> > 
> > It's smaller (data is compressed), and it contains a header with some
> > useful information (e.g. the crashed kernel's version and release).
> 
> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?

Not the same thing. With KDUMP, each page is compressed separately, so
if a utility like crash needs a page from the middle, it can find it
and unpack it immediately. If we had an ELF.xz, then the whole file
must be unpacked before it can be used. And unpacking a few terabytes
takes ... a while. ;-)

> How does QEMU figure out the kernel version information?

Good question. Who can answer this part?

Petr T

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Christopher Covington]

On 11/12/2014 08:26 AM, Petr Tesarik wrote:
> On Wed, 12 Nov 2014 08:18:04 -0500
> Christopher Covington <cov@codeaurora.org> wrote:
> 
>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>
>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>> keep me CC'd.)
>>>>>
>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>
>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>
>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>> kdump kernel), and has more information about the original guest kernel
>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>> is opaque.
>>>>>
>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>> "0").
>>>>>
>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>
>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>
>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>
>>>>> This trips up the "crash" utility.
>>>>>
>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>
>>>> What advantages does KDUMP have over ELF?
>>>
>>> It's smaller (data is compressed), and it contains a header with some
>>> useful information (e.g. the crashed kernel's version and release).
>>
>> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?
> 
> Not the same thing. With KDUMP, each page is compressed separately, so
> if a utility like crash needs a page from the middle, it can find it
> and unpack it immediately. If we had an ELF.xz, then the whole file
> must be unpacked before it can be used. And unpacking a few terabytes
> takes ... a while. ;-)

Understood on the ELF.xz approach, but why couldn't each page (or maybe a
configurable size) be a SHF_COMPRESSED section?

Thanks,
Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Petr Tesarik]

V Wed, 12 Nov 2014 08:28:54 -0500
Christopher Covington <cov@codeaurora.org> napsáno:

> On 11/12/2014 08:26 AM, Petr Tesarik wrote:
> > On Wed, 12 Nov 2014 08:18:04 -0500
> > Christopher Covington <cov@codeaurora.org> wrote:
> > 
> >> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
> >>> On Tue, 11 Nov 2014 12:27:44 -0500
> >>> Christopher Covington <cov@codeaurora.org> wrote:
> >>>
> >>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
> >>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
> >>>>> keep me CC'd.)
> >>>>>
> >>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> >>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
> >>>>>
> >>>>> The resultant vmcore is usually analyzed with the "crash" utility.
> >>>>>
> >>>>> The original tool producing such files is kdump. Unlike the procedure
> >>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> >>>>> kdump kernel), and has more information about the original guest kernel
> >>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> >>>>> is opaque.
> >>>>>
> >>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
> >>>>> of fields in the kdump header. The direct issue is the "phys_base"
> >>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
> >>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> >>>>> "0").
> >>>>>
> >>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
> >>>>>
> >>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
> >>>>>
> >>>>> This works in most cases, because the guest Linux kernel indeed tends to
> >>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
> >>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> >>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> >>>>> sync with the phys_base=0 setting visible in the KDUMP header.
> >>>>>
> >>>>> This trips up the "crash" utility.
> >>>>>
> >>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
> >>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
> >>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
> >>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> >>>>
> >>>> What advantages does KDUMP have over ELF?
> >>>
> >>> It's smaller (data is compressed), and it contains a header with some
> >>> useful information (e.g. the crashed kernel's version and release).
> >>
> >> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?
> > 
> > Not the same thing. With KDUMP, each page is compressed separately, so
> > if a utility like crash needs a page from the middle, it can find it
> > and unpack it immediately. If we had an ELF.xz, then the whole file
> > must be unpacked before it can be used. And unpacking a few terabytes
> > takes ... a while. ;-)
> 
> Understood on the ELF.xz approach, but why couldn't each page (or maybe a
> configurable size) be a SHF_COMPRESSED section?

A machine with 64TB of RAM (already manufactured by SGI) has
17,179,869,184 pages. When KDUMP (or, actually diskdump) format was
invented, ELF files could have at most 2^16 = 65,536 program headers.
Since then, ELF specification has been extended (PN_XNUM), so the
number of sections can be stored in the sh_info field of the first ELF
section, but that only increases the number of possible sections to
2^32 = 4,294,967,296.

Yes, we could divide memory into larger chunks than pages, but:

  1. you're probably the first one to have the idea, and
  2. this is easy if you save the complete RAM content, but not quite
     that easy if some pages should be filtered out (makedumpfile).

There are a few other (minor) points, e.g.:

  * Each program header consumes 56 bytes in ELF64, while a single
    bit is sufficient in KDUMP compressed files to tell if the
    corresponding page is stored or not.

  * SHF_COMPRESSED currently supports only zlib compression, which is
    rather slow. KDUMP supports zlib, lzo and snappy.

  * Support for KDUMP files is already present in the crash utility,
    while I don't think there is any support for SHF_COMPRESSED
    segments.

In short, SHF_COMPRESSED looks like a viable alternative, but right now
KDUMP is the better choice in terms of features and interoperability.

Just my two cents,
Petr T

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

On 11/12/14 14:28, Christopher Covington wrote:
> On 11/12/2014 08:26 AM, Petr Tesarik wrote:
>> On Wed, 12 Nov 2014 08:18:04 -0500
>> Christopher Covington <cov@codeaurora.org> wrote:
>>
>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>>
>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>>> keep me CC'd.)
>>>>>>
>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>>
>>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>>
>>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>>> kdump kernel), and has more information about the original guest kernel
>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>>> is opaque.
>>>>>>
>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>>> "0").
>>>>>>
>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>>
>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>>
>>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>>
>>>>>> This trips up the "crash" utility.
>>>>>>
>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>>
>>>>> What advantages does KDUMP have over ELF?
>>>>
>>>> It's smaller (data is compressed), and it contains a header with some
>>>> useful information (e.g. the crashed kernel's version and release).
>>>
>>> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?
>>
>> Not the same thing. With KDUMP, each page is compressed separately, so
>> if a utility like crash needs a page from the middle, it can find it
>> and unpack it immediately. If we had an ELF.xz, then the whole file
>> must be unpacked before it can be used. And unpacking a few terabytes
>> takes ... a while. ;-)
> 
> Understood on the ELF.xz approach, but why couldn't each page (or maybe a
> configurable size) be a SHF_COMPRESSED section?

Perhaps it could, technically -- it's just not how Qiao Nuohan
implemented the feature. I didn't research the background for this.

Thanks
Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

On 11/12/14 14:26, Petr Tesarik wrote:
> On Wed, 12 Nov 2014 08:18:04 -0500
> Christopher Covington <cov@codeaurora.org> wrote:
> 
>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>
>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>> keep me CC'd.)
>>>>>
>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>
>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>
>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>> kdump kernel), and has more information about the original guest kernel
>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>> is opaque.
>>>>>
>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>> "0").
>>>>>
>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>
>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>
>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>
>>>>> This trips up the "crash" utility.
>>>>>
>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>
>>>> What advantages does KDUMP have over ELF?
>>>
>>> It's smaller (data is compressed), and it contains a header with some
>>> useful information (e.g. the crashed kernel's version and release).

Another advantage is that all zero-filled pages are represented in the
kdump file by one shared zero page.

The difference in speed of dumping is stunning.

>> What if the ELF dumper used SHF_COMPRESSED or could dump an ELF.xz?
> 
> Not the same thing. With KDUMP, each page is compressed separately, so
> if a utility like crash needs a page from the middle, it can find it
> and unpack it immediately. If we had an ELF.xz, then the whole file
> must be unpacked before it can be used. And unpacking a few terabytes
> takes ... a while. ;-)
> 
>> How does QEMU figure out the kernel version information?
> 
> Good question. Who can answer this part?

I can.

(Apologies for being a bit non-responsive, I'm swamped. I figured I'd
let the discussion unfold a bit between the kdump experts.)

So, QEMU doesn't figure out the kernel version information. It just
dumps the guest-physical frames, and that's it.

I linked the code before that populates the kdump header. The links and
function names are still visible above.

Thanks
Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Christopher Covington]

Thanks Petr and Laszlo for entertaining my questions. I've got one last one if
you have the time.

On 11/12/2014 09:10 AM, Laszlo Ersek wrote:
> On 11/12/14 14:26, Petr Tesarik wrote:
>> On Wed, 12 Nov 2014 08:18:04 -0500
>> Christopher Covington <cov@codeaurora.org> wrote:
>>
>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>>
>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>>> keep me CC'd.)
>>>>>>
>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>>
>>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>>
>>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>>> kdump kernel), and has more information about the original guest kernel
>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>>> is opaque.
>>>>>>
>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>>> "0").
>>>>>>
>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>>
>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>>
>>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>>
>>>>>> This trips up the "crash" utility.
>>>>>>
>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>>
>>>>> What advantages does KDUMP have over ELF?
>>>>
>>>> It's smaller (data is compressed), and it contains a header with some
>>>> useful information (e.g. the crashed kernel's version and release).
> 
> Another advantage is that all zero-filled pages are represented in the
> kdump file by one shared zero page.
> 
> The difference in speed of dumping is stunning.

Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Thanks,
Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

On 11/12/14 15:48, Christopher Covington wrote:
> Thanks Petr and Laszlo for entertaining my questions. I've got one last one if
> you have the time.
> 
> On 11/12/2014 09:10 AM, Laszlo Ersek wrote:
>> On 11/12/14 14:26, Petr Tesarik wrote:
>>> On Wed, 12 Nov 2014 08:18:04 -0500
>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>
>>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>>>
>>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>>>> keep me CC'd.)
>>>>>>>
>>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>>>
>>>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>>>
>>>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>>>> kdump kernel), and has more information about the original guest kernel
>>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>>>> is opaque.
>>>>>>>
>>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>>>> "0").
>>>>>>>
>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>>>
>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>>>
>>>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>>>
>>>>>>> This trips up the "crash" utility.
>>>>>>>
>>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>>>
>>>>>> What advantages does KDUMP have over ELF?
>>>>>
>>>>> It's smaller (data is compressed), and it contains a header with some
>>>>> useful information (e.g. the crashed kernel's version and release).
>>
>> Another advantage is that all zero-filled pages are represented in the
>> kdump file by one shared zero page.
>>
>> The difference in speed of dumping is stunning.
> 
> Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?

Sorry, I don't know what SHT_NOBITS is.

Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Christopher Covington]

On 11/12/2014 10:03 AM, Laszlo Ersek wrote:
> On 11/12/14 15:48, Christopher Covington wrote:
>> Thanks Petr and Laszlo for entertaining my questions. I've got one last one if
>> you have the time.
>>
>> On 11/12/2014 09:10 AM, Laszlo Ersek wrote:
>>> On 11/12/14 14:26, Petr Tesarik wrote:
>>>> On Wed, 12 Nov 2014 08:18:04 -0500
>>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>>
>>>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
>>>>>> On Tue, 11 Nov 2014 12:27:44 -0500
>>>>>> Christopher Covington <cov@codeaurora.org> wrote:
>>>>>>
>>>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>>>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>>>>>>>> keep me CC'd.)
>>>>>>>>
>>>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>>>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>>>>>>>
>>>>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
>>>>>>>>
>>>>>>>> The original tool producing such files is kdump. Unlike the procedure
>>>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>>>>>>>> kdump kernel), and has more information about the original guest kernel
>>>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>>>>>>>> is opaque.
>>>>>>>>
>>>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>>>>>>>> of fields in the kdump header. The direct issue is the "phys_base"
>>>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
>>>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>>>>>>>> "0").
>>>>>>>>
>>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>>>>>>>
>>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>>>>>>>
>>>>>>>> This works in most cases, because the guest Linux kernel indeed tends to
>>>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>>>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>>>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>>>>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
>>>>>>>>
>>>>>>>> This trips up the "crash" utility.
>>>>>>>>
>>>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>>>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
>>>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
>>>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
>>>>>>>
>>>>>>> What advantages does KDUMP have over ELF?
>>>>>>
>>>>>> It's smaller (data is compressed), and it contains a header with some
>>>>>> useful information (e.g. the crashed kernel's version and release).
>>>
>>> Another advantage is that all zero-filled pages are represented in the
>>> kdump file by one shared zero page.
>>>
>>> The difference in speed of dumping is stunning.
>>
>> Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?
> 
> Sorry, I don't know what SHT_NOBITS is.

My newbie understanding is that SHT_NOBITS is the section type of the .bss
section in an everyday executable. It's what makes the section only need a
header table entry in the ELF file, and not sh_size (Size) worth of zeros. The
ELF loader will essentially zero memory beginning at sh_addr (Address) for
sh_size (Size).

http://stackoverflow.com/questions/22855320/address-space-of-a-bss-section-space-in-elf-file

http://people.redhat.com/mpolacek/src/devconf2012.pdf

Chris

-- 
Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Petr Tesarik]

On Wed, 12 Nov 2014 10:43:59 -0500
Christopher Covington <cov@codeaurora.org> wrote:

> On 11/12/2014 10:03 AM, Laszlo Ersek wrote:
> > On 11/12/14 15:48, Christopher Covington wrote:
> >> Thanks Petr and Laszlo for entertaining my questions. I've got one last one if
> >> you have the time.
> >>
> >> On 11/12/2014 09:10 AM, Laszlo Ersek wrote:
> >>> On 11/12/14 14:26, Petr Tesarik wrote:
> >>>> On Wed, 12 Nov 2014 08:18:04 -0500
> >>>> Christopher Covington <cov@codeaurora.org> wrote:
> >>>>
> >>>>> On 11/12/2014 03:05 AM, Petr Tesarik wrote:
> >>>>>> On Tue, 11 Nov 2014 12:27:44 -0500
> >>>>>> Christopher Covington <cov@codeaurora.org> wrote:
> >>>>>>
> >>>>>>> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
> >>>>>>>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
> >>>>>>>> keep me CC'd.)
> >>>>>>>>
> >>>>>>>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
> >>>>>>>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
> >>>>>>>>
> >>>>>>>> The resultant vmcore is usually analyzed with the "crash" utility.
> >>>>>>>>
> >>>>>>>> The original tool producing such files is kdump. Unlike the procedure
> >>>>>>>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
> >>>>>>>> kdump kernel), and has more information about the original guest kernel
> >>>>>>>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
> >>>>>>>> is opaque.
> >>>>>>>>
> >>>>>>>> For this reason, the kdump preparation logic in QEMU hardcodes a number
> >>>>>>>> of fields in the kdump header. The direct issue is the "phys_base"
> >>>>>>>> field. Refer to dump.c, functions create_header32(), create_header64(),
> >>>>>>>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
> >>>>>>>> "0").
> >>>>>>>>
> >>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
> >>>>>>>>
> >>>>>>>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
> >>>>>>>>
> >>>>>>>> This works in most cases, because the guest Linux kernel indeed tends to
> >>>>>>>> be loaded at guest-phys address 0. However, when the guest Linux kernel
> >>>>>>>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
> >>>>>>>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
> >>>>>>>> sync with the phys_base=0 setting visible in the KDUMP header.
> >>>>>>>>
> >>>>>>>> This trips up the "crash" utility.
> >>>>>>>>
> >>>>>>>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
> >>>>>>>> can identify QEMU as the originator of the vmcore by finding the QEMU
> >>>>>>>> notes in the ELF vmcore. If those are present, then "crash" employs a
> >>>>>>>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> >>>>>>>
> >>>>>>> What advantages does KDUMP have over ELF?
> >>>>>>
> >>>>>> It's smaller (data is compressed), and it contains a header with some
> >>>>>> useful information (e.g. the crashed kernel's version and release).
> >>>
> >>> Another advantage is that all zero-filled pages are represented in the
> >>> kdump file by one shared zero page.
> >>>
> >>> The difference in speed of dumping is stunning.
> >>
> >> Would you expect using SHT_NOBITS to give a similar speedup to the ELF dumper?
> > 
> > Sorry, I don't know what SHT_NOBITS is.
> 
> My newbie understanding is that SHT_NOBITS is the section type of the .bss
> section in an everyday executable.

Heh, yes and no. Let's clarify a few things.

First, a Linux kernel dump (or a QEMU ELF dump) does not contain any
sections. It only contains program headers. The reason is that program
headers can specify both the virtual address and the physical address.
BTW this feature is used to determine the physical base of the Linux
kernel when dumping via kexec. Sections can only specify the virtual
address. Of course, program headers can achieve an effect similar to
SHT_NOBITS: by specifying a larger memory size than file size.

Now, all this does not mean you can't create a new standard that uses
sections instead (in fact, Xen DomU dumps already use ELF sections).
But of course, this new standard won't be understood by the existing
tools until somebody (you?) adds support for it. Same thing is true
for the SHF_COMPRESSED section flag.

Anyway, reuse of the zero page is a minor point. The main difference in
speed comes from using a faster compression algorithm (LZO or snappy).
Of course, nothing prevents you from using those algorithms in ELF
compressed sections, but you'd have to extend the ELF standard first,
adding magic numbers for these algorithms. To me it sounds like a long
way to go.

In short, there's no technical reason why ELF couldn't achieve results
similar to KDUMP. It's "merely" not implemented, and for sure, I'm not
going to push all the necessary changes. ;-)

Petr T

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

On 11/11/14 18:27, Christopher Covington wrote:
> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>> keep me CC'd.)
>>
>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>
>> The resultant vmcore is usually analyzed with the "crash" utility.
>>
>> The original tool producing such files is kdump. Unlike the procedure
>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>> kdump kernel), and has more information about the original guest kernel
>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>> is opaque.
>>
>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>> of fields in the kdump header. The direct issue is the "phys_base"
>> field. Refer to dump.c, functions create_header32(), create_header64(),
>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>> "0").
>>
>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>
>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>
>> This works in most cases, because the guest Linux kernel indeed tends to
>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>> sync with the phys_base=0 setting visible in the KDUMP header.
>>
>> This trips up the "crash" utility.
>>
>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>> can identify QEMU as the originator of the vmcore by finding the QEMU
>> notes in the ELF vmcore. If those are present, then "crash" employs a
>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> 
> What advantages does KDUMP have over ELF?

This has been discussed, but I'd like to give a short perspective from
personal experience.

The more obvious advantage is the smaller size, due to (a) per-page
compression (which preserves random-access for "crash"), and (b) zero
page sharing. A smaller dump file is easier to store, and easier to
upload if you're requesting assitance with debugging.

The perhaps less obvious advantage is the speed at which qemu writes the
dump. We're talking orders of magnitude, especially on rotational media.
This is because lzo and snappy are *incredibly* fast (put differently:
they incur very little CPU penalty for the same guest RAM size). The CPU
penalty is actually so small that in almost all cases the dumping
procedure stays IO-bound (in my experience: even on an SSD!). Now
combine that with a potential reduction of 4GB -> 256MB in size: that's
a sixteen-fold speedup.

(I'm allowed to praise this qemu feature, I didn't write it. :))

Thanks
Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

adding back a few CC's because this discussion is useful

On 11/12/14 19:43, Petr Tesarik wrote:
> V Wed, 12 Nov 2014 15:50:32 +0100
> Laszlo Ersek <lersek@redhat.com> napsáno:
> 
>> On 11/12/14 09:04, Petr Tesarik wrote:
>>> On Wed, 12 Nov 2014 12:08:38 +0900 (JST)
>>> HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> wrote:
>>
>>>> Anyway, phys_base is kernel information. To make it available for qemu
>>>> side, there's need to prepare a mechanism for qemu to have any access
>>>> to it.
>>>
>>> Yes. I wonder if you can have access without some sort of co-operation
>>> from the guest kernel itself. I guess not.
>>
>> Propagating any kind of additional information from the guest kernel
>> (which is unprivileged and potentially malicious) to the host-side qemu
>> process (which is by definition more privileged, although still confined
>> by various measures) is something we'd explicitly like to avoid.
>>
>> Think of it like this. I throw a physical box at you, running Linux,
>> that has frozen in time. Can "crash" work with nothing else but the
>> contents of the memory, and information about the CPUs?
> 
> If only you could save the _complete_ state of the CPU... For example
> the content of CR3 would be quite useful.

(1) CR3 is already saved, in both the ELF and the kdump compressed formats.

- ELF case:

qmp_dump_guest_memory() [dump.c]
  create_vmcore()
    dump_begin()
      write_elf64_notes()

        loop from 1 to #vcpu:
          cpu_write_elf64_note() [qom/cpu.c]
            x86_64_write_elf64_note() [target-i386/arch_dump.c]
              writes "CORE"

        loop from 1 to #vcpu:
          cpu_write_elf64_qemunote() [qom/cpu.c]
            x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
              cpu_write_qemu_note()
                qemu_get_cpustate()
                  s->cr[3] = env->cr[3]; <---------- here
                writes "QEMU"

Hence, the information is part of the QEMU note.

- kdump case:

qmp_dump_guest_memory() [dump.c]
  create_kdump_vmcore()
    write_dump_header()
      create_header64()
        write_elf64_notes()
          [... same as above ...]

The trick here is that the note-writer functions use a callback function
for actually outputting the data. So while in the ELF case the stuff
goes directly to a file, in the kdump case the notes are first saved in
a memory buffer, and then later saved in the file at offset
KdumpSubHeader64.offset_note. (... Which is then represented in the
flattened file format of course.)

So, the information is there in both cases.


(2) Dave -- this just made me realize that the QEMU note is *already*
there in the kdump file as well; pointed-to by
KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.

>From your other email
<http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:

>     sub_header_kdump: 1c9cff0
>              phys_base: 0
>             dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
>                  split: 0
>              start_pfn: (unused)
>                end_pfn: (unused)
>      offset_vmcoreinfo: 0 (0x0)
>        size_vmcoreinfo: 0 (0x0)
>            offset_note: 4200 (0x1068)       <----------- here
>              size_note: 3232 (0xca0)        <-----------
>     num_prstatus_notes: 4
>              notes_buf: 1c9e000
>               notes[0]: 1c9e000
>               notes[1]: 1c9e164
>               notes[2]: 1c9e2c8
>               notes[3]: 1c9e42c
>     NT_PRSTATUS_offset: 1068
>                         11cc
>                         1330
>                         1494
>       offset_eraseinfo: 0 (0x0)
>         size_eraseinfo: 0 (0x0)
>           start_pfn_64: (unused)
>             end_pfn_64: (unused)
>           max_mapnr_64: 1245184 (0x130000)

Can you fetch that in "crash"? If you can, then there's nothing to do on
the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

I think "crash" already iterates over all of the notes in the note
buffer, but skips everything different from NT_PRSTATUS.


(3) Regarding the structure of the notes, we have to consider the
placement of the notes and their internal structure. The placement is
different between the ELF and the KDUMP file format. The internal
structure of the notes is identical between the two file formats.

For example, for a 4 VCPU guest, you end up with note names like

  CORE
  CORE
  CORE
  CORE
  QEMU
  QEMU
  QEMU
  QEMU

All of these are Elf64_Nhdr structures. The CORE ones have type
NT_PRSTATUS, and the QEMU ones have type 0.

(3a) The placement in the ELF file is already handled by "crash". Each
note "simply" gets its own ELF note segment/section.

(3b) In the kdump file, the Elf64_Nhdr structures (8 pieces in total, in
the above example -- 4x CORE, 4x QEMU) are concatenated in that order,
and finally stored at "offset_note".

(3c) Regarding the internal structure of the notes. The CORE ones are
already known and handled. The QEMU notes have the following structure:

> Elf64_Nhdr:
> n_namesz: 5 ("QEMU")
> n_descsz: 432
>   n_type: 0 (?)
>           000001b000000001 0000000000000000
            |------||------| |--------------|
            size    version  rax

>           0000000000000000 0000000000000000
            |--------------| |--------------|
            rbx              rcx

>           0000000000000000 0000000000000001
            |--------------| |--------------|
            rdx              rsi

>           ffffffff81dd5228 ffffffff81a01ec8
            |--------------| |--------------|
            rdi              rsp

>           ffffffff81a01ec8 0000000000000000
            |--------------| |--------------|
            rbp              r8

>           0000000000000000 00000013911d5f29
            |--------------| |--------------|
            r9               r10

>           0000000000000000 ffffffff81c00480
            |--------------| |--------------|
            r11              r12

>           0000000000000000 ffffffffffffffff
            |--------------| |--------------|
            r13              r14

>           000000000309f000 ffffffff810375ab
            |--------------| |--------------|
            r15              rip

>           0000000000000246 ffffffff00000010
            |--------------| |------||------|
            rflags           cs/lim  cs/sel

>           0000000000a09b00 0000000000000000
            |------||------| |--------------|
            cs/pad  cs/flags cs/base

>           ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ds/lim  ds/sel   ds/pad  ds/flags

>           0000000000000000 ffffffff00000018
            |--------------| |------||------|
            ds/base          es/lim  es/sel

>           0000000000c09300 0000000000000000
            |------||------| |--------------|
            es/pad  es/flags es/base

>           ffffffff00000000 0000000000000000
            |------||------| |------||------|
            fs/lim  fs/sel   fs/pad  fs/flags

>           0000000000000000 ffffffff00000000
            |--------------| |------||------|
            fs/base          gs/lim  gs/sel

>           0000000000000000 ffff880003200000
            |------||------| |--------------|
            gs/pad  gs/flags gs/base

>           ffffffff00000018 0000000000c09300
            |------||------| |------||------|
            ss/lim  ss/sel   ss/pad  ss/flags

>           0000000000000000 ffffffff00000000
            |--------------| |------||------|
            ss/base          ldt...

>           0000000000000000 0000000000000000
            |------||------| |--------------|
                                       ...ldt

>           0000208700000040 0000000000008b00
            |------||------| |------||------|
            tr...

>           ffff880003213b40 0000007f00000000
            |--------------| |------||------|
                       ...tr gdt...

>           0000000000000000 ffff880003204000
            |------||------| |--------------|
                                       ...gdt

>           00000fff00000000 0000000000000000
            |------||------| |------||------|
            idt...

>           ffffffff81dd2000 000000008005003b
            |--------------| |--------------|
                      ...idt cr0

>           0000000000000000 0000000001b2e000
            |--------------| |--------------|
            cr1              cr2

>           0000000007b18000 00000000000006f0
            |--------------| |--------------|
            cr3              cr4

>From "target-i386/arch_dump.c":

> struct QEMUCPUSegment {
>     uint32_t selector;
>     uint32_t limit;
>     uint32_t flags;
>     uint32_t pad;
>     uint64_t base;
> };
>
> typedef struct QEMUCPUSegment QEMUCPUSegment;
>
> struct QEMUCPUState {
>     uint32_t version;
>     uint32_t size;
>     uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp;
>     uint64_t r8, r9, r10, r11, r12, r13, r14, r15;
>     uint64_t rip, rflags;
>     QEMUCPUSegment cs, ds, es, fs, gs, ss;
>     QEMUCPUSegment ldt, tr, gdt, idt;
>     uint64_t cr[5];
> };
>
> typedef struct QEMUCPUState QEMUCPUState;


Summary: I think the info is all there.

Thanks
Laszlo

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Dave Anderson]

----- Original Message -----
> adding back a few CC's because this discussion is useful
> 
> On 11/12/14 19:43, Petr Tesarik wrote:
> > V Wed, 12 Nov 2014 15:50:32 +0100
> > Laszlo Ersek <lersek@redhat.com> napsáno:
> > 
> >> On 11/12/14 09:04, Petr Tesarik wrote:
> >>> On Wed, 12 Nov 2014 12:08:38 +0900 (JST)
> >>> HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> wrote:
> >>
> >>>> Anyway, phys_base is kernel information. To make it available for qemu
> >>>> side, there's need to prepare a mechanism for qemu to have any access
> >>>> to it.
> >>>
> >>> Yes. I wonder if you can have access without some sort of co-operation
> >>> from the guest kernel itself. I guess not.
> >>
> >> Propagating any kind of additional information from the guest kernel
> >> (which is unprivileged and potentially malicious) to the host-side qemu
> >> process (which is by definition more privileged, although still confined
> >> by various measures) is something we'd explicitly like to avoid.
> >>
> >> Think of it like this. I throw a physical box at you, running Linux,
> >> that has frozen in time. Can "crash" work with nothing else but the
> >> contents of the memory, and information about the CPUs?
> > 
> > If only you could save the _complete_ state of the CPU... For example
> > the content of CR3 would be quite useful.
> 
> (1) CR3 is already saved, in both the ELF and the kdump compressed formats.
> 
> - ELF case:
> 
> qmp_dump_guest_memory() [dump.c]
>   create_vmcore()
>     dump_begin()
>       write_elf64_notes()
> 
>         loop from 1 to #vcpu:
>           cpu_write_elf64_note() [qom/cpu.c]
>             x86_64_write_elf64_note() [target-i386/arch_dump.c]
>               writes "CORE"
> 
>         loop from 1 to #vcpu:
>           cpu_write_elf64_qemunote() [qom/cpu.c]
>             x86_cpu_write_elf64_qemunote() [target-i386/arch_dump.c]
>               cpu_write_qemu_note()
>                 qemu_get_cpustate()
>                   s->cr[3] = env->cr[3]; <---------- here
>                 writes "QEMU"
> 
> Hence, the information is part of the QEMU note.
> 
> - kdump case:
> 
> qmp_dump_guest_memory() [dump.c]
>   create_kdump_vmcore()
>     write_dump_header()
>       create_header64()
>         write_elf64_notes()
>           [... same as above ...]
> 
> The trick here is that the note-writer functions use a callback function
> for actually outputting the data. So while in the ELF case the stuff
> goes directly to a file, in the kdump case the notes are first saved in
> a memory buffer, and then later saved in the file at offset
> KdumpSubHeader64.offset_note. (... Which is then represented in the
> flattened file format of course.)
> 
> So, the information is there in both cases.
> 
> 
> (2) Dave -- this just made me realize that the QEMU note is *already*
> there in the kdump file as well; pointed-to by
> KdumpSubHeader64.offset_note, for a length of KdumpSubHeader64.note_size.
> 
> From your other email
> <http://thread.gmane.org/gmane.linux.kernel.kexec/12787/focus=12797>:
> 
> >     sub_header_kdump: 1c9cff0
> >              phys_base: 0
> >             dump_level: 1 (0x1) (DUMP_EXCLUDE_ZERO)
> >                  split: 0
> >              start_pfn: (unused)
> >                end_pfn: (unused)
> >      offset_vmcoreinfo: 0 (0x0)
> >        size_vmcoreinfo: 0 (0x0)
> >            offset_note: 4200 (0x1068)       <----------- here
> >              size_note: 3232 (0xca0)        <-----------
> >     num_prstatus_notes: 4
> >              notes_buf: 1c9e000
> >               notes[0]: 1c9e000
> >               notes[1]: 1c9e164
> >               notes[2]: 1c9e2c8
> >               notes[3]: 1c9e42c
> >     NT_PRSTATUS_offset: 1068
> >                         11cc
> >                         1330
> >                         1494
> >       offset_eraseinfo: 0 (0x0)
> >         size_eraseinfo: 0 (0x0)
> >           start_pfn_64: (unused)
> >             end_pfn_64: (unused)
> >           max_mapnr_64: 1245184 (0x130000)
> 
> Can you fetch that in "crash"? If you can, then there's nothing to do on
> the qemu side (and I'll have to apologize for spamming a bunch of lists :/).

Sure enough...

I was just playing with process_el64_notes() to check/read the note name strings,
and noticed that I can certainly see them.  But as you noted, only the NT_PRSTATUS
notes are stored in the "notes[]" array. so I was under the impression that the
QEMU notes were completely missing.

That being the case -- we're pretty much done!

I'll put a patch in the next upstream release of crash.

Thanks,
  Dave




> 
> I think "crash" already iterates over all of the notes in the note
> buffer, but skips everything different from NT_PRSTATUS.
> 
> 
> (3) Regarding the structure of the notes, we have to consider the
> placement of the notes and their internal structure. The placement is
> different between the ELF and the KDUMP file format. The internal
> structure of the notes is identical between the two file formats.
> 
> For example, for a 4 VCPU guest, you end up with note names like
> 
>   CORE
>   CORE
>   CORE
>   CORE
>   QEMU
>   QEMU
>   QEMU
>   QEMU
> 
> All of these are Elf64_Nhdr structures. The CORE ones have type
> NT_PRSTATUS, and the QEMU ones have type 0.
> 
> (3a) The placement in the ELF file is already handled by "crash". Each
> note "simply" gets its own ELF note segment/section.
> 
> (3b) In the kdump file, the Elf64_Nhdr structures (8 pieces in total, in
> the above example -- 4x CORE, 4x QEMU) are concatenated in that order,
> and finally stored at "offset_note".
> 
> (3c) Regarding the internal structure of the notes. The CORE ones are
> already known and handled. The QEMU notes have the following structure:
> 
> > Elf64_Nhdr:
> > n_namesz: 5 ("QEMU")
> > n_descsz: 432
> >   n_type: 0 (?)
> >           000001b000000001 0000000000000000
>             |------||------| |--------------|
>             size    version  rax
> 
> >           0000000000000000 0000000000000000
>             |--------------| |--------------|
>             rbx              rcx
> 
> >           0000000000000000 0000000000000001
>             |--------------| |--------------|
>             rdx              rsi
> 
> >           ffffffff81dd5228 ffffffff81a01ec8
>             |--------------| |--------------|
>             rdi              rsp
> 
> >           ffffffff81a01ec8 0000000000000000
>             |--------------| |--------------|
>             rbp              r8
> 
> >           0000000000000000 00000013911d5f29
>             |--------------| |--------------|
>             r9               r10
> 
> >           0000000000000000 ffffffff81c00480
>             |--------------| |--------------|
>             r11              r12
> 
> >           0000000000000000 ffffffffffffffff
>             |--------------| |--------------|
>             r13              r14
> 
> >           000000000309f000 ffffffff810375ab
>             |--------------| |--------------|
>             r15              rip
> 
> >           0000000000000246 ffffffff00000010
>             |--------------| |------||------|
>             rflags           cs/lim  cs/sel
> 
> >           0000000000a09b00 0000000000000000
>             |------||------| |--------------|
>             cs/pad  cs/flags cs/base
> 
> >           ffffffff00000018 0000000000c09300
>             |------||------| |------||------|
>             ds/lim  ds/sel   ds/pad  ds/flags
> 
> >           0000000000000000 ffffffff00000018
>             |--------------| |------||------|
>             ds/base          es/lim  es/sel
> 
> >           0000000000c09300 0000000000000000
>             |------||------| |--------------|
>             es/pad  es/flags es/base
> 
> >           ffffffff00000000 0000000000000000
>             |------||------| |------||------|
>             fs/lim  fs/sel   fs/pad  fs/flags
> 
> >           0000000000000000 ffffffff00000000
>             |--------------| |------||------|
>             fs/base          gs/lim  gs/sel
> 
> >           0000000000000000 ffff880003200000
>             |------||------| |--------------|
>             gs/pad  gs/flags gs/base
> 
> >           ffffffff00000018 0000000000c09300
>             |------||------| |------||------|
>             ss/lim  ss/sel   ss/pad  ss/flags
> 
> >           0000000000000000 ffffffff00000000
>             |--------------| |------||------|
>             ss/base          ldt...
> 
> >           0000000000000000 0000000000000000
>             |------||------| |--------------|
>                                        ...ldt
> 
> >           0000208700000040 0000000000008b00
>             |------||------| |------||------|
>             tr...
> 
> >           ffff880003213b40 0000007f00000000
>             |--------------| |------||------|
>                        ...tr gdt...
> 
> >           0000000000000000 ffff880003204000
>             |------||------| |--------------|
>                                        ...gdt
> 
> >           00000fff00000000 0000000000000000
>             |------||------| |------||------|
>             idt...
> 
> >           ffffffff81dd2000 000000008005003b
>             |--------------| |--------------|
>                       ...idt cr0
> 
> >           0000000000000000 0000000001b2e000
>             |--------------| |--------------|
>             cr1              cr2
> 
> >           0000000007b18000 00000000000006f0
>             |--------------| |--------------|
>             cr3              cr4
> 
> From "target-i386/arch_dump.c":
> 
> > struct QEMUCPUSegment {
> >     uint32_t selector;
> >     uint32_t limit;
> >     uint32_t flags;
> >     uint32_t pad;
> >     uint64_t base;
> > };
> >
> > typedef struct QEMUCPUSegment QEMUCPUSegment;
> >
> > struct QEMUCPUState {
> >     uint32_t version;
> >     uint32_t size;
> >     uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp;
> >     uint64_t r8, r9, r10, r11, r12, r13, r14, r15;
> >     uint64_t rip, rflags;
> >     QEMUCPUSegment cs, ds, es, fs, gs, ss;
> >     QEMUCPUSegment ldt, tr, gdt, idt;
> >     uint64_t cr[5];
> > };
> >
> > typedef struct QEMUCPUState QEMUCPUState;
> 
> 
> Summary: I think the info is all there.
> 
> Thanks
> Laszlo
> 

Re: [Qemu-devel] [Crash-utility] uniquely identifying KDUMP files that originate from QEMU

[Dave Anderson]

----- Original Message -----

> > Can you fetch that in "crash"? If you can, then there's nothing to do on
> > the qemu side (and I'll have to apologize for spamming a bunch of lists
> > :/).

Well, let's be clear -- I was the one who put you up to it...

But no apology is required -- and in fact, if today's discussion results in
the "phys_base" vmcoreinfo issue being resolved in the kernel, we'll all be
eternally grateful.

Thanks again,
  Dave

Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Petr Tesarik]

On Wed, 12 Nov 2014 21:30:20 +0100
Laszlo Ersek <lersek@redhat.com> wrote:

> adding back a few CC's because this discussion is useful
> 
> On 11/12/14 19:43, Petr Tesarik wrote:
> > V Wed, 12 Nov 2014 15:50:32 +0100
> > Laszlo Ersek <lersek@redhat.com> napsáno:
> > 
> >> On 11/12/14 09:04, Petr Tesarik wrote:
> >>> On Wed, 12 Nov 2014 12:08:38 +0900 (JST)
> >>> HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> wrote:
> >>
> >>>> Anyway, phys_base is kernel information. To make it available for qemu
> >>>> side, there's need to prepare a mechanism for qemu to have any access
> >>>> to it.
> >>>
> >>> Yes. I wonder if you can have access without some sort of co-operation
> >>> from the guest kernel itself. I guess not.
> >>
> >> Propagating any kind of additional information from the guest kernel
> >> (which is unprivileged and potentially malicious) to the host-side qemu
> >> process (which is by definition more privileged, although still confined
> >> by various measures) is something we'd explicitly like to avoid.
> >>
> >> Think of it like this. I throw a physical box at you, running Linux,
> >> that has frozen in time. Can "crash" work with nothing else but the
> >> contents of the memory, and information about the CPUs?
> > 
> > If only you could save the _complete_ state of the CPU... For example
> > the content of CR3 would be quite useful.
> 
> (1) CR3 is already saved, in both the ELF and the kdump compressed formats.

Sweet. :-)

So, there's no need for any heuristics. Since CR3 gives the physical
address of the PML4 table, I can use it to translate __START_KERNEL_map
(0xffffffff80000000UL on all Linux kernels since introduction of
x86_64) to a physical address and compute phys_base from that.

In fact, QEMU could do the same if you can live with hardcoding a
Linux-kernel specific constant into the tool...

Petr T