[Qemu-devel] Booting kernels with PVHVM documentation?

[Qemu-devel] Booting kernels with PVHVM documentation?

[Alex Bennée]

Hi,

I've been looking at using PVH as an alternative to a long bios boot
sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
hard to piece together all the bits but I naively thought it would just
be a case of adding a few ELF NOTES to my boot.S with something like:

          ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
          ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
          ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry == virtbase */
          ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)

          .code64
          .section .text
          /* Kernel Entry Point */
  .global _start
  _start:
          // Setup stack ASAP
          movq $stack_end,%rsp

However I'm running into lots of head scratching as the get_elf_note
code seems to skip over the notes before failing:

  ./qemu-system-x86_64 -monitor none -display none \
    -chardev stdio,id=out -device isa-debugcon,chardev=out \
    -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
  load_elf64: processing hdr:0 of type 1
  load_elf64: processing hdr:1 of type 4
  get_elf_note_type64: looking for type 18, first is 3
  get_elf_note_type64: 4/20
  get_elf_note_type64: offset is 36
  get_elf_note_type64: note is 0
  get_elf_note_type64: 0/123713
  get_elf_note_type64: offset is 123728
  load_elf64: processing hdr:2 of type 1685382481
  qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note

So I thought I'd go back to the Linux kernel and see if I could get it
to boot up. So I built an x86_64 kernel with:

  CONFIG_XEN_PVHVM=y
  CONFIG_XEN_PVHVM_SMP=y
  CONFIG_XEN_PVH=y
  CONFIG_PVH=y

And tried to boot that, it certainly gets a lot further but in detecting
the note 18 it's looking for but then doesn't provide any output. So I
started digging around the patches and saw talk of a PVH option ROM
which does all the x86 mode escalation before booting the kernel.
However I was unable to find any documentation about if I should be
adding this manually to my command line or if it is auto-magiced into
place. So I have a number of questions:

  * what's the canonical command line for booting a Linux PVHVM kernel?
  * should this work in TCG as well?
  * are they any special linker rules required for the Xen.notes?

And finally:

  * is this idea of mine a weird abuse of the PVHVM boot protocol or
    does it make sense?

Thanks in advance for any elucidation ;-)

--
Alex Bennée

Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Liam Merwick]

On 06/03/2019 17:51, Alex Bennée wrote:
> 
> Hi,
> 
> I've been looking at using PVH as an alternative to a long bios boot
> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
> hard to piece together all the bits but I naively thought it would just
> be a case of adding a few ELF NOTES to my boot.S with something like:
> 
>            ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
>            ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
>            ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry == virtbase */
>            ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)
> 
>            .code64
>            .section .text
>            /* Kernel Entry Point */
>    .global _start
>    _start:
>            // Setup stack ASAP
>            movq $stack_end,%rsp
> 
> However I'm running into lots of head scratching as the get_elf_note
> code seems to skip over the notes before failing:
> 
>    ./qemu-system-x86_64 -monitor none -display none \
>      -chardev stdio,id=out -device isa-debugcon,chardev=out \
>      -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
>    load_elf64: processing hdr:0 of type 1
>    load_elf64: processing hdr:1 of type 4
>    get_elf_note_type64: looking for type 18, first is 3
>    get_elf_note_type64: 4/20
>    get_elf_note_type64: offset is 36
>    get_elf_note_type64: note is 0
>    get_elf_note_type64: 0/123713
>    get_elf_note_type64: offset is 123728
>    load_elf64: processing hdr:2 of type 1685382481
>    qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note



What does 'readelf -Wn ./tests/hello' or 'readelf -p .notes 
./tests/hello' show?



> 
> So I thought I'd go back to the Linux kernel and see if I could get it
> to boot up. So I built an x86_64 kernel with:
> 
>    CONFIG_XEN_PVHVM=y
>    CONFIG_XEN_PVHVM_SMP=y
>    CONFIG_XEN_PVH=y
>    CONFIG_PVH=y
> 
> And tried to boot that, it certainly gets a lot further but in detecting
> the note 18 it's looking for but then doesn't provide any output. So I
> started digging around the patches and saw talk of a PVH option ROM
> which does all the x86 mode escalation before booting the kernel.
> However I was unable to find any documentation about if I should be
> adding this manually to my command line or if it is auto-magiced into
> place. So I have a number of questions:
> 
>    * what's the canonical command line for booting a Linux PVHVM kernel?

I had been using '-bios ./qboot.bin' (using 
https://github.com/bonzini/qboot/pull/17/files) but Stefano's changes 
mean that isn't necessary. Here's a command line I used

sudo $QEMU \
       -name testvm01 \
       -machine 
q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \
       -cpu host \
       -m 1024,maxmem=20G,slots=2 \
       -smp 1 \
       -no-user-config \
       -nodefaults \
       -object 
memory-backend-file,id=mem0,share,mem-path=$IMAGE,size=235929600 \
       -device nvdimm,memdev=mem0,id=nv0 \
       -append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw 
tsc=reliable no_timer_check noreplace-smp init=/usr/lib/systemd/systemd 
rootfstype=ext4 rcupdate.rcu_expedited=1 reboot=f' \
       -no-reboot \
       -serial mon:stdio \
       -kernel $KERNEL


>    * should this work in TCG as well?
>    * are they any special linker rules required for the Xen.notes?
> 
> And finally:
> 
>    * is this idea of mine a weird abuse of the PVHVM boot protocol or
>      does it make sense?
> 


Regards,
Liam

Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Alex Bennée]

Liam Merwick <liam.merwick@oracle.com> writes:

> On 06/03/2019 17:51, Alex Bennée wrote:
>>
>> Hi,
>>
>> I've been looking at using PVH as an alternative to a long bios boot
>> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
>> hard to piece together all the bits but I naively thought it would just
>> be a case of adding a few ELF NOTES to my boot.S with something like:
>>
>>            ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
>>            ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
>>            ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry == virtbase */
>>            ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)
>>
>>            .code64
>>            .section .text
>>            /* Kernel Entry Point */
>>    .global _start
>>    _start:
>>            // Setup stack ASAP
>>            movq $stack_end,%rsp
>>
>> However I'm running into lots of head scratching as the get_elf_note
>> code seems to skip over the notes before failing:
>>
>>    ./qemu-system-x86_64 -monitor none -display none \
>>      -chardev stdio,id=out -device isa-debugcon,chardev=out \
>>      -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
>>    load_elf64: processing hdr:0 of type 1
>>    load_elf64: processing hdr:1 of type 4
>>    get_elf_note_type64: looking for type 18, first is 3
>>    get_elf_note_type64: 4/20
>>    get_elf_note_type64: offset is 36
>>    get_elf_note_type64: note is 0
>>    get_elf_note_type64: 0/123713
>>    get_elf_note_type64: offset is 123728
>>    load_elf64: processing hdr:2 of type 1685382481
>>    qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
>
>
>
> What does 'readelf -Wn ./tests/hello' or 'readelf -p .notes
> ./tests/hello' show?

$ readelf -Wn tests/hello

Displaying notes found in: .note.gnu.build-id
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)         Build ID: 919289306d162384e3c8a38441088cf6c511f623

Displaying notes found in: .note.Xen
  Owner                 Data size       Description
  Xen                  0x00000008       Unknown note type: (0x00000003)    description data: 00 00 10 00 00 00 00 00
  Xen                  0x00000008       NT_VERSION (version)       description data: 40 00 10 00 00 00 00 00
  Xen                  0x00000008       Unknown note type: (0x00000012)    description data: 00 00 00 00 00 00 00 00
  Xen                  0x00000008       Unknown note type: (0x00000004)    description data: 00 00 00 00 00 00 00 00

$ readelf -p .notes tests/hello
readelf: Warning: Section '.notes' was not dumped because it does not exist!

>>
>> So I thought I'd go back to the Linux kernel and see if I could get it
>> to boot up. So I built an x86_64 kernel with:
>>
>>    CONFIG_XEN_PVHVM=y
>>    CONFIG_XEN_PVHVM_SMP=y
>>    CONFIG_XEN_PVH=y
>>    CONFIG_PVH=y
>>
>> And tried to boot that, it certainly gets a lot further but in detecting
>> the note 18 it's looking for but then doesn't provide any output. So I
>> started digging around the patches and saw talk of a PVH option ROM
>> which does all the x86 mode escalation before booting the kernel.
>> However I was unable to find any documentation about if I should be
>> adding this manually to my command line or if it is auto-magiced into
>> place. So I have a number of questions:
>>
>>    * what's the canonical command line for booting a Linux PVHVM kernel?
>
> I had been using '-bios ./qboot.bin' (using
> https://github.com/bonzini/qboot/pull/17/files) but Stefano's changes
> mean that isn't necessary. Here's a command line I used

I think they are all in.

>
> sudo $QEMU \
>       -name testvm01 \
>       -machine
> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \

Hmm if I use the q35 machine type I might have to rethink the approach
for getting output and results as I'm currently using the isa debugcon
and debug exit devices.

I wonder how much framework I would need for a minimal virtio or PV
serial implementation? Currently the ISA stuff is dumb as bricks and
simple:

        /* Output a single character to serial port */
        .global __sys_outc
__sys_outc:
        pushl %ebp
        movl %esp, %ebp
	out %al,$0xE9
        movl %ebp, %esp
        popl %ebp
        ret

and:

        /* output any non-zero result in eax to isa-debug-exit device */
        test %al, %al
        jz 1f
        out %ax, $0xf4

1:      /* QEMU ACPI poweroff */
	mov $0x604,%edx
	mov $0x2000,%eax
	out %ax,%dx
	hlt
	jmp 1b



>       -cpu host \
>       -m 1024,maxmem=20G,slots=2 \
>       -smp 1 \
>       -no-user-config \
>       -nodefaults \
>       -object
> memory-backend-file,id=mem0,share,mem-path=$IMAGE,size=235929600 \
>       -device nvdimm,memdev=mem0,id=nv0 \
>       -append 'console=ttyS0,115200,8n1 root=/dev/pmem0p1 panic=1 rw
> tsc=reliable no_timer_check noreplace-smp
> init=/usr/lib/systemd/systemd rootfstype=ext4 rcupdate.rcu_expedited=1
> reboot=f' \
>       -no-reboot \
>       -serial mon:stdio \
>       -kernel $KERNEL
>
>
>>    * should this work in TCG as well?
>>    * are they any special linker rules required for the Xen.notes?
>>
>> And finally:
>>
>>    * is this idea of mine a weird abuse of the PVHVM boot protocol or
>>      does it make sense?
>>


--
Alex Bennée

Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Paolo Bonzini]

On 08/03/19 11:43, Alex Bennée wrote:
>>
>> sudo $QEMU \
>>       -name testvm01 \
>>       -machine
>> q35,accel=kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit \
> 
> Hmm if I use the q35 machine type I might have to rethink the approach
> for getting output and results as I'm currently using the isa debugcon
> and debug exit devices.

q35 supports ISA, but it shouldn't be needed.  pc should work.

Paolo

Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Stefano Garzarella]

Hi Alex,
sorry for the big delay, but I was traveling without my PC.

On Wed, Mar 06, 2019 at 05:51:05PM +0000, Alex Bennée wrote:
> 
> Hi,
> 
> I've been looking at using PVH as an alternative to a long bios boot
> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
> hard to piece together all the bits but I naively thought it would just
> be a case of adding a few ELF NOTES to my boot.S with something like:
> 
>           ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
>           ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
>           ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry == virtbase */
>           ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)
> 
>           .code64

Can you try to use .code32?
The pvh.bin optionrom will jump to the image in 32-bit mode.
I don't have a lot of experience, but looking at arch/x86/platform/pvh/head.S
(Linux), I saw that entry point is under .code32, than it will switch to
64-bit mode.

>           .section .text
>           /* Kernel Entry Point */
>   .global _start
>   _start:
>           // Setup stack ASAP
>           movq $stack_end,%rsp
> 
> However I'm running into lots of head scratching as the get_elf_note
> code seems to skip over the notes before failing:
> 
>   ./qemu-system-x86_64 -monitor none -display none \
>     -chardev stdio,id=out -device isa-debugcon,chardev=out \
>     -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello
>   load_elf64: processing hdr:0 of type 1
>   load_elf64: processing hdr:1 of type 4
>   get_elf_note_type64: looking for type 18, first is 3
>   get_elf_note_type64: 4/20
>   get_elf_note_type64: offset is 36
>   get_elf_note_type64: note is 0
>   get_elf_note_type64: 0/123713
>   get_elf_note_type64: offset is 123728
>   load_elf64: processing hdr:2 of type 1685382481
>   qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note
> 
> So I thought I'd go back to the Linux kernel and see if I could get it
> to boot up. So I built an x86_64 kernel with:
> 
>   CONFIG_XEN_PVHVM=y
>   CONFIG_XEN_PVHVM_SMP=y
>   CONFIG_XEN_PVH=y
>   CONFIG_PVH=y

I enabled only CONFIG_PVH to boot a vmlinux image with PVH support.

> 
> And tried to boot that, it certainly gets a lot further but in detecting
> the note 18 it's looking for but then doesn't provide any output. So I
> started digging around the patches and saw talk of a PVH option ROM
> which does all the x86 mode escalation before booting the kernel.
> However I was unable to find any documentation about if I should be
> adding this manually to my command line or if it is auto-magiced into
> place. So I have a number of questions:

Sorry for that, I'll wrote some docs to cover this feature.

> 
>   * what's the canonical command line for booting a Linux PVHVM kernel?

You can use the standard -kernel parameter specifying the path to the
vmlinux image compiled with CONFIG_PVH=y. For example I'm using this
command:
./x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=kvm \
    -kernel /path/to/vmlinux \
    -drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
    -append 'root=/dev/vda console=ttyS0' -vga none -display none \
    -serial mon:stdio

QEMU will detect the PVH image and it will use SeaBIOS with the new pvh.bin
optionrom to boot the image.

>   * should this work in TCG as well?

Yes, I tried the following command and it works:
./x86_64-softmmu/qemu-system-x86_64 -machine pc,accel=tcg \
    -kernel /path/to/vmlinux \
    -drive file=/path/to/rootfs.ext2,if=virtio,format=raw \
    -append 'root=/dev/vda ro console=ttyS0' -vga none -display none \
    -serial mon:stdio

>   * are they any special linker rules required for the Xen.notes?

I don't know, but I'll investigate on it.

> 
> And finally:
> 
>   * is this idea of mine a weird abuse of the PVHVM boot protocol or
>     does it make sense?

IMHO it isn't an abuse and make sense to boot a bare-metal application
directly in 32-bit mode using the PVH loader.

If you want to share with me a part of your code, I'll try to play with
it.

Cheers,
Stefano

Re: [Qemu-devel] Booting kernels with PVHVM documentation?

[Alex Bennée]

Stefano Garzarella <sgarzare@redhat.com> writes:

> Hi Alex,
> sorry for the big delay, but I was traveling without my PC.
>
> On Wed, Mar 06, 2019 at 05:51:05PM +0000, Alex Bennée wrote:
>>
>> Hi,
>>
>> I've been looking at using PVH as an alternative to a long bios boot
>> sequence to boot some x86_64 test kernels for tests/tcg. I'm finding it
>> hard to piece together all the bits but I naively thought it would just
>> be a case of adding a few ELF NOTES to my boot.S with something like:
>>
>>           ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE,      _ASM_PTR 0x100000)
>>           ELFNOTE(Xen, XEN_ELFNOTE_ENTRY,          _ASM_PTR _start)
>>           ELFNOTE(Xen, XEN_ELFNOTE_PHYS32_ENTRY,   _ASM_PTR 0)    /* entry == virtbase */
>>           ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET,   _ASM_PTR 0)
>>
>>           .code64
>
> Can you try to use .code32?
> The pvh.bin optionrom will jump to the image in 32-bit mode.
> I don't have a lot of experience, but looking at arch/x86/platform/pvh/head.S
> (Linux), I saw that entry point is under .code32, than it will switch to
> 64-bit mode.

That doesn't seem to make any difference:

./qemu-system-x86_64 -monitor none -display none -chardev stdio,id=out -device isa-debugcon,chardev=out -device isa-debug-exit,iobase=0xf4,iosize=0x4 -kernel ./tests/hello -vga none
qemu-system-x86_64: Error loading uncompressed kernel without PVH ELF Note


<snip>
>
>>   * are they any special linker rules required for the Xen.notes?
>
> I don't know, but I'll investigate on it.
>
>>
>> And finally:
>>
>>   * is this idea of mine a weird abuse of the PVHVM boot protocol or
>>     does it make sense?
>
> IMHO it isn't an abuse and make sense to boot a bare-metal application
> directly in 32-bit mode using the PVH loader.
>
> If you want to share with me a part of your code, I'll try to play with
> it.

My current hacking branch is:

  https://github.com/stsquad/qemu/tree/testing/next-with-x86-64-tests

--
Alex Bennée