[xen-unstable-smoke test] 155612: regressions - FAIL

[xen-unstable-smoke test] 155612: regressions - FAIL

[osstest service owner]

flight 155612 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/155612/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 test-arm64-arm64-xl-xsm       8 xen-boot                 fail REGR. vs. 155584

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-libvirt     15 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          15 migrate-support-check        fail   never pass
 test-armhf-armhf-xl          16 saverestore-support-check    fail   never pass

version targeted for testing:
 xen                  8a62dee9ceff3056c7e0bd9632bac39bee2a51b3
baseline version:
 xen                  25849c8b16f2a5b7fcd0a823e80a5f1b590291f9

Last test of basis   155584  2020-10-09 02:01:25 Z    0 days
Testing same since   155612  2020-10-09 18:01:22 Z    0 days    1 attempts

------------------------------------------------------------
People who touched revisions under test:
  Andrew Cooper <andrew.cooper3@citrix.com>
  Jan Beulich <jbeulich@suse.com>
  Roger Pau Monné <roger.pau@citrix.com>
  Trammell Hudson <hudson@trmm.net>

jobs:
 build-arm64-xsm                                              pass    
 build-amd64                                                  pass    
 build-armhf                                                  pass    
 build-amd64-libvirt                                          pass    
 test-armhf-armhf-xl                                          pass    
 test-arm64-arm64-xl-xsm                                      fail    
 test-amd64-amd64-xl-qemuu-debianhvm-amd64                    pass    
 test-amd64-amd64-libvirt                                     pass    


------------------------------------------------------------
sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
    http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
    http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
    http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Not pushing.

------------------------------------------------------------
commit 8a62dee9ceff3056c7e0bd9632bac39bee2a51b3
Author: Jan Beulich <jbeulich@suse.com>
Date:   Fri Oct 2 12:30:34 2020 +0200

    x86/vLAPIC: don't leak regs page from vlapic_init() upon error
    
    Fixes: 8a981e0bf25e ("Make map_domain_page_global fail")
    Signed-off-by: Jan Beulich <jbeulich@suse.com>
    Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

commit 8a71d50ed40bfa78c37722dc11995ac2563662c3
Author: Trammell Hudson <hudson@trmm.net>
Date:   Fri Oct 2 07:18:21 2020 -0400

    efi: Enable booting unified hypervisor/kernel/initrd images
    
    This patch adds support for bundling the xen.efi hypervisor, the xen.cfg
    configuration file, the Linux kernel and initrd, as well as the XSM,
    and architectural specific files into a single "unified" EFI executable.
    This allows an administrator to update the components independently
    without requiring rebuilding xen, as well as to replace the components
    in an existing image.
    
    The resulting EFI executable can be invoked directly from the UEFI Boot
    Manager, removing the need to use a separate loader like grub as well
    as removing dependencies on local filesystem access.  And since it is
    a single file, it can be signed and validated by UEFI Secure Boot without
    requring the shim protocol.
    
    It is inspired by systemd-boot's unified kernel technique and borrows the
    function to locate PE sections from systemd's LGPL'ed code.  During EFI
    boot, Xen looks at its own loaded image to locate the PE sections for
    the Xen configuration (`.config`), dom0 kernel (`.kernel`), dom0 initrd
    (`.ramdisk`), and XSM config (`.xsm`), which are included after building
    xen.efi using objcopy to add named sections for each input file.
    
    For x86, the CPU ucode can be included in a section named `.ucode`,
    which is loaded in the efi_arch_cfg_file_late() stage of the boot process.
    
    On ARM systems the Device Tree can be included in a section named
    `.dtb`, which is loaded during the efi_arch_cfg_file_early() stage of
    the boot process.
    
    Note that the system will fall back to loading files from disk if
    the named sections do not exist. This allows distributions to continue
    with the status quo if they want a signed kernel + config, while still
    allowing a user provided initrd (which is how the shim protocol currently
    works as well).
    
    This patch also adds constness to the section parameter of
    efi_arch_cfg_file_early() and efi_arch_cfg_file_late(),
    changes pe_find_section() to use a const CHAR16 section name,
    and adds pe_name_compare() to match section names.
    
    Signed-off-by: Trammell Hudson <hudson@trmm.net>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>
    [Fix ARM build by including pe.init.o]
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

commit 4dced5df761e36fa2561f6f0f6563b3580d95e7f
Author: Trammell Hudson <hudson@trmm.net>
Date:   Fri Oct 2 07:18:20 2020 -0400

    efi/boot.c: add handle_file_info()
    
    Add a separate function to display the address ranges used by
    the files and call `efi_arch_handle_module()` on the modules.
    
    Signed-off-by: Trammell Hudson <hudson@trmm.net>
    Acked-by: Jan Beulich <jbeulich@suse.com>

commit 04be2c3a067899a3860fc2c7bc7a1599502ed1c5
Author: Trammell Hudson <hudson@trmm.net>
Date:   Fri Oct 2 07:18:19 2020 -0400

    efi/boot.c: add file.need_to_free
    
    The config file, kernel, initrd, etc should only be freed if they
    are allocated with the UEFI allocator.  On x86 the ucode, and on
    ARM the dtb, are also marked as need_to_free when allocated or
    expanded.
    
    This also fixes a memory leak in ARM fdt_increase_size() if there
    is an error in building the new device tree.
    
    Signed-off-by: Trammell Hudson <hudson@trmm.net>
    Reviewed-by: Jan Beulich <jbeulich@suse.com>

commit afef39241b66df7d5fd66b07dc13350370a4991a
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date:   Wed Apr 1 15:51:08 2020 +0100

    x86/ucode: Trivial further cleanup
    
     * Drop unused include in private.h.
     * Used explicit width integers for Intel header fields.
     * Adjust comment to better describe the extended header.
     * Drop unnecessary __packed attribute for AMD header.
     * Fix types and style.
    
    No functional change.
    
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
    Acked-by: Roger Pau Monné <roger.pau@citrix.com>

commit 8d255609930bed04c6436974bd895be9a405d0c1
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date:   Fri Oct 2 12:20:44 2020 +0100

    x86/hvm: Correct error message in check_segment()
    
    The error message is wrong (given AMD's older interpretation of what a NUL
    segment should contain, attribute wise), and actively unhelpful because you
    only get it in response to a hypercall where the one piece of information you
    cannot provide is the segment selector.
    
    Fix the message to talk about segment attributes, rather than the selector.
    
    Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
    Acked-by: Jan Beulich <jbeulich@suse.com>
(qemu changes not included)

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Andrew Cooper]

On 09/10/2020 21:20, osstest service owner wrote:
> flight 155612 xen-unstable-smoke real [real]
> http://logs.test-lab.xenproject.org/osstest/logs/155612/
>
> Regressions :-(
>
> Tests which did not succeed and are blocking,
> including tests which could not be run:
>  test-arm64-arm64-xl-xsm       8 xen-boot                 fail REGR. vs. 155584

From
http://logs.test-lab.xenproject.org/osstest/logs/155612/test-arm64-arm64-xl-xsm/serial-laxton0.log

Oct  9 18:45:20.790611
Xen 4.15-unstable (c/s Fri Oct 2 12:30:34 2020 +0200 git:8a62dee9ce) EFI
loader
Oct  9 18:45:20.934506 Using configuration file 'xen.cfg'
Oct  9 18:45:20.934558 vmlinuz: 0x00000083fb1e2000-0x00000083fc8b6a00
Oct  9 18:45:20.946436 initrd.gz: 0x00000083f94ef000-0x00000083fb1e1c5a
Oct  9 18:45:21.618435 xenpolicy: 0x00000083f94ec000-0x00000083f94eea0a
Oct  9 18:45:21.774473 Oct  9 18:51:08.218564 <client 0x1b7f430
connected - now 1 clients>
Oct  9 18:51:08.976537 <client 0x1b7f430 disconnected - now 0 clients>

Looks like arm64 is crashing fairly early on boot.

This is probably caused by "efi: Enable booting unified
hypervisor/kernel/initrd images".

~Andrew

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Trammell Hudson]

On Friday, October 9, 2020 10:27 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> [...]
> Looks like arm64 is crashing fairly early on boot.
>
> This is probably caused by "efi: Enable booting unified
> hypervisor/kernel/initrd images".

Darn it.  I'm working out how to build and boot qemu aarch64 so
that I can figure out what is going on.

Also, I'm not sure that it is possible to build a unified arm
image right now; objcopy (and all of the obj* tools) say
"File format not recognized" on the xen.efi file.  The MZ file
is not what they are expecting for ARM executables.

--
Trammell

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Trammell Hudson]

On Saturday, October 10, 2020 1:42 PM, Trammell Hudson <hudson@trmm.net> wrote:
> On Friday, October 9, 2020 10:27 PM, Andrew Cooper andrew.cooper3@citrix.com wrote:
> > [...]
> > Looks like arm64 is crashing fairly early on boot.
> > This is probably caused by "efi: Enable booting unified
> > hypervisor/kernel/initrd images".
>
> Darn it. I'm working out how to build and boot qemu aarch64 so
> that I can figure out what is going on.

Unfortunately qemu 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.32)
doesn't replicate this crash using the command line from
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64


qemu-system-aarch64 \
        -cpu cortex-a57 \
        -smp 4 -m 4096 \
        -machine virt,gic_version=3 \
        -machine virtualization=true \
        -machine type=virt \
        -display none \
        -serial mon:stdio \
        -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd \
        -drive if=none,file=xenial-server-cloudimg-arm64-uefi1.img,id=hd0 \
        -device virtio-blk-device,drive=hd0 \
        -boot order=d
[...]
Xen 4.15-unstable (c/s Fri Oct 2 12:30:34 2020 +0200 git:8a62dee9ce) EFI loader
Using configuration file 'BOOTAA64.cfg'
virt-gicv3.dtb: 0x000000013bebe000-0x000000013bece000
kernel: 0x000000013aecd000-0x000000013bd12200
initrd: 0x0000000138ced000-0x000000013aecc3bb
 __  __            _  _    _ ____                     _        _     _
 \ \/ /___ _ __   | || |  / | ___|    _   _ _ __  ___| |_ __ _| |__ | | ___
  \  // _ \ '_ \  | || |_ | |___ \ __| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | | |__   _|| |___) |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_|    |_|(_)_|____/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

(XEN) Xen version 4.15-unstable (hudson@) (aarch64-linux-gnu-gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0) debug=n  Sat Oct 10 08:42:30 CEST 2020
(XEN) Latest ChangeSet: Fri Oct 2 12:30:34 2020 +0200 git:8a62dee9ce
(XEN) Processor: 411fd070: "ARM Limited", variant: 0x1, part 0xd07, rev 0x0

[...]


It makes it all the way into the Linux kernel and user space
without crashing.  Hopefully someone with better access to ARM
hardware can help debug!

--
Trammell

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Julien Grall]

Hi,

On 10/10/2020 12:42, Trammell Hudson wrote:
> On Friday, October 9, 2020 10:27 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> [...]
>> Looks like arm64 is crashing fairly early on boot.
>>
>> This is probably caused by "efi: Enable booting unified
>> hypervisor/kernel/initrd images".
> 
> Darn it.  I'm working out how to build and boot qemu aarch64 so
> that I can figure out what is going on.

FWIW, in OSSTest, we are chainloading Xen from GRUB. I have tried to 
chainloading on QEMU but couldn't get to work so far (even without your 
series).

Although, I have no trouble to boot using the GRUB way (i.e. via multiboot).

> 
> Also, I'm not sure that it is possible to build a unified arm
> image right now; objcopy (and all of the obj* tools) say
> "File format not recognized" on the xen.efi file.  The MZ file
> is not what they are expecting for ARM executables.

IIUC, you are trying to add section into the EFI binary and not the ELF. 
Is it correct?

I don't know what x86 is doing but for Arm, xen.efi (and Linux Image) is 
custom built. So it may lack information to be recognized by objdump.

My knowledge of objdump is fairly limited. If you are interested to fix 
it, then I would suggest to ask the binutils community what they expect.

We could then adapt so objdump can recognize it.

Cheers,

-- 
Julien Grall

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Julien Grall]

Hi,

On 10/10/2020 13:20, Trammell Hudson wrote:
> On Saturday, October 10, 2020 1:42 PM, Trammell Hudson <hudson@trmm.net> wrote:
>> On Friday, October 9, 2020 10:27 PM, Andrew Cooper andrew.cooper3@citrix.com wrote:
>>> [...]
>>> Looks like arm64 is crashing fairly early on boot.
>>> This is probably caused by "efi: Enable booting unified
>>> hypervisor/kernel/initrd images".
>>
>> Darn it. I'm working out how to build and boot qemu aarch64 so
>> that I can figure out what is going on.
> 
> Unfortunately qemu 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.32)
> doesn't replicate this crash using the command line from
> https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/qemu-system-aarch64

Are you chainloading Xen?

Cheers,

-- 
Julien Grall

Re: [xen-unstable-smoke test] 155612: regressions - FAIL

[Stefano Stabellini]

On Sat, 10 Oct 2020, Julien Grall wrote:
> Hi,
> 
> On 10/10/2020 12:42, Trammell Hudson wrote:
> > On Friday, October 9, 2020 10:27 PM, Andrew Cooper
> > <andrew.cooper3@citrix.com> wrote:
> > > [...]
> > > Looks like arm64 is crashing fairly early on boot.
> > > 
> > > This is probably caused by "efi: Enable booting unified
> > > hypervisor/kernel/initrd images".
> > 
> > Darn it.  I'm working out how to build and boot qemu aarch64 so
> > that I can figure out what is going on.
> 
> FWIW, in OSSTest, we are chainloading Xen from GRUB. I have tried to
> chainloading on QEMU but couldn't get to work so far (even without your
> series).
> 
> Although, I have no trouble to boot using the GRUB way (i.e. via multiboot).

It took me a while to set it up, but now I have a test environment based
on RPi4 where I can chainload Xen from Grub EFI and boot successfully up
until the rootfs (I don't have a rootfs setup correctly yet, so it
breaks with the usual "Cannot open root device".) Which means I can get
both Xen and the Dom0 kernel to boot.

I hope it will be useful in the future, but in this case it didn't help
because I get the same behavior with and without Trammell's patches. For
me the chainload boot doesn't break.

Could it be down to the Grub version in-use? I am using UBoot EFI to
load Grub, but I doubt that could be a meaningful difference.


> > Also, I'm not sure that it is possible to build a unified arm
> > image right now; objcopy (and all of the obj* tools) say
> > "File format not recognized" on the xen.efi file.  The MZ file
> > is not what they are expecting for ARM executables.
> 
> IIUC, you are trying to add section into the EFI binary and not the ELF. Is it
> correct?
> 
> I don't know what x86 is doing but for Arm, xen.efi (and Linux Image) is
> custom built.

Specifically, see:

xen/arch/arm/arm64/head.S:efi_head


> So it may lack information to be recognized by objdump.
> 
> My knowledge of objdump is fairly limited. If you are interested to fix it,
> then I would suggest to ask the binutils community what they expect.
> 
> We could then adapt so objdump can recognize it.