Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Itaru Kitayama]

Hi,

When I see a Realm VM creation fails with:

Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none

do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?

Thanks,
Itaru.

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Peter Maydell]

On Thu, 30 May 2024 at 14:26, Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>
> Hi,
>
> When I see a Realm VM creation fails with:
>
> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:m
> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument

The file target/arm/kvm-rme.c doesn't exist in upstream QEMU,
so you must be using a fork or not-yet-committed patchseries.
I would suggest starting by asking the person who wrote those
patches...

thanks
-- PMM

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Philippe Mathieu-Daudé]

Cc'ing more developers

On 30/5/24 06:30, Itaru Kitayama wrote:
> Hi,
> 
> When I see a Realm VM creation fails with:
> 
> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
> 
> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
> 
> Thanks,
> Itaru.

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Itaru Kitayama]

> On May 30, 2024, at 22:30, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> 
> Cc'ing more developers
> 
> On 30/5/24 06:30, Itaru Kitayama wrote:
>> Hi,
>> When I see a Realm VM creation fails with:
>> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
>> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
>> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
>> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
>> Thanks,
>> Itaru.
> 

I’m testing Jean’s repo at:

https://git.codelinaro.org/linaro/dcap/qemu/-/tree/cca/v2?ref_type=heads

Itaru.

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

On 5/31/24 14:19, Itaru Kitayama wrote:
>> On May 30, 2024, at 22:30, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>> Cc'ing more developers
>>
>> On 30/5/24 06:30, Itaru Kitayama wrote:
>>> Hi,
>>> When I see a Realm VM creation fails with:
>>> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
>>> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
>>> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
>>> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
>>> Thanks,
>>> Itaru.
>>
> 
> I’m testing Jean’s repo at:
> 
> https://git.codelinaro.org/linaro/dcap/qemu/-/tree/cca/v2?ref_type=heads
> 

I got a chance to try CCA software components, suggested by [1]. However, the edk2
is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm replying
to see if anyone has a idea.

[1] https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU


---> tf-rmm

# git clone https://git.codelinaro.org/linaro/dcap/rmm.git tf-rmm
# cd tf-rmm
# git checkout origin/cca/v2 -b cca/v2
# git submodule update --init --recursive
# export CROSS_COMPILE=
# cmake -DCMAKE_BUILD_TYPE=Debug -DRMM_CONFIG=qemu_virt_defcfg -B build-qemu
# cmake --build build-qemu

---> edk2

# git clone git@github.com:tianocore/edk2.git
# cd edk2
# git submodule update --init --recursive
# source edksetup.sh
# make -j -C BaseTools
# export GCC5_AARCH64_PREFIX=
# build -b RELEASE -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

---> tf-a

# git clone https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git tf-a
# cd tf-a
# git checkout origin/cca/v2 -b cca/v2
# make -j CROSS_COMPILE= PLAT=qemu ENABLE_RME=1 DEBUG=1 LOG_LEVEL=40 \
   QEMU_USE_GIC_DRIVER=QEMU_GICV3 RMM=../rmm/build-qemu/Debug/rmm.img \
   BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/RELEASE_GCC5/FV/QEMU_EFI.fd all fip
# dd if=build/qemu/debug/bl1.bin of=flash.bin
# dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096

---> QEMU

# git clone https://git.qemu.org/git/qemu.git qemu.main
# cd qemu.main
# ./configure --target-list=aarch64-softmmu
# make -j 60

---> boot the host on the emulated hardware

/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                            \
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off                         \
-cpu max,x-rme=on -m 8G -smp 8                                                     \
-nodefaults -monitor none -serial mon:stdio -nographic                             \
-bios /home/gavin/sandbox/CCA/tf-a/flash.bin                                       \
-kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image                        \
-drive format=raw,if=none,file=${CCA}/buildroot/output/images/rootfs.ext4,id=hd0   \
-device virtio-blk-pci,drive=hd0                                                   \
-append "root=/dev/vda console=ttyAMA0"
   :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL1: Built : 00:31:35, May 31 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0xe06b000
INFO:    Image id=1 loaded: 0xe06b000 - 0xe0742d1
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0xe06b000
INFO:    SPSR = 0x3cd
INFO:    [GPT] Boot Configuration
INFO:      PPS/T:     0x2/40
INFO:      PGS/P:     0x0/12
INFO:      L0GPTSZ/S: 0x0/30
INFO:      PAS count: 0x6
INFO:      L0 base:   0xedfe000
INFO:    [GPT] PAS[0]: base 0xe001000, size 0xff000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[1]: base 0xe100000, size 0xcfe000, GPI 0x8, type 0x1
INFO:    [GPT] PAS[2]: base 0xedfe000, size 0x202000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[3]: base 0x40000000, size 0x100000, GPI 0x9, type 0x1
INFO:    [GPT] PAS[4]: base 0x40100000, size 0x2800000, GPI 0xb, type 0x1
INFO:    [GPT] PAS[5]: base 0x42900000, size 0x1fd700000, GPI 0x9, type 0x1
INFO:    Enabling Granule Protection Checks
NOTICE:  BL2: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL2: Built : 00:31:35, May 31 2024
INFO:    BL2: Doing platform setup
INFO:    Reserved RMM memory [0x40100000, 0x428fffff] in Device tree
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0xe0a0000
INFO:    Image id=3 loaded: 0xe0a0000 - 0xe0b10c4
INFO:    BL2: Loading image id 35
INFO:    Loading image id=35 at address 0x40100000
INFO:    Image id=35 loaded: 0x40100000 - 0x403033b0
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x60000000
INFO:    Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0xe0a0000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL31: Built : 00:31:35, May 31 2024
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 287
INFO:    BL31: Initializing runtime services
INFO:    RMM setup done.
INFO:    BL31: Initializing RMM
INFO:    RMM init start.
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: May 31 2024 00:21:59
INFO:    RMM init end.
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x60000000
INFO:    SPSR = 0x3c9
UEFI firmware (version  built at 01:31:23 on May 31 2024)

The boot is stuck and no more output after that. I tried adding more verbose output
from edk2 and found it's stuck at the following point.


ArmVirtPkg/PrePi/PrePi.c::PrePiMain
rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim

  #ifdef MDE_CPU_AARCH64
   //
   // Set the SMCCC conduit to SMC if executing at EL2, which is typically the
   // exception level that services HVCs rather than the one that invokes them.
   //
   if (ArmReadCurrentEL () == AARCH64_EL2) {
     Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE);       // The function is never returned in my case
     ASSERT_EFI_ERROR (Status);
   }
  #endif

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Peter Maydell]

On Fri, 31 May 2024 at 05:20, Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>
>
>
> > On May 30, 2024, at 22:30, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> >
> > Cc'ing more developers
> >
> > On 30/5/24 06:30, Itaru Kitayama wrote:
> >> Hi,
> >> When I see a Realm VM creation fails with:
> >> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
> >> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
> >> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
> >> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
> >> Thanks,
> >> Itaru.
> >
>
> I’m testing Jean’s repo at:
>
> https://git.codelinaro.org/linaro/dcap/qemu/-/tree/cca/v2?ref_type=heads

OK, we should cc Jean-Philippe then.

I'm wondering if this is as simple as "RME via KVM doesn't support SVE yet",
perhaps.

thanks
-- PMM

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

Hi Itaru,

On Fri, May 31, 2024 at 10:57:13AM +0100, Peter Maydell wrote:
> On Fri, 31 May 2024 at 05:20, Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
> >
> >
> >
> > > On May 30, 2024, at 22:30, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
> > >
> > > Cc'ing more developers
> > >
> > > On 30/5/24 06:30, Itaru Kitayama wrote:
> > >> Hi,
> > >> When I see a Realm VM creation fails with:
> > >> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
> > >> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
> > >> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
> > >> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
> > >> Thanks,
> > >> Itaru.
> > >
> >
> > I’m testing Jean’s repo at:
> >
> > https://git.codelinaro.org/linaro/dcap/qemu/-/tree/cca/v2?ref_type=heads

Thanks again for testing, you can report issues by replying directly to
my posting, so I can get to them quicker. If you want I can Cc you on the
next one. The latest is:

[PATCH v2 00/22] arm: Run CCA VMs with KVM
https://lore.kernel.org/qemu-devel/20240419155709.318866-2-jean-philippe@linaro.org/

That does sound like the KVM host doesn't support SVE, but the QEMU VMM
version is also too old: in the latest series 'sve-vector-length' was
removed and we use the existing -cpu parameters to configure SVE. Please
make sure that the QEMU branch is cca/v2 to match the Linux KVM branch,
because the older QEMU patches doesn't work with the newest KVM patches.
You'll need to update the command-line as well, because paramaters have
changed for cca/v2.

This may be the case of older build directories that aren't properly
synchronized. They can be removed manually but the quicker way is usually
to remove all source and build directories and start anew.

Thanks,
Jean


> 
> OK, we should cc Jean-Philippe then.
> 
> I'm wondering if this is as simple as "RME via KVM doesn't support SVE yet",
> perhaps.
> 
> thanks
> -- PMM

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Itaru Kitayama]

Hi Jean,

> On May 31, 2024, at 19:21, Jean-Philippe Brucker <jean-philippe@linaro.org> wrote:
> 
> Hi Itaru,
> 
> On Fri, May 31, 2024 at 10:57:13AM +0100, Peter Maydell wrote:
>> On Fri, 31 May 2024 at 05:20, Itaru Kitayama <itaru.kitayama@linux.dev> wrote:
>>> 
>>> 
>>> 
>>>> On May 30, 2024, at 22:30, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>>> 
>>>> Cc'ing more developers
>>>> 
>>>> On 30/5/24 06:30, Itaru Kitayama wrote:
>>>>> Hi,
>>>>> When I see a Realm VM creation fails with:
>>>>> Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159:
>>>>> qemu-system-aarch64: RME: failed to configure SVE: Invalid argument
>>>>> test.sh: line 8:  2502 Aborted                 qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' -M 'confidential-guest-support=rme0' -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' -kernel Image -initrd rootfs.cpio -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none
>>>>> do I need to suspect first the VMM, QEMU, or the Image? The kernel is built with LLVM, does it matter?
>>>>> Thanks,
>>>>> Itaru.
>>>> 
>>> 
>>> I’m testing Jean’s repo at:
>>> 
>>> https://git.codelinaro.org/linaro/dcap/qemu/-/tree/cca/v2?ref_type=heads
> 
> Thanks again for testing, you can report issues by replying directly to
> my posting, so I can get to them quicker. If you want I can Cc you on the
> next one. The latest is:
> 
> [PATCH v2 00/22] arm: Run CCA VMs with KVM
> https://lore.kernel.org/qemu-devel/20240419155709.318866-2-jean-philippe@linaro.org/

Thanks! I wasn’t aware of it The good news is that after whole day of try and error attempts I was able to
bring up a Realm VM on FVP. Here’s my version of overlay yaml, cca-v2.yaml:

build:
  linux:
    repo:
      revision: cca-full/v2

#  kvmtool:
#    repo:
#      revision: cca/v2

  rmm:
    repo:
      revision: main



  tfa:
    repo:
      revision: master

  kvm-unit-tests:
    repo:
      revision: cca/v2

… and the QEMU options are below:

qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' \
-cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' \
-M 'confidential-guest-support=rme0' \
-object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' \
-kernel Image -initrd rootfs.cpio \
-append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none

Thanks,
Itaru.

> 
> That does sound like the KVM host doesn't support SVE, but the QEMU VMM
> version is also too old: in the latest series 'sve-vector-length' was
> removed and we use the existing -cpu parameters to configure SVE. Please
> make sure that the QEMU branch is cca/v2 to match the Linux KVM branch,
> because the older QEMU patches doesn't work with the newest KVM patches.
> You'll need to update the command-line as well, because paramaters have
> changed for cca/v2.
> 
> This may be the case of older build directories that aren't properly
> synchronized. They can be removed manually but the quicker way is usually
> to remove all source and build directories and start anew.
> 
> Thanks,
> Jean
> 
> 
>> 
>> OK, we should cc Jean-Philippe then.
>> 
>> I'm wondering if this is as simple as "RME via KVM doesn't support SVE yet",
>> perhaps.
>> 
>> thanks
>> -- PMM

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

Hi Gavin,

On Fri, May 31, 2024 at 04:23:13PM +1000, Gavin Shan wrote:
> I got a chance to try CCA software components, suggested by [1]. However, the edk2
> is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm replying
> to see if anyone has a idea.
...
> INFO:    BL31: Preparing for EL3 exit to normal world
> INFO:    Entry point address = 0x60000000
> INFO:    SPSR = 0x3c9
> UEFI firmware (version  built at 01:31:23 on May 31 2024)
> 
> The boot is stuck and no more output after that. I tried adding more verbose output
> from edk2 and found it's stuck at the following point.
> 
> 
> ArmVirtPkg/PrePi/PrePi.c::PrePiMain
> rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim
> 
>  #ifdef MDE_CPU_AARCH64
>   //
>   // Set the SMCCC conduit to SMC if executing at EL2, which is typically the
>   // exception level that services HVCs rather than the one that invokes them.
>   //
>   if (ArmReadCurrentEL () == AARCH64_EL2) {
>     Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE);       // The function is never returned in my case
>     ASSERT_EFI_ERROR (Status);
>   }
>  #endif

I'm able to reproduce this even without RME. This code was introduced
recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.

A slightly reduced reproducer:

$ cd edk2/
$ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
$ cd ..

$ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
$ cd tf-a/
$ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
  dd if=build/qemu/debug/bl1.bin of=flash.bin && \
  dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
$ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin

Thanks,
Jean

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Ard Biesheuvel]

On Fri, 31 May 2024 at 17:09, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> Hi Gavin,
>
> On Fri, May 31, 2024 at 04:23:13PM +1000, Gavin Shan wrote:
> > I got a chance to try CCA software components, suggested by [1]. However, the edk2
> > is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm replying
> > to see if anyone has a idea.
> ...
> > INFO:    BL31: Preparing for EL3 exit to normal world
> > INFO:    Entry point address = 0x60000000
> > INFO:    SPSR = 0x3c9
> > UEFI firmware (version  built at 01:31:23 on May 31 2024)
> >
> > The boot is stuck and no more output after that. I tried adding more verbose output
> > from edk2 and found it's stuck at the following point.
> >
> >
> > ArmVirtPkg/PrePi/PrePi.c::PrePiMain
> > rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim
> >
> >  #ifdef MDE_CPU_AARCH64
> >   //
> >   // Set the SMCCC conduit to SMC if executing at EL2, which is typically the
> >   // exception level that services HVCs rather than the one that invokes them.
> >   //
> >   if (ArmReadCurrentEL () == AARCH64_EL2) {
> >     Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE);       // The function is never returned in my case
> >     ASSERT_EFI_ERROR (Status);
> >   }
> >  #endif
>
> I'm able to reproduce this even without RME. This code was introduced
> recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
> conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
>
> A slightly reduced reproducer:
>
> $ cd edk2/
> $ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
> $ cd ..
>
> $ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
> $ cd tf-a/
> $ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
>   dd if=build/qemu/debug/bl1.bin of=flash.bin && \
>   dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
> $ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
>

Hmm, this is not something I anticipated.

The problem here is that ArmVirtQemuKernel does not actually support
dynamic PCDs, so instead, the PCD here is 'patchable', which means
that the underlying value is just overwritten in the binary image, and
does not propagate to the rest of the firmware. I assume the write
ends up targettng a location that does not tolerate this.

Running ArmVirtQemu or ArmVirtQemuKernel at EL2 has really only ever
worked by accident, it was simply never intended for that. The fix in
question was a last minute tweak to prevent some CVE fixes pushed by
MicroSoft from breaking network boot entirely, and now that the
release has been made, I guess we should revisit this and fix it
properly.

So the underlying issue here is that on these platforms, we need to
decide at runtime whether to use HVC or SMC instructions for SMCCC
calls. This code attempts to record this into a dynamic PCD once at
boot, in a way that permits other users of the same library to simply
hardcode this in the platform definition (given that bare metal
platforms never need this flexibility).

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

On Fri, May 31, 2024 at 11:16:30PM +0900, Itaru Kitayama wrote:
> Thanks! I wasn’t aware of it The good news is that after whole day of try and error attempts I was able to
> bring up a Realm VM on FVP. Here’s my version of overlay yaml, cca-v2.yaml:

That is good news, thanks for the update

> build:
>   linux:
>     repo:
>       revision: cca-full/v2
> 
> #  kvmtool:
> #    repo:
> #      revision: cca/v2
> 
>   rmm:
>     repo:
>       revision: main
> 
> 
> 
>   tfa:
>     repo:
>       revision: master
> 
>   kvm-unit-tests:
>     repo:
>       revision: cca/v2
> 
> … and the QEMU options are below:
> 
> qemu-system-aarch64 -M 'virt,acpi=off,gic-version=3' \
> -cpu host -enable-kvm -smp 2 -m 512M -overcommit 'mem-lock=on' \
> -M 'confidential-guest-support=rme0' \
> -object 'rme-guest,id=rme0,measurement-algo=sha512,num-pmu-counters=6,sve-vector-length=256' \
> -kernel Image -initrd rootfs.cpio \
> -append 'earycon console=ttyAMA0 rdinit=/sbin/init' -nographic -net none

Note that the cca/v2 branch of QEMU would reject 'num-pmu-counters' and
'sve-vector-length' arguments, so if this works it means an older version
of the QEMU patches is being used (which also means an older Linux branch
is being used). It's possible that shrinkwrap is caching all the build
files, so I'd remove the ~/.shrinkwrap/ files to start fresh

Thanks,
Jean

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

Hi Jean and Ard,

On 6/1/24 01:09, Jean-Philippe Brucker wrote:
> On Fri, May 31, 2024 at 04:23:13PM +1000, Gavin Shan wrote:
>> I got a chance to try CCA software components, suggested by [1]. However, the edk2
>> is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm replying
>> to see if anyone has a idea.
> ...
>> INFO:    BL31: Preparing for EL3 exit to normal world
>> INFO:    Entry point address = 0x60000000
>> INFO:    SPSR = 0x3c9
>> UEFI firmware (version  built at 01:31:23 on May 31 2024)
>>
>> The boot is stuck and no more output after that. I tried adding more verbose output
>> from edk2 and found it's stuck at the following point.
>>
>>
>> ArmVirtPkg/PrePi/PrePi.c::PrePiMain
>> rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim
>>
>>   #ifdef MDE_CPU_AARCH64
>>    //
>>    // Set the SMCCC conduit to SMC if executing at EL2, which is typically the
>>    // exception level that services HVCs rather than the one that invokes them.
>>    //
>>    if (ArmReadCurrentEL () == AARCH64_EL2) {
>>      Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE);       // The function is never returned in my case
>>      ASSERT_EFI_ERROR (Status);
>>    }
>>   #endif
> 
> I'm able to reproduce this even without RME. This code was introduced
> recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
> conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
> 
> A slightly reduced reproducer:
> 
> $ cd edk2/
> $ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
> $ cd ..
> 
> $ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
> $ cd tf-a/
> $ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
>    dd if=build/qemu/debug/bl1.bin of=flash.bin && \
>    dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
> $ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
> 

Thanks for the hints. Eventually, I'm able to start the host with 'edk2-stable202402'.
Note that 'edk2-stable202405' doesn't work. However, I failed to build the edk2 for
guest and unable to start the guest successfully, more information is provided below.

--> host's edk2

# git clone git@github.com:gwshan/edk2.git edk2
# cd edk2; git checkout edk2-stable202402 -b stable202402
# git submodule update --init --recursive;      \
   source edksetup.sh; make -j -C BaseTools;     \
   export GCC5_AARCH64_PREFIX=;                  \
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

---> tf-a: rebuild using commands as you suggested.

---> Boot host

/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                                           \
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off                                        \
-cpu max,x-rme=on -m 8G -smp 8                                                                    \
-nodefaults -monitor none -serial mon:stdio -nographic                                            \
-bios /home/gavin/sandbox/CCA/tf-a/flash.bin                                                      \
-kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image                                       \
-drive format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0 \
-device virtio-blk-pci,drive=hd0                                                                  \
-netdev tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown                 \
-device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0                                          \
-append root=/dev/vda console=ttyAMA0                                                             \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0                                                \
-fsdev local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0
   :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL1: Built : 00:31:35, May 31 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
INFO:    BL1: Loading BL2
   :
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: May 31 2024 00:21:59
INFO:    RMM init end.
   :
UEFI firmware (version  built at 04:07:42 on Jun  1 2024)
PlatformPeim: PL011 UART (console) @ 0x9000000
PlatformPeim: PL011 UART (debug) @ 0x9000000
   :
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
   :
Welcome to Buildroot
buildroot login:
# ifconfig eth0 | grep 'inet addr'
           inet addr:10.26.1.212  Bcast:10.26.1.255  Mask:255.255.255.0

---> guest edk2

# git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
# cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
# git submodule update --init --recursive;  \
   source edksetup.sh; make -j -C BaseTools; \
   export GCC5_AARCH64_PREFIX=;              \
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
"gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
"GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
GenFw: ERROR 3000: Invalid
   :
build.py...
  : error 7000: Failed to execute command
	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]


build.py...
  : error F002: Failed to build module
	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]

- Failed -
Build end time: 05:42:19, Jun.01 2024
Build total time: 00:00:31

---> Use the edk2 image from the latest QEMU source

# cd /home/gavin/sandbox/CCA
# cp /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd ./

---> Start the guest and no output from the console

host# mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# export SHR_DIR="/mnt/shr0"
host# qemu-system-aarch64 -accel kvm                              \
       -machine virt,gic-version=3,confidential-guest-support=rme0 \
       -cpu host -smp 2 -m 512M                                    \
       -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
       -monitor none -serial mon:stdio -nographic                  \
       -bios /mnt/edk2-aarch64-code.fd                             \
       -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
       -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
       -append 'console=ttyAMA0'
         :
       <no output from the console>
         :
       (QEMU) q

There are some messages from host's console indicating RMI/RMM servicing
states when the guest is running at background. After the guest is terminated,
the host crashes.

SMC_RMM_RTT_CREATE            102dff000 122c2e000 1e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 1234a7000 2000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 1235bd000 2200000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 12387c000 2400000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123a5a000 2600000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 12407d000 2800000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124109000 2a00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123e49000 2c00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124275000 2e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123138000 3000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124d07000 3200000 3 > RMI_SUCCESS
  :
  :
[22768.994481] rcu: INFO: rcu_preempt self-detected stall on CPU
[22769.006861] rcu: 	3-....: (2751 ticks this GP) idle=93ec/1/0x4000000000000000 softirq=114451/115721 fqs=1160
[22769.020475] rcu: 	(t=5257 jiffies g=531913 q=7 ncpus=8)
[22769.030547] CPU: 3 PID: 198 Comm: qemu-system-aar Not tainted 6.9.0-rc1-gavin-gfcfc92d6ff07 #1
[22769.041847] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[22769.050548] pstate: 60402009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[22769.059382] pc : kvm_realm_unmap_range+0x308/0x32c
[22769.070275] lr : kvm_realm_unmap_range+0x304/0x32c
[22769.075893] sp : ffff800080a3b930
[22769.079929] x29: ffff800080a3b930 x28: 00000000003d7000 x27: 00000000003d6000
[22769.092990] x26: 00000000c4000152 x25: ffffffffffffffff x24: 0000000000000000
[22769.101150] x23: 0000010000000000 x22: 00000000c4000155 x21: 0000000102dff000
[22769.109056] x20: ffff8000801a5e00 x19: 0000000000000000 x18: 0000000000000001
[22769.117042] x17: 0000000000000000 x16: 000000000000000e x15: 0000000000000000
[22769.124991] x14: 0000ffff7fa14000 x13: 0000000000000002 x12: 000000000010d594
[22769.134213] x11: 0000000000000002 x10: 00000000ffffffff x9 : ffffffffffffffff
[22769.142951] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 00000000c2dff000
[22769.151413] x5 : 0000000102f56000 x4 : 0000000000000015 x3 : 0000000000000000
[22769.159932] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[22769.169674] Call trace:
[22769.174026]  kvm_realm_unmap_range+0x308/0x32c
[22769.181046]  __unmap_stage2_range+0x60/0x7c
[22769.186396]  kvm_free_stage2_pgd+0xa0/0xd4
[22769.191766]  kvm_arch_flush_shadow_all+0x1c/0x34
[22769.197879]  kvm_mmu_notifier_release+0x30/0x84
[22769.203304]  __mmu_notifier_release+0x7c/0x1f8
[22769.209340]  exit_mmap+0x264/0x274
[22769.213992]  __mmput+0x40/0x150
[22769.218635]  mmput+0x50/0x5c
[22769.222606]  do_exit+0x288/0x92c
[22769.226935]  do_group_exit+0x34/0x90
[22769.231359]  get_signal+0x814/0x820
[22769.236537]  do_signal+0x90/0x1320
[22769.241145]  do_notify_resume+0xc8/0x140
[22769.246458]  el0_svc+0xc8/0xdc
[22769.250913]  el0t_64_sync_handler+0x13c/0x158
[22769.256045]  el0t_64_sync+0x190/0x194

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

Hi Gavin,

On Sat, Jun 01, 2024 at 08:14:46PM +1000, Gavin Shan wrote:
> ---> guest edk2
> 
> # git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
> # cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
> # git submodule update --init --recursive;  \
>   source edksetup.sh; make -j -C BaseTools; \
>   export GCC5_AARCH64_PREFIX=;              \

Doesn't this needs a cross-compiler, something like "aarch64-linux-gnu-" ?

>   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>    :
>   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
> "gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
> "GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
> GenFw: ERROR 3000: Invalid
>   :
> build.py...
>  : error 7000: Failed to execute command
> 	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
> 
> 
> build.py...
>  : error F002: Failed to build module
> 	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]
> 
> - Failed -
> Build end time: 05:42:19, Jun.01 2024
> Build total time: 00:00:31
> 
> ---> Use the edk2 image from the latest QEMU source

Unfortunately this can't work at the moment because edk2 needs several
changes in order to run in a Realm

> 
> # cd /home/gavin/sandbox/CCA
> # cp /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd ./
> 
> ---> Start the guest and no output from the console
> 
> host# mount | grep 9p
> shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
> host# export SHR_DIR="/mnt/shr0"
> host# qemu-system-aarch64 -accel kvm                              \
>       -machine virt,gic-version=3,confidential-guest-support=rme0 \
>       -cpu host -smp 2 -m 512M                                    \
>       -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
>       -monitor none -serial mon:stdio -nographic                  \
>       -bios /mnt/edk2-aarch64-code.fd                             \
>       -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
>       -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
>       -append 'console=ttyAMA0'
>         :
>       <no output from the console>
>         :
>       (QEMU) q
> 
> There are some messages from host's console indicating RMI/RMM servicing
> states when the guest is running at background. After the guest is terminated,
> the host crashes.
> 
> SMC_RMM_RTT_CREATE            102dff000 122c2e000 1e00000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 1234a7000 2000000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 1235bd000 2200000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 12387c000 2400000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 123a5a000 2600000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 12407d000 2800000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 124109000 2a00000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 123e49000 2c00000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 124275000 2e00000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 123138000 3000000 3 > RMI_SUCCESS
> SMC_RMM_RTT_CREATE            102dff000 124d07000 3200000 3 > RMI_SUCCESS
>  :
>  :
> [22768.994481] rcu: INFO: rcu_preempt self-detected stall on CPU

This is a warning rather than a crash. The current KVM patches spend too
much time tearing down guest page tables (SMC calls to RMM) while holding
the mmu lock. Not very nice but harmless.

Thanks,
Jean

> [22769.006861] rcu: 	3-....: (2751 ticks this GP) idle=93ec/1/0x4000000000000000 softirq=114451/115721 fqs=1160
> [22769.020475] rcu: 	(t=5257 jiffies g=531913 q=7 ncpus=8)
> [22769.030547] CPU: 3 PID: 198 Comm: qemu-system-aar Not tainted 6.9.0-rc1-gavin-gfcfc92d6ff07 #1
> [22769.041847] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
> [22769.050548] pstate: 60402009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [22769.059382] pc : kvm_realm_unmap_range+0x308/0x32c
> [22769.070275] lr : kvm_realm_unmap_range+0x304/0x32c
> [22769.075893] sp : ffff800080a3b930
> [22769.079929] x29: ffff800080a3b930 x28: 00000000003d7000 x27: 00000000003d6000
> [22769.092990] x26: 00000000c4000152 x25: ffffffffffffffff x24: 0000000000000000
> [22769.101150] x23: 0000010000000000 x22: 00000000c4000155 x21: 0000000102dff000
> [22769.109056] x20: ffff8000801a5e00 x19: 0000000000000000 x18: 0000000000000001
> [22769.117042] x17: 0000000000000000 x16: 000000000000000e x15: 0000000000000000
> [22769.124991] x14: 0000ffff7fa14000 x13: 0000000000000002 x12: 000000000010d594
> [22769.134213] x11: 0000000000000002 x10: 00000000ffffffff x9 : ffffffffffffffff
> [22769.142951] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 00000000c2dff000
> [22769.151413] x5 : 0000000102f56000 x4 : 0000000000000015 x3 : 0000000000000000
> [22769.159932] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
> [22769.169674] Call trace:
> [22769.174026]  kvm_realm_unmap_range+0x308/0x32c
> [22769.181046]  __unmap_stage2_range+0x60/0x7c
> [22769.186396]  kvm_free_stage2_pgd+0xa0/0xd4
> [22769.191766]  kvm_arch_flush_shadow_all+0x1c/0x34
> [22769.197879]  kvm_mmu_notifier_release+0x30/0x84
> [22769.203304]  __mmu_notifier_release+0x7c/0x1f8
> [22769.209340]  exit_mmap+0x264/0x274
> [22769.213992]  __mmput+0x40/0x150
> [22769.218635]  mmput+0x50/0x5c
> [22769.222606]  do_exit+0x288/0x92c
> [22769.226935]  do_group_exit+0x34/0x90
> [22769.231359]  get_signal+0x814/0x820
> [22769.236537]  do_signal+0x90/0x1320
> [22769.241145]  do_notify_resume+0xc8/0x140
> [22769.246458]  el0_svc+0xc8/0xdc
> [22769.250913]  el0t_64_sync_handler+0x13c/0x158
> [22769.256045]  el0t_64_sync+0x190/0x194
> 
> Thanks,
> Gavin
> 
> 
> 
> 

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

Hi Jean,

On 6/3/24 18:24, Jean-Philippe Brucker wrote:
> On Sat, Jun 01, 2024 at 08:14:46PM +1000, Gavin Shan wrote:
>> ---> guest edk2
>>
>> # git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
>> # cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
>> # git submodule update --init --recursive;  \
>>    source edksetup.sh; make -j -C BaseTools; \
>>    export GCC5_AARCH64_PREFIX=;              \
> 
> Doesn't this needs a cross-compiler, something like "aarch64-linux-gnu-" ?
> 

No, I was building everything using a native compiler instead of a cross compiler.
All packages were compiled on a NVidia's grace-hopper machine.

[root@nvidia-grace-hopper-05 ~]# cat /etc/system-release
Red Hat Enterprise Linux release 9.5 Beta (Plow)
[root@nvidia-grace-hopper-05 ~]# uname -r
6.7.0-rc2-gavin+
[root@nvidia-grace-hopper-05 ~]# gcc --version
gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I tried the cross compiler and encountered the same build error.

[root@nvidia-grace-hopper-05 edk2-guest]# export | grep GCC5_AARCH64_PREFIX
declare -x GCC5_AARCH64_PREFIX="aarch64-linux-gnu-"
[root@nvidia-grace-hopper-05 edk2-guest]# build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
   :
--add-gnu-debuglink=/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPrePeiCore.debug
"GenFw" -e SEC -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
GenFw: ERROR 3000: Invalid
   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
GenFw: ERROR 3000: Invalid
   :

>>    build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>     :
>>    WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
>> "gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
>> "GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>> GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
>> GenFw: ERROR 3000: Invalid
>>    :
>> build.py...
>>   : error 7000: Failed to execute command
>> 	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
>>
>>
>> build.py...
>>   : error F002: Failed to build module
>> 	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]
>>
>> - Failed -
>> Build end time: 05:42:19, Jun.01 2024
>> Build total time: 00:00:31
>>
>> ---> Use the edk2 image from the latest QEMU source
> 
> Unfortunately this can't work at the moment because edk2 needs several
> changes in order to run in a Realm
> 
>>
>> # cd /home/gavin/sandbox/CCA
>> # cp /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd ./
>>
>> ---> Start the guest and no output from the console
>>
>> host# mount | grep 9p
>> shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
>> host# export SHR_DIR="/mnt/shr0"
>> host# qemu-system-aarch64 -accel kvm                              \
>>        -machine virt,gic-version=3,confidential-guest-support=rme0 \
>>        -cpu host -smp 2 -m 512M                                    \
>>        -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
>>        -monitor none -serial mon:stdio -nographic                  \
>>        -bios /mnt/edk2-aarch64-code.fd                             \
>>        -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
>>        -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
>>        -append 'console=ttyAMA0'
>>          :
>>        <no output from the console>
>>          :
>>        (QEMU) q
>>
>> There are some messages from host's console indicating RMI/RMM servicing
>> states when the guest is running at background. After the guest is terminated,
>> the host crashes.
>>
>> SMC_RMM_RTT_CREATE            102dff000 122c2e000 1e00000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 1234a7000 2000000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 1235bd000 2200000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 12387c000 2400000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 123a5a000 2600000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 12407d000 2800000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 124109000 2a00000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 123e49000 2c00000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 124275000 2e00000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 123138000 3000000 3 > RMI_SUCCESS
>> SMC_RMM_RTT_CREATE            102dff000 124d07000 3200000 3 > RMI_SUCCESS
>>   :
>>   :
>> [22768.994481] rcu: INFO: rcu_preempt self-detected stall on CPU
> 
> This is a warning rather than a crash. The current KVM patches spend too
> much time tearing down guest page tables (SMC calls to RMM) while holding
> the mmu lock. Not very nice but harmless.
> 

Ok, I can look into this deeply after I can bring up the guest successfully.

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

Hi Gavin,

On Tue, Jun 04, 2024 at 01:02:08PM +1000, Gavin Shan wrote:
> Hi Jean,
> 
> On 6/3/24 18:24, Jean-Philippe Brucker wrote:
> > On Sat, Jun 01, 2024 at 08:14:46PM +1000, Gavin Shan wrote:
> > > ---> guest edk2
> > > 
> > > # git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
> > > # cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
> > > # git submodule update --init --recursive;  \
> > >    source edksetup.sh; make -j -C BaseTools; \
> > >    export GCC5_AARCH64_PREFIX=;              \
> > 
> > Doesn't this needs a cross-compiler, something like "aarch64-linux-gnu-" ?
> > 
> 
> No, I was building everything using a native compiler instead of a cross compiler.
> All packages were compiled on a NVidia's grace-hopper machine.
> 
> [root@nvidia-grace-hopper-05 ~]# cat /etc/system-release
> Red Hat Enterprise Linux release 9.5 Beta (Plow)
> [root@nvidia-grace-hopper-05 ~]# uname -r
> 6.7.0-rc2-gavin+
> [root@nvidia-grace-hopper-05 ~]# gcc --version
> gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
> Copyright (C) 2021 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> I tried the cross compiler and encountered the same build error.
> 
> [root@nvidia-grace-hopper-05 edk2-guest]# export | grep GCC5_AARCH64_PREFIX
> declare -x GCC5_AARCH64_PREFIX="aarch64-linux-gnu-"
> [root@nvidia-grace-hopper-05 edk2-guest]# build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>   :
> --add-gnu-debuglink=/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPrePeiCore.debug
> "GenFw" -e SEC -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
> GenFw: ERROR 3000: Invalid
>   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.

Ah I've seen this once but it disappeared as I tried to investigate and
I've since changed the implementation, so I don't have many notes about
it.

Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
implementation for SetAttribute", but it may give false positives if the
error depends on some random linker placement. Could be
"ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
adds a 4k page to the data section for the ealy RSI config call, though
that has explicit 4kB alignment.

In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
the link flags may have made the error disappear, but I doubt it's the
right fix.

I'll try GCC 11 to see if I can reproduce.

> GenFw: ERROR 3000: Invalid
>   :
> 
> > >    build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
> > >     :
> > >    WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> > > cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
> > > cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
> > > "gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
> > > "GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> > > GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
> > > cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
> > > cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
> > > GenFw: ERROR 3000: Invalid
> > >    :
> > > build.py...
> > >   : error 7000: Failed to execute command
> > > 	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
> > > 
> > > 
> > > build.py...
> > >   : error F002: Failed to build module
> > > 	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]
> > > 
> > > - Failed -
> > > Build end time: 05:42:19, Jun.01 2024
> > > Build total time: 00:00:31
> > > 


> Ok, I can look into this deeply after I can bring up the guest successfully.

Note that the guest edk2 is optional and experimental, you can use direct
kernel boot to get a working demo quicker.

Thanks,
Jean

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

On Fri, May 31, 2024 at 05:24:44PM +0200, Ard Biesheuvel wrote:
> > I'm able to reproduce this even without RME. This code was introduced
> > recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
> > conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
> >
> > A slightly reduced reproducer:
> >
> > $ cd edk2/
> > $ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
> > $ cd ..
> >
> > $ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
> > $ cd tf-a/
> > $ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
> >   dd if=build/qemu/debug/bl1.bin of=flash.bin && \
> >   dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
> > $ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
> >
> 
> Hmm, this is not something I anticipated.
> 
> The problem here is that ArmVirtQemuKernel does not actually support
> dynamic PCDs, so instead, the PCD here is 'patchable', which means
> that the underlying value is just overwritten in the binary image, and
> does not propagate to the rest of the firmware. I assume the write
> ends up targettng a location that does not tolerate this.

Yes, the QemuVirtMemInfoLib declares this region read-only, so we end up
with a permission fault

  // Map the FV region as normal executable memory
  VirtualMemoryTable[2].PhysicalBase = PcdGet64 (PcdFvBaseAddress);
  VirtualMemoryTable[2].VirtualBase  = VirtualMemoryTable[2].PhysicalBase;
  VirtualMemoryTable[2].Length       = FixedPcdGet32 (PcdFvSize);
  VirtualMemoryTable[2].Attributes   = ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK_RO;

Making it writable doesn't seem sufficient, since I then get a "HVC issued
at EL2" fault. I'll keep debugging.

Thanks,
Jean

> 
> Running ArmVirtQemu or ArmVirtQemuKernel at EL2 has really only ever
> worked by accident, it was simply never intended for that. The fix in
> question was a last minute tweak to prevent some CVE fixes pushed by
> MicroSoft from breaking network boot entirely, and now that the
> release has been made, I guess we should revisit this and fix it
> properly.
> 
> So the underlying issue here is that on these platforms, we need to
> decide at runtime whether to use HVC or SMC instructions for SMCCC
> calls. This code attempts to record this into a dynamic PCD once at
> boot, in a way that permits other users of the same library to simply
> hardcode this in the platform definition (given that bare metal
> platforms never need this flexibility).

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Ard Biesheuvel]

On Tue, 4 Jun 2024 at 20:08, Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Fri, May 31, 2024 at 05:24:44PM +0200, Ard Biesheuvel wrote:
> > > I'm able to reproduce this even without RME. This code was introduced
> > > recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
> > > conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
> > >
> > > A slightly reduced reproducer:
> > >
> > > $ cd edk2/
> > > $ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
> > > $ cd ..
> > >
> > > $ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
> > > $ cd tf-a/
> > > $ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
> > >   dd if=build/qemu/debug/bl1.bin of=flash.bin && \
> > >   dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
> > > $ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
> > >
> >
> > Hmm, this is not something I anticipated.
> >
> > The problem here is that ArmVirtQemuKernel does not actually support
> > dynamic PCDs, so instead, the PCD here is 'patchable', which means
> > that the underlying value is just overwritten in the binary image, and
> > does not propagate to the rest of the firmware. I assume the write
> > ends up targettng a location that does not tolerate this.
>
> Yes, the QemuVirtMemInfoLib declares this region read-only, so we end up
> with a permission fault
>
>   // Map the FV region as normal executable memory
>   VirtualMemoryTable[2].PhysicalBase = PcdGet64 (PcdFvBaseAddress);
>   VirtualMemoryTable[2].VirtualBase  = VirtualMemoryTable[2].PhysicalBase;
>   VirtualMemoryTable[2].Length       = FixedPcdGet32 (PcdFvSize);
>   VirtualMemoryTable[2].Attributes   = ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK_RO;
>
> Making it writable doesn't seem sufficient, since I then get a "HVC issued
> at EL2" fault. I'll keep debugging.
>

That is expected, sadly. As I said, this code was never intended to run at EL2.

The dynamic PCD will propagate to other boot stages. However, the
'patchable' PCD that we use in ArmVirtQemuKernel is local to the
driver, and other users of the PCD will see the default value of
'HVC'. Which would be fine if we only executed at EL1.

So I know exactly what is wrong and have an idea how to fix it - I
just need to find the time for it.

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

Hi Jean,

On 6/4/24 21:15, Jean-Philippe Brucker wrote:
> On Tue, Jun 04, 2024 at 01:02:08PM +1000, Gavin Shan wrote:
>> On 6/3/24 18:24, Jean-Philippe Brucker wrote:
>>> On Sat, Jun 01, 2024 at 08:14:46PM +1000, Gavin Shan wrote:
>>>> ---> guest edk2
>>>>
>>>> # git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
>>>> # cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
>>>> # git submodule update --init --recursive;  \
>>>>     source edksetup.sh; make -j -C BaseTools; \
>>>>     export GCC5_AARCH64_PREFIX=;              \
>>>
>>> Doesn't this needs a cross-compiler, something like "aarch64-linux-gnu-" ?
>>>
>>
>> No, I was building everything using a native compiler instead of a cross compiler.
>> All packages were compiled on a NVidia's grace-hopper machine.
>>
>> [root@nvidia-grace-hopper-05 ~]# cat /etc/system-release
>> Red Hat Enterprise Linux release 9.5 Beta (Plow)
>> [root@nvidia-grace-hopper-05 ~]# uname -r
>> 6.7.0-rc2-gavin+
>> [root@nvidia-grace-hopper-05 ~]# gcc --version
>> gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
>> Copyright (C) 2021 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> I tried the cross compiler and encountered the same build error.
>>
>> [root@nvidia-grace-hopper-05 edk2-guest]# export | grep GCC5_AARCH64_PREFIX
>> declare -x GCC5_AARCH64_PREFIX="aarch64-linux-gnu-"
>> [root@nvidia-grace-hopper-05 edk2-guest]# build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>    :
>> --add-gnu-debuglink=/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPrePeiCore.debug
>> "GenFw" -e SEC -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
>> GenFw: ERROR 3000: Invalid
>>    WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> 
> Ah I've seen this once but it disappeared as I tried to investigate and
> I've since changed the implementation, so I don't have many notes about
> it.
> 
> Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
> implementation for SetAttribute", but it may give false positives if the
> error depends on some random linker placement. Could be
> "ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
> adds a 4k page to the data section for the ealy RSI config call, though
> that has explicit 4kB alignment.
> 
> In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
> the link flags may have made the error disappear, but I doubt it's the
> right fix.
> 
> I'll try GCC 11 to see if I can reproduce.
> 

Ok. I run a git-bisect and the first problematic commit is 1153ae939c
("ArmVirtPkg/ArmPlatformLibQemu: Add a third-level page table for the UART idmap")

I'm not familiar with edk2. The error is raised by BaseTools/Source/C/GenFw/Elf64Convert.c::WriteSections64()
where the relocatable address isn't properly aligned to 4KB. So I modified the code
as below, but I have to run two consecutive builds. In the first attempt build, I
still hit the same error.

---> VirtPkg/Library/ArmPlatformLibQemu/IdMap.S

   .align    12
   .globl    idmap
   .globl    uart_pte
   .section  ".data.idmap", "aw", %progbits
   .align    12

# source edksetup.sh; export GCC5_AARCH64_PREFIX=
# make -j -C BaseTools; \                                               <<< Failed on the first attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
make: *** [GNUmakefile:405: /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi] Error 2

# make -j -C BaseTools; \                                              <<< Succeed on the second attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
Generating FVMAIN FV
######
Fd File Name:QEMU_VARS (/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
    :
- Done -
Build end time: 21:04:05, Jun.04 2024
Build total time: 00:00:06

After that, I'm unable to start the guest with the edk2 image successfully.

host# # mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# cat ./realm.sh
#!/bin/sh

SHR_DIR="/mnt/shr0"

qemu-system-aarch64 -accel kvm                              \
-machine virt,gic-version=3,confidential-guest-support=rme0 \
-cpu host -smp 2 -m 512M                                    \
-object 'rme-guest,id=rme0,measurement-algo=sha512'         \
-monitor none -serial mon:stdio -nographic                  \
-bios ${SHR_DIR}/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd \
-kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
-initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
-append 'console=ttyAMA0'

host# ./realm.sh
UEFI firmware (version  built at 19:56:47 on Jun  4 2024)
add-symbol-file /home/gavin/sandbox/C                              <<< I don't see more output after it

>> GenFw: ERROR 3000: Invalid
>>    :
>>
>>>>     build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>>>      :
>>>>     WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
>>>> "gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
>>>> "GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>>>> GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
>>>> GenFw: ERROR 3000: Invalid
>>>>     :
>>>> build.py...
>>>>    : error 7000: Failed to execute command
>>>> 	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
>>>>
>>>>
>>>> build.py...
>>>>    : error F002: Failed to build module
>>>> 	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]
>>>>
>>>> - Failed -
>>>> Build end time: 05:42:19, Jun.01 2024
>>>> Build total time: 00:00:31
>>>>
> 
> 
>> Ok, I can look into this deeply after I can bring up the guest successfully.
> 
> Note that the guest edk2 is optional and experimental, you can use direct
> kernel boot to get a working demo quicker.
> 

I never did this before. Could you please provide the detailed steps on this?

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

On Wed, Jun 05, 2024 at 11:28:47AM +1000, Gavin Shan wrote:
> > >    WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> > 
> > Ah I've seen this once but it disappeared as I tried to investigate and
> > I've since changed the implementation, so I don't have many notes about
> > it.
> > 
> > Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
> > implementation for SetAttribute", but it may give false positives if the
> > error depends on some random linker placement. Could be
> > "ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
> > adds a 4k page to the data section for the ealy RSI config call, though
> > that has explicit 4kB alignment.
> > 
> > In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
> > the link flags may have made the error disappear, but I doubt it's the
> > right fix.
> > 
> > I'll try GCC 11 to see if I can reproduce.
> > 
> 
> Ok. I run a git-bisect and the first problematic commit is 1153ae939c
> ("ArmVirtPkg/ArmPlatformLibQemu: Add a third-level page table for the UART idmap")

Ah thanks, I'm able to reproduce the problem now, it was my local config
that masked it.

> 
> I'm not familiar with edk2. The error is raised by BaseTools/Source/C/GenFw/Elf64Convert.c::WriteSections64()
> where the relocatable address isn't properly aligned to 4KB. So I modified the code
> as below, but I have to run two consecutive builds. In the first attempt build, I
> still hit the same error.

This seems to be because GenFw generates a file even on error, so it
doesn't retry the second time.

This commit moves the page tables from .rodata to .data. When linking
IdMap.obj into ArmPlatformPrePeiCore.dll, the alignment of the .text
section changes from 0x1000 to 0x800. This change comes from the linker
script putting .rodata into .text. I don't know why the included .rodata
alignment affects the .text alignment, but I don't think it matters here.

In GenFw, ScanSections64() calculates a mCoffAlignment as the max
.text/.data/.hii section alignement. Since with this commit, .data
alignement (0x1000) becomes larger than .text (0x800), it picks 0x1000 as
the output text offset, and then WriteSections64() complains that this
offset isn't equal to the input .text alignment modulo 0x1000.

The linker script says:

  /*
   * The alignment of the .data section should be less than or equal to the
   * alignment of the .text section. This ensures that the relative offset
   * between these sections is the same in the ELF and the PE/COFF versions of
   * this binary.
   */

but that's not what we're getting. I don't have a fix yet, other than
forcing the .text and .data alignment to 4k.

> ---> VirtPkg/Library/ArmPlatformLibQemu/IdMap.S
> 
>   .align    12
>   .globl    idmap
>   .globl    uart_pte
>   .section  ".data.idmap", "aw", %progbits
>   .align    12
> 
> # source edksetup.sh; export GCC5_AARCH64_PREFIX=
> # make -j -C BaseTools; \                                               <<< Failed on the first attempt
>   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>    :
> WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> make: *** [GNUmakefile:405: /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi] Error 2
> 
> # make -j -C BaseTools; \                                              <<< Succeed on the second attempt
>   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>    :
> Generating FVMAIN FV
> ######
> Fd File Name:QEMU_VARS (/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
>    :
> - Done -
> Build end time: 21:04:05, Jun.04 2024
> Build total time: 00:00:06
> 
> After that, I'm unable to start the guest with the edk2 image successfully.
> 
> host# # mount | grep 9p
> shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
> host# cat ./realm.sh
> #!/bin/sh
> 
> SHR_DIR="/mnt/shr0"
> 
> qemu-system-aarch64 -accel kvm                              \
> -machine virt,gic-version=3,confidential-guest-support=rme0 \
> -cpu host -smp 2 -m 512M                                    \
> -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
> -monitor none -serial mon:stdio -nographic                  \
> -bios ${SHR_DIR}/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd \
> -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
> -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
> -append 'console=ttyAMA0'
> 
> host# ./realm.sh
> UEFI firmware (version  built at 19:56:47 on Jun  4 2024)
> add-symbol-file /home/gavin/sandbox/C                              <<< I don't see more output after it

I'm guessing in this case the firmware was corrupted because GenFw fails the
first time and never generated a complete binary

> > 
> > Note that the guest edk2 is optional and experimental, you can use direct
> > kernel boot to get a working demo quicker.
> > 
> 
> I never did this before. Could you please provide the detailed steps on this?

Removing the -bios parameter to QEMU should be enough. You can also add
'earlycon' to -append to show early boot errors.

Thanks,
Jean

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

On 6/6/24 01:56, Jean-Philippe Brucker wrote:
> On Wed, Jun 05, 2024 at 11:28:47AM +1000, Gavin Shan wrote:
>>>>     WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
>>>
>>> Ah I've seen this once but it disappeared as I tried to investigate and
>>> I've since changed the implementation, so I don't have many notes about
>>> it.
>>>
>>> Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
>>> implementation for SetAttribute", but it may give false positives if the
>>> error depends on some random linker placement. Could be
>>> "ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
>>> adds a 4k page to the data section for the ealy RSI config call, though
>>> that has explicit 4kB alignment.
>>>
>>> In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
>>> the link flags may have made the error disappear, but I doubt it's the
>>> right fix.
>>>
>>> I'll try GCC 11 to see if I can reproduce.
>>>
>>
>> Ok. I run a git-bisect and the first problematic commit is 1153ae939c
>> ("ArmVirtPkg/ArmPlatformLibQemu: Add a third-level page table for the UART idmap")
> 
> Ah thanks, I'm able to reproduce the problem now, it was my local config
> that masked it.
> 
>>
>> I'm not familiar with edk2. The error is raised by BaseTools/Source/C/GenFw/Elf64Convert.c::WriteSections64()
>> where the relocatable address isn't properly aligned to 4KB. So I modified the code
>> as below, but I have to run two consecutive builds. In the first attempt build, I
>> still hit the same error.
> 
> This seems to be because GenFw generates a file even on error, so it
> doesn't retry the second time.
> 
> This commit moves the page tables from .rodata to .data. When linking
> IdMap.obj into ArmPlatformPrePeiCore.dll, the alignment of the .text
> section changes from 0x1000 to 0x800. This change comes from the linker
> script putting .rodata into .text. I don't know why the included .rodata
> alignment affects the .text alignment, but I don't think it matters here.
> 
> In GenFw, ScanSections64() calculates a mCoffAlignment as the max
> .text/.data/.hii section alignement. Since with this commit, .data
> alignement (0x1000) becomes larger than .text (0x800), it picks 0x1000 as
> the output text offset, and then WriteSections64() complains that this
> offset isn't equal to the input .text alignment modulo 0x1000.
> 
> The linker script says:
> 
>    /*
>     * The alignment of the .data section should be less than or equal to the
>     * alignment of the .text section. This ensures that the relative offset
>     * between these sections is the same in the ELF and the PE/COFF versions of
>     * this binary.
>     */
> 
> but that's not what we're getting. I don't have a fix yet, other than
> forcing the .text and .data alignment to 4k.
> 

Jean, thanks for your explanation. Right, the issue is caused by mismatched
alignments for ELF and PE/COFF sections. I ever dumped the variables at the
failing point, showing the mismatched alignments (0x800 vs 0x1000). Apart from
that, the virtual address of 'text' section is aligned to 0x800 instead of
0x1000 after ArmPlatformPrePeiCore.dll is dumped by 'readelf'.

SecHdr->sh_addr:                    0x800              <<< Mismatched alignment between ELF and PE/COFF
SecOffset:                          0x1000
SymShdr->sh_addr:                   0x800
mCoffSectionsOffset[Sym->st_shndx]: 0x1000
GenFw: ERROR 3000: Invalid
   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.

# readelf -S Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
Section Headers:
   [Nr] Name              Type             Address           Offset
        Size              EntSize          Flags  Link  Info  Align
   [ 0]                   NULL             0000000000000000  00000000
        0000000000000000  0000000000000000           0     0     0
   [ 1] .text             PROGBITS         0000000000000800  00000800       <<< Aligned to 0x800
        00000000000051b8  0000000000000000  AX       0     0     2048

With the following changes, I'm able to build the firmware successfully. I don't
see how COMMONPAGESIZE is sorted out because I don't find its definition in the
source code.

diff --git a/BaseTools/Scripts/GccBase.lds b/BaseTools/Scripts/GccBase.lds
index 9f27e83bb0..5463df47a9 100644
--- a/BaseTools/Scripts/GccBase.lds
+++ b/BaseTools/Scripts/GccBase.lds
@@ -20,7 +20,8 @@ SECTIONS {
     */
    . = PECOFF_HEADER_SIZE;
  
-  .text : ALIGN(CONSTANT(COMMONPAGESIZE)) {
+  /* .text : ALIGN(CONSTANT(COMMONPAGESIZE)) { */^M
+  .text : ALIGN(4096) {^M

# <rebuild>
# readelf -S Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
There are 24 section headers, starting at offset 0x63050:

Section Headers:
   [Nr] Name              Type             Address           Offset
        Size              EntSize          Flags  Link  Info  Align
   [ 0]                   NULL             0000000000000000  00000000
        0000000000000000  0000000000000000           0     0     0
   [ 1] .text             PROGBITS         0000000000001000  00001000     <<< Aligned to 0x1000 now
        00000000000051b8  0000000000000000  AX       0     0     4096

Even the edk2 for the guest can be built successfully, but I'm not able to try it
because I'm unable to bring up the host now. I tried to rebuild the environment
from scratch, the host runs into crash inside EDK2 unfortunately...

   TF-RMM:   https://git.codelinaro.org/linaro/dcap/rmm.git                       (branch: cca/v2)
   EDK2:     git@github.com:tianocore/edk2.git                                    (tag:    edk2-stable202402)
   TF-A:     https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git   (branch: cca/v2)
   QEMU:     https://git.qemu.org/git/qemu.git                                    (branch: master)
   KERNEL:   https://git.gitlab.arm.com/linux-arm/linux-cca.git                   (branch: cca-full/v2)
   BuildRoot: <doesn't matter at present>

arm64-server# home/gavin/sandbox/qemu.main/build/qemu-system-aarch64      \
               -M virt,virtualization=on,secure=on,gic-version=3,acpi=off  \
               -cpu max,x-rme=on -m 8G -smp 8                              \
               -monitor none -serial mon:stdio -nographic -nodefaults      \
               -bios /home/gavin/sandbox/CCA/tf-a/flash.bin                \
               -kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image \
               -append console=ttyAMA0 root=/dev/vda                       \
               -drive format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0 \
               -device virtio-blk-pci,drive=hd0                            \
               -netdev tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
               -device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0                          \
               -fsdev local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0             \
               -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
                  :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL1: Built : 23:14:56, Jun  5 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
INFO:    BL1: Loading BL2
INFO:    Loading image id=1 at address 0xe06b000
INFO:    Image id=1 loaded: 0xe06b000 - 0xe0742d1
NOTICE:  BL1: Booting BL2
INFO:    Entry point address = 0xe06b000
INFO:    SPSR = 0x3cd
INFO:    [GPT] Boot Configuration
INFO:      PPS/T:     0x2/40
INFO:      PGS/P:     0x0/12
INFO:      L0GPTSZ/S: 0x0/30
INFO:      PAS count: 0x6
INFO:      L0 base:   0xedfe000
INFO:    [GPT] PAS[0]: base 0xe001000, size 0xff000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[1]: base 0xe100000, size 0xcfe000, GPI 0x8, type 0x1
INFO:    [GPT] PAS[2]: base 0xedfe000, size 0x202000, GPI 0xa, type 0x1
INFO:    [GPT] PAS[3]: base 0x40000000, size 0x100000, GPI 0x9, type 0x1
INFO:    [GPT] PAS[4]: base 0x40100000, size 0x2800000, GPI 0xb, type 0x1
INFO:    [GPT] PAS[5]: base 0x42900000, size 0x1fd700000, GPI 0x9, type 0x1
INFO:    Enabling Granule Protection Checks
NOTICE:  BL2: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL2: Built : 23:14:56, Jun  5 2024
INFO:    BL2: Doing platform setup
INFO:    Reserved RMM memory [0x40100000, 0x428fffff] in Device tree
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0xe0a0000
INFO:    Image id=3 loaded: 0xe0a0000 - 0xe0b10c4
INFO:    BL2: Loading image id 35
INFO:    Loading image id=35 at address 0x40100000
INFO:    Image id=35 loaded: 0x40100000 - 0x403033b0
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x60000000
INFO:    Image id=5 loaded: 0x60000000 - 0x60200000
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0xe0a0000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL31: Built : 23:14:56, Jun  5 2024
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    Maximum SPI INTID supported: 287
INFO:    BL31: Initializing runtime services
INFO:    RMM setup done.
INFO:    BL31: Initializing RMM
INFO:    RMM init start.
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: Jun  5 2024 23:03:00
INFO:    RMM init end.
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x60000000
INFO:    SPSR = 0x3c9
Loading driver at 0x00060009160 EntryPoint=0x00000000000
ArmVirtGetMemoryMap: Dumping System DRAM Memory Map:
	PhysicalBase: 0x40000000
	VirtualBase: 0x40000000
	Length: 0x200000000
UEFI firmware (version  built at 23:28:51 on Jun  5 2024)
PlatformPeim: PL011 UART (console) @ 0x9000000
PlatformPeim: PL011 UART (debug) @ 0x9000000
   :
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
SetMemoryAttributes: BaseAddress == 0x22DC00000, Length == 0x1CE0000, Attributes == 0x20000
SetMemoryAttributes: BaseAddress == 0x22F8E0000, Length == 0xE50000, Attributes == 0x4000
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
EFI stub: update_fdt() ... done
EFI stub: efi_exit_boot_services: enter
EFI stub: efi_exit_boot_services: efi_pci_disable_bridge_busmaster
EFI stub: efi_exit_boot_services: efi_get_memory_map
EFI stub: efi_exit_boot_services: priv_func
=====> CoreExitBootServices
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x000000023BE60000 - 0x0000000000040000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238AF0000 - 0x0000000000040000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238AA0000 - 0x0000000000040000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238A50000 - 0x0000000000040000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238960000 - 0x0000000000040000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x000000023BE20000 - 0x0000000000030000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238860000 - 0x0000000000030000 (0x0000000000000008)
MemoryProtectionExitBootServicesCallback - 0
SetUefiImageMemoryAttributes - 0x0000000238820000 - 0x0000000000030000 (0x0000000000000008)
CoreExitBootServices: MemoryProtectionExitBootServicesCallback
CoreExitBootServices: SaveAndSetDebugTimer
CoreExitBootServices: gCpu->DisableInterrupt
CoreExitBootServices: CalculateEfiHdrCrc
CoreExitBootServices: Return with status=0x0


Synchronous Exception at 0x000000023248E9B4
PC 0x00023248E9B4
PC 0x00023248EA70
PC 0x00023248EBA8
PC 0x000232490FC8
PC 0x00023248A004
PC 0x00023248973C
PC 0x0002324894DC
PC 0x00023F2C7FA8 (0x00023F2C1000+0x00006FA8) [ 1] DxeCore.dll
PC 0x00023BCC6604 (0x00023BCBE000+0x00008604) [ 2] BdsDxe.dll
PC 0x00023F2CBC68 (0x00023F2C1000+0x0000AC68) [ 3] DxeCore.dll
[ 1] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 2] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 3] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

   X0 0x0000000000000000   X1 0x000000023F2C0740   X2 0x000000000000000A   X3 0x00000002325D3E26
   X4 0x0000000000000020   X5 0xFFFFFFFFFFFFFFFE   X6 0x0000000000000000   X7 0x0000000000000000
   X8 0x0000000000000000   X9 0x0000000238820000  X10 0x000000000000002E  X11 0x00000000000023D0
  X12 0x00000000000023C4  X13 0x0000000000000001  X14 0x0000000000000002  X15 0x0000000000000000
  X16 0x000000023BD30280  X17 0x000000000066BD90  X18 0x0000000000000000  X19 0x000000023F2C0740
  X20 0x00000002325D3E27  X21 0x000000000000FFFF  X22 0x000000000000000D  X23 0x000000000010FFFF
  X24 0x000000000000D800  X25 0x000000023F2C0AA8  X26 0x00000002325E2000  X27 0x0000000000000002
  X28 0x0000000000000018   FP 0x000000023F2C06F0   LR 0x000000023248EA70

   V0 0x0000000000000000 0000000000000000   V1 0xFFFFFF80FFFFFFD0 000000023F2C0800
   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000
   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000
   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000
   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000
  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000
  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000
  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000
  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000
  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000
  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000
  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000
  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000
  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000
  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000
  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000

   SP 0x000000023F2C06F0  ELR 0x000000023248E9B4  SPSR 0xA00002C9  FPSR 0x00000000
  ESR 0x96000006          FAR 0x0000000000000008

  ESR : EC 0x25  IL 0x1  ISS 0x00000006

Data abort: Translation fault, second level

Stack dump:
   000023F2C05F0: 000000023F2C06A0 000000023BD31FF4 0000000238820000 0000000000000002
   000023F2C0610: 000000023B1E2000 000000023FFF9000 0000000000000E20 00000000001FFFFF
   000023F2C0630: 0000000238850000 0000000000000001 0000000000000003 000000023FFF9E20
   000023F2C0650: 000000000000070C 0000000000000000 000000023F2C06D0 000000023F2DD644
   000023F2C0670: 000000023F2C07C8 000000023F2C095F 0000000000000001 000000023BD358ED
   000023F2C0690: 000000000000070C 0000000000000000 000000023F2C0750 000000023BD31FF4
   000023F2C06B0: 0000000238820000 0000000000000001 000000023FFF9000 000000023FFFA000
   000023F2C06D0: 000000023F2C07F0 000000023F2C2534 0000000000000001 0000000000000000
> 000023F2C06F0: 000000023F2C0700 000000023248EA70 000000023F2C0840 000000023248EBA8
   000023F2C0710: 00000002325D42BF 000000023B17E918 00000002325E2000 0000000232489978
   000023F2C0730: 000000023F2C0AB0 00000002325E2000 0020004900460045 0062007500740073
   000023F2C0750: 000000000020003A 0000000000000001 000000023F2C0860 0000000000000001
   000023F2C0770: 0000000000000002 00000000000000FF 0000000000000000 0000007F00000000
   000023F2C0790: 000000023F2C07F0 000000023F2C2548 000000023F2C07C0 000000023F2DEDD8
   000023F2C07B0: 000000023F2C088D 000000023F2EA000 000000023F2C07F0 000000023F2C2548
   000023F2C07D0: 0000000000000001 0000000000000000 000000023F2EB000 000000023F2EA000
ASSERT [ArmCpuDxe] /home/gavin/sandbox/CCA/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(343): ((BOOLEAN)(0==1))
  

>> ---> VirtPkg/Library/ArmPlatformLibQemu/IdMap.S
>>
>>    .align    12
>>    .globl    idmap
>>    .globl    uart_pte
>>    .section  ".data.idmap", "aw", %progbits
>>    .align    12
>>
>> # source edksetup.sh; export GCC5_AARCH64_PREFIX=
>> # make -j -C BaseTools; \                                               <<< Failed on the first attempt
>>    build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>     :
>> WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
>> make: *** [GNUmakefile:405: /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi] Error 2
>>
>> # make -j -C BaseTools; \                                              <<< Succeed on the second attempt
>>    build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>     :
>> Generating FVMAIN FV
>> ######
>> Fd File Name:QEMU_VARS (/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
>>     :
>> - Done -
>> Build end time: 21:04:05, Jun.04 2024
>> Build total time: 00:00:06
>>
>> After that, I'm unable to start the guest with the edk2 image successfully.
>>
>> host# # mount | grep 9p
>> shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
>> host# cat ./realm.sh
>> #!/bin/sh
>>
>> SHR_DIR="/mnt/shr0"
>>
>> qemu-system-aarch64 -accel kvm                              \
>> -machine virt,gic-version=3,confidential-guest-support=rme0 \
>> -cpu host -smp 2 -m 512M                                    \
>> -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
>> -monitor none -serial mon:stdio -nographic                  \
>> -bios ${SHR_DIR}/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd \
>> -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
>> -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
>> -append 'console=ttyAMA0'
>>
>> host# ./realm.sh
>> UEFI firmware (version  built at 19:56:47 on Jun  4 2024)
>> add-symbol-file /home/gavin/sandbox/C                              <<< I don't see more output after it
> 
> I'm guessing in this case the firmware was corrupted because GenFw fails the
> first time and never generated a complete binary
> 

Yeah, the binary isn't reliable in this case.

>>>
>>> Note that the guest edk2 is optional and experimental, you can use direct
>>> kernel boot to get a working demo quicker.
>>>
>>
>> I never did this before. Could you please provide the detailed steps on this?
> 
> Removing the -bios parameter to QEMU should be enough. You can also add
> 'earlycon' to -append to show early boot errors.
> 

I didn't get a chance to try this yet since the host can't be brought up now.
I will try this later. I originally thought some sort of boot wrapper is needed
so that the kernel image has the capability to boot itself. For example, Mark
Rutland's boot wrapper [1] can be leveraged in this case. I don't think Image has
the capability to boot itself, right?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan]

On 6/6/24 15:05, Gavin Shan wrote:
> Even the edk2 for the guest can be built successfully, but I'm not able to try it
> because I'm unable to bring up the host now. I tried to rebuild the environment
> from scratch, the host runs into crash inside EDK2 unfortunately...
> 
>    TF-RMM:   https://git.codelinaro.org/linaro/dcap/rmm.git                       (branch: cca/v2)
>    EDK2:     git@github.com:tianocore/edk2.git                                    (tag:    edk2-stable202402)
>    TF-A:     https://git.codelinaro.org/linaro/dcap/tf-a/trusted-firmware-a.git   (branch: cca/v2)
>    QEMU:     https://git.qemu.org/git/qemu.git                                    (branch: master)
>    KERNEL:   https://git.gitlab.arm.com/linux-arm/linux-cca.git                   (branch: cca-full/v2)
>    BuildRoot: <doesn't matter at present>
> 
> arm64-server# home/gavin/sandbox/qemu.main/build/qemu-system-aarch64      \
>                -M virt,virtualization=on,secure=on,gic-version=3,acpi=off  \
>                -cpu max,x-rme=on -m 8G -smp 8                              \
>                -monitor none -serial mon:stdio -nographic -nodefaults      \
>                -bios /home/gavin/sandbox/CCA/tf-a/flash.bin                \
>                -kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image \
>                -append console=ttyAMA0 root=/dev/vda                       \
>                -drive format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0 \
>                -device virtio-blk-pci,drive=hd0                            \
>                -netdev tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
>                -device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0                          \
>                -fsdev local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0             \
>                -device virtio-9p-device,fsdev=shr0,mount_tag=shr0
>                   :
> NOTICE:  Booting Trusted Firmware
> NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
> NOTICE:  BL1: Built : 23:14:56, Jun  5 2024
> INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
> INFO:    BL1: Loading BL2
> INFO:    Loading image id=1 at address 0xe06b000
> INFO:    Image id=1 loaded: 0xe06b000 - 0xe0742d1
> NOTICE:  BL1: Booting BL2
> INFO:    Entry point address = 0xe06b000
> INFO:    SPSR = 0x3cd
> INFO:    [GPT] Boot Configuration
> INFO:      PPS/T:     0x2/40
> INFO:      PGS/P:     0x0/12
> INFO:      L0GPTSZ/S: 0x0/30
> INFO:      PAS count: 0x6
> INFO:      L0 base:   0xedfe000
> INFO:    [GPT] PAS[0]: base 0xe001000, size 0xff000, GPI 0xa, type 0x1
> INFO:    [GPT] PAS[1]: base 0xe100000, size 0xcfe000, GPI 0x8, type 0x1
> INFO:    [GPT] PAS[2]: base 0xedfe000, size 0x202000, GPI 0xa, type 0x1
> INFO:    [GPT] PAS[3]: base 0x40000000, size 0x100000, GPI 0x9, type 0x1
> INFO:    [GPT] PAS[4]: base 0x40100000, size 0x2800000, GPI 0xb, type 0x1
> INFO:    [GPT] PAS[5]: base 0x42900000, size 0x1fd700000, GPI 0x9, type 0x1
> INFO:    Enabling Granule Protection Checks
> NOTICE:  BL2: v2.10.0(debug):99e0b97aa-dirty
> NOTICE:  BL2: Built : 23:14:56, Jun  5 2024
> INFO:    BL2: Doing platform setup
> INFO:    Reserved RMM memory [0x40100000, 0x428fffff] in Device tree
> INFO:    BL2: Loading image id 3
> INFO:    Loading image id=3 at address 0xe0a0000
> INFO:    Image id=3 loaded: 0xe0a0000 - 0xe0b10c4
> INFO:    BL2: Loading image id 35
> INFO:    Loading image id=35 at address 0x40100000
> INFO:    Image id=35 loaded: 0x40100000 - 0x403033b0
> INFO:    BL2: Loading image id 5
> INFO:    Loading image id=5 at address 0x60000000
> INFO:    Image id=5 loaded: 0x60000000 - 0x60200000
> NOTICE:  BL2: Booting BL31
> INFO:    Entry point address = 0xe0a0000
> INFO:    SPSR = 0x3cd
> NOTICE:  BL31: v2.10.0(debug):99e0b97aa-dirty
> NOTICE:  BL31: Built : 23:14:56, Jun  5 2024
> INFO:    GICv3 without legacy support detected.
> INFO:    ARM GICv3 driver initialized in EL3
> INFO:    Maximum SPI INTID supported: 287
> INFO:    BL31: Initializing runtime services
> INFO:    RMM setup done.
> INFO:    BL31: Initializing RMM
> INFO:    RMM init start.
> Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
> RMM-EL3 Interface v.0.2
> Boot Manifest Interface v.0.3
> RMI/RSI ABI v.1.0/1.0 built: Jun  5 2024 23:03:00
> INFO:    RMM init end.
> INFO:    BL31: Preparing for EL3 exit to normal world
> INFO:    Entry point address = 0x60000000
> INFO:    SPSR = 0x3c9
> Loading driver at 0x00060009160 EntryPoint=0x00000000000
> ArmVirtGetMemoryMap: Dumping System DRAM Memory Map:
>      PhysicalBase: 0x40000000
>      VirtualBase: 0x40000000
>      Length: 0x200000000
> UEFI firmware (version  built at 23:28:51 on Jun  5 2024)
> PlatformPeim: PL011 UART (console) @ 0x9000000
> PlatformPeim: PL011 UART (debug) @ 0x9000000
>    :
> EFI stub: Booting Linux Kernel...
> EFI stub: EFI_RNG_PROTOCOL unavailable
> SetMemoryAttributes: BaseAddress == 0x22DC00000, Length == 0x1CE0000, Attributes == 0x20000
> SetMemoryAttributes: BaseAddress == 0x22F8E0000, Length == 0xE50000, Attributes == 0x4000
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services...
> EFI stub: update_fdt() ... done
> EFI stub: efi_exit_boot_services: enter
> EFI stub: efi_exit_boot_services: efi_pci_disable_bridge_busmaster
> EFI stub: efi_exit_boot_services: efi_get_memory_map
> EFI stub: efi_exit_boot_services: priv_func
> =====> CoreExitBootServices
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x000000023BE60000 - 0x0000000000040000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238AF0000 - 0x0000000000040000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238AA0000 - 0x0000000000040000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238A50000 - 0x0000000000040000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238960000 - 0x0000000000040000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x000000023BE20000 - 0x0000000000030000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238860000 - 0x0000000000030000 (0x0000000000000008)
> MemoryProtectionExitBootServicesCallback - 0
> SetUefiImageMemoryAttributes - 0x0000000238820000 - 0x0000000000030000 (0x0000000000000008)
> CoreExitBootServices: MemoryProtectionExitBootServicesCallback
> CoreExitBootServices: SaveAndSetDebugTimer
> CoreExitBootServices: gCpu->DisableInterrupt
> CoreExitBootServices: CalculateEfiHdrCrc
> CoreExitBootServices: Return with status=0x0
> 
> 
> Synchronous Exception at 0x000000023248E9B4
> PC 0x00023248E9B4
> PC 0x00023248EA70
> PC 0x00023248EBA8
> PC 0x000232490FC8
> PC 0x00023248A004
> PC 0x00023248973C
> PC 0x0002324894DC
> PC 0x00023F2C7FA8 (0x00023F2C1000+0x00006FA8) [ 1] DxeCore.dll
> PC 0x00023BCC6604 (0x00023BCBE000+0x00008604) [ 2] BdsDxe.dll
> PC 0x00023F2CBC68 (0x00023F2C1000+0x0000AC68) [ 3] DxeCore.dll
> [ 1] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> [ 2] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
> [ 3] /home/gavin/sandbox/CCA/edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> 
>    X0 0x0000000000000000   X1 0x000000023F2C0740   X2 0x000000000000000A   X3 0x00000002325D3E26
>    X4 0x0000000000000020   X5 0xFFFFFFFFFFFFFFFE   X6 0x0000000000000000   X7 0x0000000000000000
>    X8 0x0000000000000000   X9 0x0000000238820000  X10 0x000000000000002E  X11 0x00000000000023D0
>   X12 0x00000000000023C4  X13 0x0000000000000001  X14 0x0000000000000002  X15 0x0000000000000000
>   X16 0x000000023BD30280  X17 0x000000000066BD90  X18 0x0000000000000000  X19 0x000000023F2C0740
>   X20 0x00000002325D3E27  X21 0x000000000000FFFF  X22 0x000000000000000D  X23 0x000000000010FFFF
>   X24 0x000000000000D800  X25 0x000000023F2C0AA8  X26 0x00000002325E2000  X27 0x0000000000000002
>   X28 0x0000000000000018   FP 0x000000023F2C06F0   LR 0x000000023248EA70
> 
>    V0 0x0000000000000000 0000000000000000   V1 0xFFFFFF80FFFFFFD0 000000023F2C0800
>    V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000
>    V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000
>    V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000
>    V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000
>   V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000
>   V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000
>   V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000
>   V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000
>   V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000
>   V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000
>   V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000
>   V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000
>   V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000
>   V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000
>   V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000
> 
>    SP 0x000000023F2C06F0  ELR 0x000000023248E9B4  SPSR 0xA00002C9  FPSR 0x00000000
>   ESR 0x96000006          FAR 0x0000000000000008
> 
>   ESR : EC 0x25  IL 0x1  ISS 0x00000006
> 
> Data abort: Translation fault, second level
> 
> Stack dump:
>    000023F2C05F0: 000000023F2C06A0 000000023BD31FF4 0000000238820000 0000000000000002
>    000023F2C0610: 000000023B1E2000 000000023FFF9000 0000000000000E20 00000000001FFFFF
>    000023F2C0630: 0000000238850000 0000000000000001 0000000000000003 000000023FFF9E20
>    000023F2C0650: 000000000000070C 0000000000000000 000000023F2C06D0 000000023F2DD644
>    000023F2C0670: 000000023F2C07C8 000000023F2C095F 0000000000000001 000000023BD358ED
>    000023F2C0690: 000000000000070C 0000000000000000 000000023F2C0750 000000023BD31FF4
>    000023F2C06B0: 0000000238820000 0000000000000001 000000023FFF9000 000000023FFFA000
>    000023F2C06D0: 000000023F2C07F0 000000023F2C2534 0000000000000001 0000000000000000
>> 000023F2C06F0: 000000023F2C0700 000000023248EA70 000000023F2C0840 000000023248EBA8
>    000023F2C0710: 00000002325D42BF 000000023B17E918 00000002325E2000 0000000232489978
>    000023F2C0730: 000000023F2C0AB0 00000002325E2000 0020004900460045 0062007500740073
>    000023F2C0750: 000000000020003A 0000000000000001 000000023F2C0860 0000000000000001
>    000023F2C0770: 0000000000000002 00000000000000FF 0000000000000000 0000007F00000000
>    000023F2C0790: 000000023F2C07F0 000000023F2C2548 000000023F2C07C0 000000023F2DEDD8
>    000023F2C07B0: 000000023F2C088D 000000023F2EA000 000000023F2C07F0 000000023F2C2548
>    000023F2C07D0: 0000000000000001 0000000000000000 000000023F2EB000 000000023F2EA000
> ASSERT [ArmCpuDxe] /home/gavin/sandbox/CCA/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(343): ((BOOLEAN)(0==1))
> 

Please ignore the crash inside edk2, which is caused by verbose messages
added to Linux's EFI driver (wrapper). Those verbose messages are added
by myself and it caused stack overrun. They lead to the crash eventually.
With those verbose messages removed, I don't see the crash. However, the
Linux boots very...very slow.

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
SetMemoryAttributes: BaseAddress == 0x22DC00000, Length == 0x1CE0000, Attributes == 0x20000
SetMemoryAttributes: BaseAddress == 0x22F8E0000, Length == 0xE50000, Attributes == 0x4000
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
SetUefiImageMemoryAttributes - 0x000000023BE60000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238AF0000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238AA0000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238A50000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238960000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000023BE20000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238860000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x0000000238820000 - 0x0000000000030000 (0x0000000000000008)   <<< At least 10 minutes' gap between this and next line of log
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
       :
[   24.057340] Remapping and enabling EFI services.
[   26.618330] smp: Bringing up secondary CPUs ...                                            <<< PSCI service responses very slow
[   28.818256] Detected PIPT I-cache on CPU1
[   28.985946] GICv3: CPU1: found redistributor 1 region 0:0x00000000080c0000
[   29.055568] GICv3: CPU1: using allocated LPI pending table @0x00000001000c0000
[   29.203572] CPU1: Booted secondary processor 0x0000000001 [0x000f0510]
[   36.075187] Detected PIPT I-cache on CPU2
[   36.119712] GICv3: CPU2: found redistributor 2 region 0:0x00000000080e0000
[   36.144795] GICv3: CPU2: using allocated LPI pending table @0x00000001000d0000
[   36.213252] CPU2: Booted secondary processor 0x0000000002 [0x000f0510]
[  115.355610] Detected PIPT I-cache on CPU3
[  115.402037] GICv3: CPU3: found redistributor 3 region 0:0x0000000008100000
[  115.426918] GICv3: CPU3: using allocated LPI pending table @0x00000001000e0000
[  115.508456] CPU3: Booted secondary processor 0x0000000003 [0x000f0510]
[  134.596700] Detected PIPT I-cache on CPU4
[  134.645280] GICv3: CPU4: found redistributor 4 region 0:0x0000000008120000
[  134.670010] GICv3: CPU4: using allocated LPI pending table @0x00000001000f0000
[  134.763347] CPU4: Booted secondary processor 0x0000000004 [0x000f0510]
[  156.200377] Detected PIPT I-cache on CPU5
[  156.251349] GICv3: CPU5: found redistributor 5 region 0:0x0000000008140000
[  156.277133] GICv3: CPU5: using allocated LPI pending table @0x0000000100100000
[  156.382948] CPU5: Booted secondary processor 0x0000000005 [0x000f0510]
[  176.521840] Detected PIPT I-cache on CPU6
[  176.575053] GICv3: CPU6: found redistributor 6 region 0:0x0000000008160000
[  176.600415] GICv3: CPU6: using allocated LPI pending table @0x0000000100110000
[  176.720944] CPU6: Booted secondary processor 0x0000000006 [0x000f0510]
[  198.444988] Detected PIPT I-cache on CPU7
[  198.499710] GICv3: CPU7: found redistributor 7 region 0:0x0000000008180000
[  198.524345] GICv3: CPU7: using allocated LPI pending table @0x0000000100120000
[  198.654758] CPU7: Booted secondary processor 0x0000000007 [0x000f0510]
[  218.456900] smp: Brought up 1 node, 8 CPUs
[  218.590983] SMP: Total of 8 processors activated.
[  218.625265] CPU: All CPU(s) started at EL2
     :
[  758.664394] PTP clock support registered
[  760.772801] EDAC MC: Ver: 3.0.0
[  764.767946] scmi_core: SCMI protocol bus registered
[  767.858837] efivars: Registered efivars operations
[  779.386548] FPGA manager framework
[  780.749886] Advanced Linux Sound Architecture Driver Initialized.
[  797.557785] vgaarb: loaded                                                            <<< No more output after this

It seems the EDK2 binary, built from upstream's 'edk2-stable202402' tag, doesn't work well.

Thanks,
Gavin

Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Jean-Philippe Brucker]

On Thu, Jun 06, 2024 at 03:05:02PM +1000, Gavin Shan wrote:
> > This commit moves the page tables from .rodata to .data. When linking
> > IdMap.obj into ArmPlatformPrePeiCore.dll, the alignment of the .text
> > section changes from 0x1000 to 0x800. This change comes from the linker
> > script putting .rodata into .text. I don't know why the included .rodata
> > alignment affects the .text alignment, but I don't think it matters here.
> > 
> > In GenFw, ScanSections64() calculates a mCoffAlignment as the max
> > .text/.data/.hii section alignement. Since with this commit, .data
> > alignement (0x1000) becomes larger than .text (0x800), it picks 0x1000 as
> > the output text offset, and then WriteSections64() complains that this
> > offset isn't equal to the input .text alignment modulo 0x1000.
> > 
> > The linker script says:
> > 
> >    /*
> >     * The alignment of the .data section should be less than or equal to the
> >     * alignment of the .text section. This ensures that the relative offset
> >     * between these sections is the same in the ELF and the PE/COFF versions of
> >     * this binary.
> >     */
> > 
> > but that's not what we're getting. I don't have a fix yet, other than
> > forcing the .text and .data alignment to 4k.
> > 
> 
> Jean, thanks for your explanation. Right, the issue is caused by mismatched
> alignments for ELF and PE/COFF sections. I ever dumped the variables at the
> failing point, showing the mismatched alignments (0x800 vs 0x1000). Apart from
> that, the virtual address of 'text' section is aligned to 0x800 instead of
> 0x1000 after ArmPlatformPrePeiCore.dll is dumped by 'readelf'.
> 
> SecHdr->sh_addr:                    0x800              <<< Mismatched alignment between ELF and PE/COFF
> SecOffset:                          0x1000
> SymShdr->sh_addr:                   0x800
> mCoffSectionsOffset[Sym->st_shndx]: 0x1000
> GenFw: ERROR 3000: Invalid
>   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> 
> # readelf -S Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
> Section Headers:
>   [Nr] Name              Type             Address           Offset
>        Size              EntSize          Flags  Link  Info  Align
>   [ 0]                   NULL             0000000000000000  00000000
>        0000000000000000  0000000000000000           0     0     0
>   [ 1] .text             PROGBITS         0000000000000800  00000800       <<< Aligned to 0x800
>        00000000000051b8  0000000000000000  AX       0     0     2048
> 
> With the following changes, I'm able to build the firmware successfully. I don't
> see how COMMONPAGESIZE is sorted out because I don't find its definition in the
> source code.

It's a ld builtin, set on the command-line with "-z common-page-size=X" by
Conf/tools_def.txt, in this case I believe DEBUG_GCC5_AARCH64_DLINK_XIPFLAGS. 

> 
> diff --git a/BaseTools/Scripts/GccBase.lds b/BaseTools/Scripts/GccBase.lds
> index 9f27e83bb0..5463df47a9 100644
> --- a/BaseTools/Scripts/GccBase.lds
> +++ b/BaseTools/Scripts/GccBase.lds
> @@ -20,7 +20,8 @@ SECTIONS {
>     */
>    . = PECOFF_HEADER_SIZE;
> -  .text : ALIGN(CONSTANT(COMMONPAGESIZE)) {
> +  /* .text : ALIGN(CONSTANT(COMMONPAGESIZE)) { */^M
> +  .text : ALIGN(4096) {^M

Build (after clean) fails for me if I only change the .text 
alignment, I need .data as well. So changing Conf/tools_def.txt is easier.
I'll try to find a proper fix but it will take me some time to understand
GenFw.


> > > > Note that the guest edk2 is optional and experimental, you can use direct
> > > > kernel boot to get a working demo quicker.
> > > > 
> > > 
> > > I never did this before. Could you please provide the detailed steps on this?
> > 
> > Removing the -bios parameter to QEMU should be enough. You can also add
> > 'earlycon' to -append to show early boot errors.
> > 
> 
> I didn't get a chance to try this yet since the host can't be brought up now.
> I will try this later. I originally thought some sort of boot wrapper is needed
> so that the kernel image has the capability to boot itself. For example, Mark
> Rutland's boot wrapper [1] can be leveraged in this case. I don't think Image has
> the capability to boot itself, right?

Yes QEMU can set up everything so that the Image boots on its own. What
the boot-wrapper does is minimal hardware initialization, handling PSCI
calls and passing the DTB pointer in x0. But that's only needed when using
the Arm FastModel (boot-wrapper is a lightweight firmware specifically for
the FastModel). QEMU can do all that itself so you can boot a kernel
without any firmware.

Using edk2 in the Realm guest will be needed for example to boot a distro
image which contains the kernel, but direct kernel boot is useful both for
prototyping and real-life use cases like confidential containers and some
cloud VMs.

Thanks,
Jean

> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git
> 
> Thanks,
> Gavin
>