Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan <gshan@redhat.com>]

Hi Jean and Ard,

On 6/1/24 01:09, Jean-Philippe Brucker wrote:
> On Fri, May 31, 2024 at 04:23:13PM +1000, Gavin Shan wrote:
>> I got a chance to try CCA software components, suggested by [1]. However, the edk2
>> is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm replying
>> to see if anyone has a idea.
> ...
>> INFO:    BL31: Preparing for EL3 exit to normal world
>> INFO:    Entry point address = 0x60000000
>> INFO:    SPSR = 0x3c9
>> UEFI firmware (version  built at 01:31:23 on May 31 2024)
>>
>> The boot is stuck and no more output after that. I tried adding more verbose output
>> from edk2 and found it's stuck at the following point.
>>
>>
>> ArmVirtPkg/PrePi/PrePi.c::PrePiMain
>> rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim
>>
>>   #ifdef MDE_CPU_AARCH64
>>    //
>>    // Set the SMCCC conduit to SMC if executing at EL2, which is typically the
>>    // exception level that services HVCs rather than the one that invokes them.
>>    //
>>    if (ArmReadCurrentEL () == AARCH64_EL2) {
>>      Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE);       // The function is never returned in my case
>>      ASSERT_EFI_ERROR (Status);
>>    }
>>   #endif
> 
> I'm able to reproduce this even without RME. This code was introduced
> recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
> conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
> 
> A slightly reduced reproducer:
> 
> $ cd edk2/
> $ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
> $ cd ..
> 
> $ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
> $ cd tf-a/
> $ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40 QEMU_USE_GIC_DRIVER=QEMU_GICV3 BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip && \
>    dd if=build/qemu/debug/bl1.bin of=flash.bin && \
>    dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
> $ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
> 

Thanks for the hints. Eventually, I'm able to start the host with 'edk2-stable202402'.
Note that 'edk2-stable202405' doesn't work. However, I failed to build the edk2 for
guest and unable to start the guest successfully, more information is provided below.

--> host's edk2

# git clone git@github.com:gwshan/edk2.git edk2
# cd edk2; git checkout edk2-stable202402 -b stable202402
# git submodule update --init --recursive;      \
   source edksetup.sh; make -j -C BaseTools;     \
   export GCC5_AARCH64_PREFIX=;                  \
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc

---> tf-a: rebuild using commands as you suggested.

---> Boot host

/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64                                           \
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off                                        \
-cpu max,x-rme=on -m 8G -smp 8                                                                    \
-nodefaults -monitor none -serial mon:stdio -nographic                                            \
-bios /home/gavin/sandbox/CCA/tf-a/flash.bin                                                      \
-kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image                                       \
-drive format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0 \
-device virtio-blk-pci,drive=hd0                                                                  \
-netdev tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown                 \
-device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0                                          \
-append root=/dev/vda console=ttyAMA0                                                             \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0                                                \
-fsdev local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0
   :
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE:  BL1: Built : 00:31:35, May 31 2024
INFO:    BL1: RAM 0xe0ee000 - 0xe0f7000
INFO:    BL1: Loading BL2
   :
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: May 31 2024 00:21:59
INFO:    RMM init end.
   :
UEFI firmware (version  built at 04:07:42 on Jun  1 2024)
PlatformPeim: PL011 UART (console) @ 0x9000000
PlatformPeim: PL011 UART (debug) @ 0x9000000
   :
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
   :
Welcome to Buildroot
buildroot login:
# ifconfig eth0 | grep 'inet addr'
           inet addr:10.26.1.212  Bcast:10.26.1.255  Mask:255.255.255.0

---> guest edk2

# git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
# cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
# git submodule update --init --recursive;  \
   source edksetup.sh; make -j -C BaseTools; \
   export GCC5_AARCH64_PREFIX=;              \
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
   WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
"gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
"GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
GenFw: ERROR 3000: Invalid
   :
build.py...
  : error 7000: Failed to execute command
	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]


build.py...
  : error F002: Failed to build module
	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]

- Failed -
Build end time: 05:42:19, Jun.01 2024
Build total time: 00:00:31

---> Use the edk2 image from the latest QEMU source

# cd /home/gavin/sandbox/CCA
# cp /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd ./

---> Start the guest and no output from the console

host# mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# export SHR_DIR="/mnt/shr0"
host# qemu-system-aarch64 -accel kvm                              \
       -machine virt,gic-version=3,confidential-guest-support=rme0 \
       -cpu host -smp 2 -m 512M                                    \
       -object 'rme-guest,id=rme0,measurement-algo=sha512'         \
       -monitor none -serial mon:stdio -nographic                  \
       -bios /mnt/edk2-aarch64-code.fd                             \
       -kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
       -initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
       -append 'console=ttyAMA0'
         :
       <no output from the console>
         :
       (QEMU) q

There are some messages from host's console indicating RMI/RMM servicing
states when the guest is running at background. After the guest is terminated,
the host crashes.

SMC_RMM_RTT_CREATE            102dff000 122c2e000 1e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 1234a7000 2000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 1235bd000 2200000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 12387c000 2400000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123a5a000 2600000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 12407d000 2800000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124109000 2a00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123e49000 2c00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124275000 2e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 123138000 3000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE            102dff000 124d07000 3200000 3 > RMI_SUCCESS
  :
  :
[22768.994481] rcu: INFO: rcu_preempt self-detected stall on CPU
[22769.006861] rcu: 	3-....: (2751 ticks this GP) idle=93ec/1/0x4000000000000000 softirq=114451/115721 fqs=1160
[22769.020475] rcu: 	(t=5257 jiffies g=531913 q=7 ncpus=8)
[22769.030547] CPU: 3 PID: 198 Comm: qemu-system-aar Not tainted 6.9.0-rc1-gavin-gfcfc92d6ff07 #1
[22769.041847] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[22769.050548] pstate: 60402009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[22769.059382] pc : kvm_realm_unmap_range+0x308/0x32c
[22769.070275] lr : kvm_realm_unmap_range+0x304/0x32c
[22769.075893] sp : ffff800080a3b930
[22769.079929] x29: ffff800080a3b930 x28: 00000000003d7000 x27: 00000000003d6000
[22769.092990] x26: 00000000c4000152 x25: ffffffffffffffff x24: 0000000000000000
[22769.101150] x23: 0000010000000000 x22: 00000000c4000155 x21: 0000000102dff000
[22769.109056] x20: ffff8000801a5e00 x19: 0000000000000000 x18: 0000000000000001
[22769.117042] x17: 0000000000000000 x16: 000000000000000e x15: 0000000000000000
[22769.124991] x14: 0000ffff7fa14000 x13: 0000000000000002 x12: 000000000010d594
[22769.134213] x11: 0000000000000002 x10: 00000000ffffffff x9 : ffffffffffffffff
[22769.142951] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 00000000c2dff000
[22769.151413] x5 : 0000000102f56000 x4 : 0000000000000015 x3 : 0000000000000000
[22769.159932] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[22769.169674] Call trace:
[22769.174026]  kvm_realm_unmap_range+0x308/0x32c
[22769.181046]  __unmap_stage2_range+0x60/0x7c
[22769.186396]  kvm_free_stage2_pgd+0xa0/0xd4
[22769.191766]  kvm_arch_flush_shadow_all+0x1c/0x34
[22769.197879]  kvm_mmu_notifier_release+0x30/0x84
[22769.203304]  __mmu_notifier_release+0x7c/0x1f8
[22769.209340]  exit_mmap+0x264/0x274
[22769.213992]  __mmput+0x40/0x150
[22769.218635]  mmput+0x50/0x5c
[22769.222606]  do_exit+0x288/0x92c
[22769.226935]  do_group_exit+0x34/0x90
[22769.231359]  get_signal+0x814/0x820
[22769.236537]  do_signal+0x90/0x1320
[22769.241145]  do_notify_resume+0xc8/0x140
[22769.246458]  el0_svc+0xc8/0xdc
[22769.250913]  el0t_64_sync_handler+0x13c/0x158
[22769.256045]  el0t_64_sync+0x190/0x194

Thanks,
Gavin