Re: Unexpected error in rme_configure_one() at ../target/arm/kvm-rme.c:159

[Gavin Shan <gshan@redhat.com>]

Hi Jean,

On 6/4/24 21:15, Jean-Philippe Brucker wrote:
> On Tue, Jun 04, 2024 at 01:02:08PM +1000, Gavin Shan wrote:
>> On 6/3/24 18:24, Jean-Philippe Brucker wrote:
>>> On Sat, Jun 01, 2024 at 08:14:46PM +1000, Gavin Shan wrote:
>>>> ---> guest edk2
>>>>
>>>> # git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
>>>> # cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
>>>> # git submodule update --init --recursive;  \
>>>>     source edksetup.sh; make -j -C BaseTools; \
>>>>     export GCC5_AARCH64_PREFIX=;              \
>>>
>>> Doesn't this needs a cross-compiler, something like "aarch64-linux-gnu-" ?
>>>
>>
>> No, I was building everything using a native compiler instead of a cross compiler.
>> All packages were compiled on a NVidia's grace-hopper machine.
>>
>> [root@nvidia-grace-hopper-05 ~]# cat /etc/system-release
>> Red Hat Enterprise Linux release 9.5 Beta (Plow)
>> [root@nvidia-grace-hopper-05 ~]# uname -r
>> 6.7.0-rc2-gavin+
>> [root@nvidia-grace-hopper-05 ~]# gcc --version
>> gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
>> Copyright (C) 2021 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> I tried the cross compiler and encountered the same build error.
>>
>> [root@nvidia-grace-hopper-05 edk2-guest]# export | grep GCC5_AARCH64_PREFIX
>> declare -x GCC5_AARCH64_PREFIX="aarch64-linux-gnu-"
>> [root@nvidia-grace-hopper-05 edk2-guest]# build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>    :
>> --add-gnu-debuglink=/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPrePeiCore.debug
>> "GenFw" -e SEC -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
>> GenFw: ERROR 3000: Invalid
>>    WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
> 
> Ah I've seen this once but it disappeared as I tried to investigate and
> I've since changed the implementation, so I don't have many notes about
> it.
> 
> Maybe you could try to bisect from "ArmVirtPkg: ArmCcaIoMmu: Provide an
> implementation for SetAttribute", but it may give false positives if the
> error depends on some random linker placement. Could be
> "ArmVirtPkg/ArmPlatformLibQemu: Setup early UART mapping in a Realm" which
> adds a 4k page to the data section for the ealy RSI config call, though
> that has explicit 4kB alignment.
> 
> In my notes I also wrote that changing "-z common-page-size=0x20" to 4k in
> the link flags may have made the error disappear, but I doubt it's the
> right fix.
> 
> I'll try GCC 11 to see if I can reproduce.
> 

Ok. I run a git-bisect and the first problematic commit is 1153ae939c
("ArmVirtPkg/ArmPlatformLibQemu: Add a third-level page table for the UART idmap")

I'm not familiar with edk2. The error is raised by BaseTools/Source/C/GenFw/Elf64Convert.c::WriteSections64()
where the relocatable address isn't properly aligned to 4KB. So I modified the code
as below, but I have to run two consecutive builds. In the first attempt build, I
still hit the same error.

---> VirtPkg/Library/ArmPlatformLibQemu/IdMap.S

   .align    12
   .globl    idmap
   .globl    uart_pte
   .section  ".data.idmap", "aw", %progbits
   .align    12

# source edksetup.sh; export GCC5_AARCH64_PREFIX=
# make -j -C BaseTools; \                                               <<< Failed on the first attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
make: *** [GNUmakefile:405: /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/OUTPUT/ArmPlatformPrePeiCore.efi] Error 2

# make -j -C BaseTools; \                                              <<< Succeed on the second attempt
   build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
    :
Generating FVMAIN FV
######
Fd File Name:QEMU_VARS (/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
    :
- Done -
Build end time: 21:04:05, Jun.04 2024
Build total time: 00:00:06

After that, I'm unable to start the guest with the edk2 image successfully.

host# # mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# cat ./realm.sh
#!/bin/sh

SHR_DIR="/mnt/shr0"

qemu-system-aarch64 -accel kvm                              \
-machine virt,gic-version=3,confidential-guest-support=rme0 \
-cpu host -smp 2 -m 512M                                    \
-object 'rme-guest,id=rme0,measurement-algo=sha512'         \
-monitor none -serial mon:stdio -nographic                  \
-bios ${SHR_DIR}/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd \
-kernel ${SHR_DIR}/linux/arch/arm64/boot/Image              \
-initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio      \
-append 'console=ttyAMA0'

host# ./realm.sh
UEFI firmware (version  built at 19:56:47 on Jun  4 2024)
add-symbol-file /home/gavin/sandbox/C                              <<< I don't see more output after it

>> GenFw: ERROR 3000: Invalid
>>    :
>>
>>>>     build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
>>>>      :
>>>>     WriteSections64(): /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll AARCH64 small code model requires identical ELF and PE/COFF section offsets modulo 4 KB.
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
>>>> "gcc" -MMD -MF /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps @/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt  -c -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj  /home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
>>>> "GenFw" -e DXE_CORE -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>>>> GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
>>>> cp -p -f /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi /home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
>>>> GenFw: ERROR 3000: Invalid
>>>>     :
>>>> build.py...
>>>>    : error 7000: Failed to execute command
>>>> 	make tbuild [/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
>>>>
>>>>
>>>> build.py...
>>>>    : error F002: Failed to build module
>>>> 	/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf [AARCH64, GCC5, DEBUG]
>>>>
>>>> - Failed -
>>>> Build end time: 05:42:19, Jun.01 2024
>>>> Build total time: 00:00:31
>>>>
> 
> 
>> Ok, I can look into this deeply after I can bring up the guest successfully.
> 
> Note that the guest edk2 is optional and experimental, you can use direct
> kernel boot to get a working demo quicker.
> 

I never did this before. Could you please provide the detailed steps on this?

Thanks,
Gavin