Re: [v2] Support for Arm CCA VMs on Linux

[Itaru Kitayama <itaru.kitayama@linux.dev>]

Hi Steven,

On Wed, Jun 26, 2024 at 02:39:27PM +0100, Steven Price wrote:
> On 24/06/2024 07:13, Itaru Kitayama wrote:
> > Hi Steven,
> > On Fri, Apr 12, 2024 at 09:40:56AM +0100, Steven Price wrote:
> >> We are happy to announce the second version of the Arm Confidential
> >> Compute Architecture (CCA) support for the Linux stack. The intention is
> >> to seek early feedback in the following areas:
> >>  * KVM integration of the Arm CCA;
> >>  * KVM UABI for managing the Realms, seeking to generalise the
> >>    operations where possible with other Confidential Compute solutions;
> >>  * Linux Guest support for Realms.
> >>
> >> See the previous RFC[1] for a more detailed overview of Arm's CCA
> >> solution, or visible the Arm CCA Landing page[2].
> >>
> >> This series is based on the final RMM v1.0 (EAC5) specification[3].
> >>
> >> Quick-start guide
> >> =================
> >>
> >> The easiest way of getting started with the stack is by using
> >> Shrinkwrap[4]. Currently Shrinkwrap has a configuration for the initial
> >> v1.0-EAC5 release[5], so the following overlay needs to be applied to
> >> the standard 'cca-3world.yaml' file. Note that the 'rmm' component needs
> >> updating to 'main' because there are fixes that are needed and are not
> >> yet in a tagged release. The following will create an overlay file and
> >> build a working environment:
> >>
> >> cat<<EOT >cca-v2.yaml
> >> build:
> >>   linux:
> >>     repo:
> >>       revision: cca-full/v2
> >>   kvmtool:
> >>     repo:
> >>       kvmtool:
> >>         revision: cca/v2
> >>   rmm:
> >>     repo:
> >>       revision: main
> >>   kvm-unit-tests:
> >>     repo:
> >>       revision: cca/v2
> >> EOT
> >>
> >> shrinkwrap build cca-3world.yaml --overlay buildroot.yaml --btvar GUEST_ROOTFS='${artifact:BUILDROOT}' --overlay cca-v2.yaml
> >>
> >> You will then want to modify the 'guest-disk.img' to include the files
> >> necessary for the realm guest (see the documentation in cca-3world.yaml
> >> for details of other options):
> >>
> >>   cd ~/.shrinkwrap/package/cca-3world
> >>   /sbin/e2fsck -fp rootfs.ext2 
> >>   /sbin/resize2fs rootfs.ext2 256M
> >>   mkdir mnt
> >>   sudo mount rootfs.ext2 mnt/
> >>   sudo mkdir mnt/cca
> >>   sudo cp guest-disk.img KVMTOOL_EFI.fd lkvm Image mnt/cca/
> >>   sudo umount mnt 
> >>   rmdir mnt/
> >>
> >> Finally you can run the FVP with the host:
> >>
> >>   shrinkwrap run cca-3world.yaml --rtvar ROOTFS=$HOME/.shrinkwrap/package/cca-3world/rootfs.ext2
> >>
> >> And once the host kernel has booted, login (user name 'root') and start
> >> a realm guest:
> >>
> >>   cd /cca
> >>   ./lkvm run --realm --restricted_mem -c 2 -m 256 -k Image -p earlycon
> >>
> >> Be patient and you should end up in a realm guest with the host's
> >> filesystem mounted via p9.
> >>
> >> It's also possible to use EFI within the realm guest, again see
> >> cca-3world.yaml within Shrinkwrap for more details.
> > 
> > I am trying to see if libvirt can work with the CCA-aware KVM with minimal Ubuntu22.10 filesystem, however virt-install triggers a system failure:
> > 
> > $ sudo virt-install -v --name f39 --ram 4096        --disk path=fedora40.img,cache=none --nographics --os-variant fedora38         --import --arch aarch64 --vcpus 4
> > [sudo] password for realm:
> > [ 3694.176579] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000e00
> > [ 3694.176687] Mem abort info:
> > [ 3694.176745]   ESR = 0x0000000096000004
> > [ 3694.176817]   EC = 0x25: DABT (current EL), IL = 32 bits
> > [ 3694.176907]   SET = 0, FnV = 0
> > [ 3694.176978]   EA = 0, S1PTW = 0
> > [ 3694.177049]   FSC = 0x04: level 0 translation fault
> > [ 3694.177132] Data abort info:
> > [ 3694.177189]   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
> > [ 3694.177276]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
> > [ 3694.177370]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> > [ 3694.177544] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000880f6e000
> > [ 3694.177649] [0000000000000e00] pgd=0000000000000000, p4d=0000000000000000
> > [ 3694.177788] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
> > [ 3694.177887] Modules linked in:
> > [ 3694.177966] CPU: 2 PID: 540 Comm: qemu-system-aar Not tainted 6.10.0-rc1-00058-gd901c27a1783 #149
> > [ 3694.178105] Hardware name: FVP Base RevC (DT)
> > [ 3694.178180] pstate: 61400009 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> > [ 3694.178315] pc : kvm_vm_ioctl_check_extension+0x1fc/0x3c4
> > [ 3694.178447] lr : kvm_vm_ioctl_check_extension_generic+0x34/0x12c
> > [ 3694.178587] sp : ffff800081523cb0
> > [ 3694.178657] x29: ffff800081523cb0 x28: 0000000000000051 x27: 0000000000000000
> > [ 3694.178840] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
> > [ 3694.179019] x23: 000000000000000a x22: 0000000000000051 x21: ffff000801075f00
> > [ 3694.179200] x20: ffff000801075f01 x19: 000000000000ae03 x18: 0000000000000000
> > [ 3694.179383] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> > [ 3694.179565] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> > [ 3694.179745] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
> > [ 3694.179923] x8 : 0000000000000000 x7 : ffff000801075f18 x6 : 00000000401c5820
> > [ 3694.180106] x5 : 000000000000000a x4 : 0000000000000800 x3 : 0000000000000000
> > [ 3694.180285] x2 : 000000000000000b x1 : 0000000100061001 x0 : 0000000000000001
> > [ 3694.180465] Call trace:
> > [ 3694.180523]  kvm_vm_ioctl_check_extension+0x1fc/0x3c4
> > [ 3694.180656]  kvm_vm_ioctl_check_extension_generic+0x34/0x12c
> > [ 3694.180794]  kvm_dev_ioctl+0x3c8/0x8b8
> > [ 3694.180938]  __arm64_sys_ioctl+0xac/0xf0
> > [ 3694.181079]  invoke_syscall+0x48/0x114
> > [ 3694.181220]  el0_svc_common.constprop.0+0x40/0xe0
> > [ 3694.181367]  do_el0_svc+0x1c/0x28
> > [ 3694.181507]  el0_svc+0x34/0xd8
> > [ 3694.181608]  el0t_64_sync_handler+0x120/0x12c
> > [ 3694.181723]  el0t_64_sync+0x190/0x194
> > [ 3694.181865] Code: 17ffffbd 97fffc9d 12001c00 17ffff91 (39780060)
> > [ 3694.181955] ---[ end trace 0000000000000000 ]---
> > 
> > I'd appreciate it if you could take a look at it.
> 
> Thanks for the bug report. I believe this is because 
> kvm_vm_ioctl_check_extension() is being called with kvm==NULL and I've 
> missed some checks. I believe the following should get things working - 
> and it's probably better than attempting to remember to check with the 
> NULL kvm at each call site.
> 
> ---8<----
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index 27c58bbdf50b..c85e5f566506 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -602,7 +602,7 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu)
>  
>  static inline bool kvm_is_realm(struct kvm *kvm)
>  {
> -       if (static_branch_unlikely(&kvm_rme_is_available))
> +       if (static_branch_unlikely(&kvm_rme_is_available) && kvm)
>                 return kvm->arch.is_realm;
>         return false;
>  }
> ---8<----

Sorry for my late reply I was away entire last week.

With the fix above, I was able to use the virt-install command on FVP without an issue.

Tested-by: Itaru Kitayama <itaru.kitayama@fujitsu.com>

Thanks,
Itaru.

> 
> Thanks,
> Steve