Re: [PATCH v17 0/7] firmware: arm_rmm: Add RMM v2.0 base RMI support

[Kohei Enju <enju.kohei@fujitsu.com>]

Hi Suzuki,

On 09/07 10:59, Suzuki K Poulose wrote:
> This series adds the generic firmware layer for talking to the Realm
> Management Monitor (RMM), as specified by the RMM v2.0-bet3
> specification[1]. It is the first part of the Arm CCA host support that
> was previously posted as part of the larger KVM series.
> 
> The split allows this RMM support to be used as a base for other work,
> including Aneesh's PCI IDE support with Arm CCA RMM as the TSM,
>  without depending on the KVM Realm support that will follow as separate
>  series. (See more on that below)
> 
> The series adds:
> 
>  * The RMI SMC definitions and direct-call wrappers.
> 
>  * RMM discovery and version checks during firmware init.
> 
>  * RMM host configuration, including the host page size.
> 
>  * Stateful RMI Operation (SRO) infrastructure for commands which the RMM
>    can complete across multiple SMC calls while requesting or returning
>    memory to the host.
> 
>  * Verification that granule tracking is available at fine granularity.
>    Fine-grained tracking allows each granule in the system to be tracked
>    independently, which is required before individual granules can be
>    delegated. A future series will add support for dynamically supplying
>    memory to the RMM for this tracking.
> 
>  * Support for fully firmware-managed systems, where the Granule Protection
>    Tables for memory regions are allocated and managed by firmware. RMM v2.0
>    also allows dynamic GPT creation on demand; support for that will be added
>    in a later series.
> 
>  * Wrappers for the RMI commands that are used for managing the "Realm VM"
>    lifecycle. This is added in to make it easier for the on-going KVM support
>    to evolve in parallel pieces.
> 
> If the platform firmware cannot manage the granule tracking or the GPTs, we
> bail out and deactivate the RMM, reclaiming any memory that we have donated.
> 
> The RMM v2.0 spec introduces Stateful RMI Operations (SROs), which allow
> the RMM to complete an operation over several SMC calls while requesting
> or returning memory to the host. This allows interrupts to be handled in
> the middle of an operation and lets the RMM dynamically allocate memory
> for internal tracking purposes. For example, RMI_REC_CREATE no longer
> needs auxiliary granules to be provided up front, and can instead
> request memory during the operation.
> 
> This series applies on v7.3-rc1 and a branch is available at [2]. The KVM
> CCA support that builds on this series is available at [3] as an integration
> branch. The KVM support depends on guest-memfd-in-place conversion support v12
> from Ackerley [4], we plan to split that into parts, which apply cleanly on
> v7.3-rcx without any dependency and is in progress. This will be made available
> as soon as it is ready. Until then [3] shows how this base series enables KVM
> CCA support. You may find the tf-RMM [5] and kvmtool support [6] below.
> 
> [1] RMM spec : https://support.arm.com/documentation/den0137/2-0bet3/
> [2] This series: https://gitlab.arm.com/linux-arm/linux-cca.git cca/cca-host/fw_rmm/v17
> [3] KVM CCA v17 integration branch: https://gitlab.arm.com/linux-arm/linux-cca.git cca/cca-host/kvm-integration/v17
> [4] Gmem inplace conversion https://github.com/googleprodkernel/linux-cc/tree/guest_memfd-inplace-conversion-v12
> [5] TF-RMM https://git.trustedfirmware.org/TF-RMM/tf-rmm.git main (commit: 134266ae)

I understand that QEMU-virt is not intended to model a realistic
production platform and that QEMU-SBSA is generally preferred. However,
I sometimes use QEMU-virt to test functionality quickly because it runs
slightly faster than QEMU-SBSA.

I tested this series with QEMU-virt and found that RMM initialization
failed during boot:

  [...]
  INFO:    BL31: Initializing RMM
  INFO:    RMM init start.
  ERROR:   RMM init failed: -8
  WARNING: BL31: RMM initialization failed

The following TF-A revision was used:
  https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git master (38269bb)

After investigating, I found that the failure was caused by the lack of
a plat_rmmd_reserve_memory() implementation for the QEMU-virt platform
in TF-A. Adding an implementation equivalent to the QEMU-SBSA one
resolved the issue:
    https://git.trustedfirmware.org/plugins/gitiles/TF-A/trusted-firmware-a.git/+/9d594743aa4d4e8160cdf025ea7538041648972c

Is QEMU virt still intended to be supported as an RME emulation
platform? In particular, is this functionality simply not implemented
yet, or is there a reason why RME support is not planned for QEMU virt?

If this is not the appropriate place to discuss this, please let me
know :)

Thanks,
Kohei

> [6] kvmtool https://gitlab.arm.com/linux-arm/kvmtool-cca.git cca/kvm-v17
> 
> Known issues: RMMv2.0
>  * RmiOpMemDonateReq:count (Uint14) is incompatible with RmiAddrRangeDesc4KB
>   (Uint10) and RmiAddrRangeDesc16KB (Uint12). i.e., a larger contiguous request
>   may not be satisfiable by the host. Arm is aware of this defect and a spec fix
>   is in progress.
> 
> 
> Steven Price (7):
>   firmware: arm_rmm: Add SMC definitions for calling the RMM
>   firmware: arm_rmm: Check for RMI support at init
>   firmware: arm_rmm: Configure the RMM with the host's page size
>   firmware: arm_rmm: Add support for SRO
>   firmware: arm_rmm: Activate the RMM
>   firmware: arm_rmm: Ensure the RMM has GPT entries for memory
>   firmware: arm_rmm: Add wrappers for Realm related RMI commands
> 
>  arch/arm64/Kconfig                |   1 +
>  arch/arm64/kernel/cpufeature.c    |   1 +
>  drivers/firmware/Kconfig          |   1 +
>  drivers/firmware/Makefile         |   1 +
>  drivers/firmware/arm_rmm/Kconfig  |  26 +
>  drivers/firmware/arm_rmm/Makefile |   2 +
>  drivers/firmware/arm_rmm/rmi.c    | 819 ++++++++++++++++++++++++++++++
>  include/linux/arm-rmi-cmds.h      | 678 +++++++++++++++++++++++++
>  include/linux/arm-smccc-rmi.h     | 494 ++++++++++++++++++
>  9 files changed, 2023 insertions(+)
>  create mode 100644 drivers/firmware/arm_rmm/Kconfig
>  create mode 100644 drivers/firmware/arm_rmm/Makefile
>  create mode 100644 drivers/firmware/arm_rmm/rmi.c
>  create mode 100644 include/linux/arm-rmi-cmds.h
>  create mode 100644 include/linux/arm-smccc-rmi.h
> 
> -- 
> 2.43.0
>