Re: [RFC 00/11] hw/arm/smmuv3: Add initial support for Secure State

[Mostafa Saleh <smostafa@google.com>]

On Wed, Aug 06, 2025 at 11:11:23PM +0800, Tao Tang wrote:
> Hi all,
> 
> This patch series introduces initial support for emulating the Arm SMMUv3
> Secure State.
> 
> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3
> emulation is a notable missing piece in QEMU. While the FVP model has
> some support, its limited PCIe capabilities make it challenging for
> complex use cases. The ability to properly manage device DMA from a
> secure context is a critical prerequisite for enabling device assignment
> (passthrough) for confidential computing solutions like Arm CCA and
> related research such as virtCCA [2]. This series aims to build that
> foundational support in QEMU.
> 
> This work is being proposed as an RFC. It introduces a significant amount
> of new logic, including the core concept of modeling parallel secure and
> non-secure contexts within a single SMMUv3 device. I am seeking feedback
> on the overall approach, the core refactoring, and the implementation
> details before proceeding further.
> 
> The series begins by implementing the components of the secure programming
> interface, then progressively refactors the core SMMU logic to handle
> secure and non-secure contexts in parallel.
> 
> Secure Interface Implementation: The initial patches add the
> secure-side registers, implement their read/write logic, and enable
> the secure command and event queues. This includes the S_INIT
> mechanism and the new secure TLB invalidation commands.
> 
> Core Logic Refactoring: The next set of patches makes the core SMMU
> functions security-state aware. This involves plumbing an is_secure
> context flag through the main code paths and adding logic to route
> SMMU-originated memory accesses to the correct (Secure or Non-secure)
> address space.
> 
> Cache Isolation: With the core logic now aware of security states,
> the following patches refactor the configuration and translation
> lookup caches. The cache keys are modified to include the security
> context, ensuring that secure and non-secure entries for the same
> device or address are properly isolated and preventing aliasing.
> 
> Framework Integration: The final patch connects the SMMU's internal
> security context to the generic QEMU IOMMU framework by using the
> iommu_index to represent the architectural SEC_SID.
> 
> To validate this work, I performed the following tests:
> 
> Non-Secure Regression: To ensure that existing functionality remains
> intact, I ran a nested virtualization test. A TCG guest was created on
> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned.
> Command line of TCG VM is below:
> 
> qemu-system-aarch64 \
> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \
> -cpu max -smp 1 -m 4080M \
> -accel tcg,thread=single,tb-size=512 \
> -kernel Image \
> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \
> -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \
> -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \
> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \
> -device virtio-blk-device,drive=hd0 \
> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \
> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \
> -device virtio-net-device,netdev=eth0 \
> -drive if=none,file=nvme.img,format=raw,id=nvme0 \
> -device nvme,drive=nvme0,serial=deadbeef \
> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic
> 
> Inside this TCG VM, a KVM guest was launched, and the same NVMe device was
> re-assigned to it via VFIO.
> Command line of KVM VM inside TCG VM is below:
> 
> sudo qemu-system-aarch64  \
> -enable-kvm  -m 1024  -cpu host  -M virt \
> -machine virt,gic-version=3 \
> -cpu max -append "nokaslr" -smp 1 \
> -monitor stdio \
> -kernel 5.15.Image \
> -initrd rootfs.cpio.gz \
> -display vnc=:22,id=primary \
> -device vfio-pci,host=00:01.0
> 
> The KVM guest was able to perform I/O on the device
> correctly, confirming that the non-secure path is not broken.

I gave the patches a quick test and they seem to have broken my
nested setup, I will look more into it and let you know what I find.

Thanks,
Mostafa

> 
> Secure Register/Command Interface: I set up an OP-TEE + Hafnium
> environment. Hafnium's smmuv3_driver_init function was used to test
> the secure register I/O and command queue functionality (excluding
> translation). As Hafnium assumes larger queue and StreamID sizes than
> are practical without TTST support, I temporarily patched Hafnium to
> use smaller values, allowing its driver to initialize the emulated
> secure SMMU successfully.
> 
> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not
> support QEMU, and no secure device assignment feature exists yet, I
> created a custom platform device to test the secure translation flow.
> To trigger the translation logic, I initiated MMIO writes to this
> device from within Hafnium. The device's MMIO callback handler then
> performed DMA accesses via its IOMMU region, exercising the secure
> translation path. While SMMUv3 is typically used for PCIe on
> physical SoCs, the architecture allows its use with platform devices
> via a stream-id binding in the device tree. The test harness
> required some non-standard modifications to decouple the SMMU from
> its tight integration with PCIe. The code for this test device is
> available for review at [3]. README.md with detailed instructions is
> also provided.
> 
> I've attempted to follow all of the guidance in the "Submitting a Patch"
> guide, but as this is my first series of this scale, I apologize if I
> missed anything and welcome all feedback.
> 
> Thanks,
> Tang
> 
> [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html
> [2] https://arxiv.org/abs/2306.11011
> [3] https://github.com/hnusdr/qemu
> 
> Tao Tang (11):
>   hw/arm/smmuv3: Introduce secure registers and commands
>   hw/arm/smmuv3: Implement read/write logic for secure registers
>   hw/arm/smmuv3: Implement S_INIT for secure initialization
>   hw/arm/smmuv3: Enable command processing for the Secure state
>   hw/arm/smmuv3: Support secure event queue and error handling
>   hw/arm/smmuv3: Plumb security state through core functions
>   hw/arm/smmuv3: Add separate address space for secure SMMU accesses
>   hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations
>   hw/arm/smmuv3: Make the configuration cache security-state aware
>   hw/arm/smmuv3: Differentiate secure TLB entries via keying
>   hw/arm/smmuv3: Use iommu_index to represent SEC_SID
> 
>  hw/arm/smmu-common.c         |  74 ++-
>  hw/arm/smmuv3-internal.h     | 128 +++++-
>  hw/arm/smmuv3.c              | 844 ++++++++++++++++++++++++++++++-----
>  hw/arm/trace-events          |   7 +-
>  hw/arm/virt.c                |   5 +
>  include/hw/arm/smmu-common.h |  23 +-
>  include/hw/arm/smmuv3.h      |  27 ++
>  7 files changed, 968 insertions(+), 140 deletions(-)
> 
> --
> 2.34.1
> 
>