[RFC PATCH 00/21] Secure VFIO, TDISP, SEV TIO

Alexey Kardashevskiy
2024-08-23T13:22:29Z

Hi everyone,

Here are some patches to enable SEV-TIO (aka TDISP, aka secure VFIO)
on AMD Turin.

The basic idea is to allow DMA to/from encrypted memory of SNP VMs and
secure MMIO in SNP VMs (i.e. with Cbit set) as well.

These include both guest and host support. QEMU also requires
some patches, links below.

The patches are organized as:
01..06 - preparing the host OS;
07 - new TSM module;
08 - add PSP SEV TIO ABI (IDE should start working at this point);
09..14 - add KVM support (TDI binding, MMIO faulting, etc);
15..19 - guest changes (the rest of SEV TIO ABI, DMA, secure MMIO).
20, 21 - some helpers for guest OS to use encrypted MMIO

This is based on a merge of
ee3248f9f8d6 Lukas Wunner spdm: Allow control of next requester nonce
through sysfs
85ef1ac03941 (AMDESE/snp-host-latest) 4 days ago Michael Roth [TEMP] KVM: guest_memfd: Update gmem_prep are hook to handle partially-allocated folios

Please comment. Thanks.

Thanks,

SEV TIO tree prototype
======================

Goal
----

Support secure PCI devices pass through to confidential VMs.
The support is defined by PCIe 6, SPDM, TDISP (not AMD) and SEV TIO
specification (by AMD).

SEV TIO spec:
https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/specifications/58271_0_70.pdf
Whitepaper:
https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/sev-tio-whitepaper.pdf
GHCB spec update is coming.

The changeset adds:
- a generic TSM.ko module;
makes necessary changes to:
- ccp (interface to AMD secure platform processor, "PSP", works on the
host),
- sev-guest (interace to PSP for confidential SNP guest),
- kvm_amd,
- memfd,
- vfio kvm device,
- kvm-x86.

Acronyms
--------

TEE - Trusted Execution Environments, a concept of managing trust
between the host
and devices
TSM - TEE Security Manager (TSM), an entity which ensures
security on the host
PSP - AMD platform secure processor (also "ASP", "AMD-SP"), acts
as TSM on AMD.
SEV TIO - the TIO protocol implemented by the PSP and used by
the host
GHCB - guest/host communication block - a protocol for
guest-to-host communication
via a shared page

CMA, IDE, TDISP,
TVM, DSM, SPDM, DOE

Code
----

Written with AMD SEV SNP in mind, TSM is the PSP and
therefore no much of IDE/TDISP
is left for the host or guest OS.

Add a common module to expose various data objects in
the same way in host and
guest OS.

Provide a know on the host to enable IDE encryption.

Add another version of Guest Request for secure
guest<->PSP communication.

Enable secure DMA by:
- configuring vTOM in a secure DTE via the PSP to cover
the entire guest RAM;
- mapping all private memory pages in IOMMU just like
as they were shared
(requires hacking iommufd);
- skipping various enforcements of non-SME or
SWIOTLB in the guest;

No mixed share+private DMA supported within the
same IOMMU.

Enable secure MMIO by:
- configuring RMP entries via the PSP;
- adding necessary helpers for mapping MMIO with
the Cbit set;
- hacking the KVM #PF handler to allow private
MMIO failts.

Based on the latest upstream KVM (at the
moment it is kvm-coco-queue).

Workflow
--------

1. Boot host OS.
2. "Connect" the physical device.
3. Bind a VF to VFIO-PCI.
4. Run QEMU _without_ the device yet.
5. Hotplug the VF to the VM.
6. (if not already) Load the device driver.
7. Right after the BusMaster is enabled,
tsm.ko performs secure DMA and MMIO setup.
8. Run tests, for example:
sudo ./pcimem/pcimem
/sys/bus/pci/devices/0000\:01\:00.0/resource4_enc
0 w*4 0xabcd

Assumptions
-----------

This requires hotpligging into the VM vs
passing the device via the command line as
VFIO maps all guest memory as the device init
step which is too soon as
SNP LAUNCH UPDATE happens later and will fail
if VFIO maps private memory before that.

This requires the BME hack as MMIO and
BusMaster enable bits cannot be 0 after MMIO
validation is done and there are moments in
the guest OS booting process when this
appens.

SVSM could help addressing these (not
implemented at the moment).

QEMU advertises TEE-IO capability to the VM.
An additional x-tio flag is added to
vfio-pci.

TODOs
-----

Deal with PCI reset. Hot unplug+plug? Power
states too.

Do better generalization, the current code
heavily uses SEV TIO defined
structures in supposedly generic code.

Fix the documentation comments of SEV TIO structures.

Git trees
---------

https://github.com/AMDESE/linux-kvm/tree/tio
https://github.com/AMDESE/qemu/tree/tio

Alexey Kardashevskiy (21):
tsm-report: Rename module to reflect what it does
pci/doe: Define protocol types and make those public
pci: Define TEE-IO bit in PCIe device capabilities
PCI/IDE: Define Integrity and Data Encryption (IDE) extended
capability
crypto/ccp: Make some SEV helpers public
crypto: ccp: Enable SEV-TIO feature in the PSP when supported
pci/tdisp: Introduce tsm module
crypto/ccp: Implement SEV TIO firmware interface
kvm: Export kvm_vm_set_mem_attributes
vfio: Export helper to get vfio_device from fd
KVM: SEV: Add TIO VMGEXIT and bind TDI
KVM: IOMMUFD: MEMFD: Map private pages
KVM: X86: Handle private MMIO as shared
RFC: iommu/iommufd/amd: Add IOMMU_HWPT_TRUSTED flag, tweak DTE's
DomainID, IOTLB
coco/sev-guest: Allow multiple source files in the driver
coco/sev-guest: Make SEV-to-PSP request helpers public
coco/sev-guest: Implement the guest side of things
RFC: pci: Add BUS_NOTIFY_PCI_BUS_MASTER event
sev-guest: Stop changing encrypted page state for TDISP devices
pci: Allow encrypted MMIO mapping via sysfs
pci: Define pci_iomap_range_encrypted

drivers/crypto/ccp/Makefile | 2 +
drivers/pci/Makefile | 1 +
drivers/virt/coco/Makefile | 3 +-
drivers/virt/coco/sev-guest/Makefile | 1 +
arch/x86/include/asm/kvm-x86-ops.h | 2 +
arch/x86/include/asm/kvm_host.h | 2 +
arch/x86/include/asm/sev.h | 23 +
arch/x86/include/uapi/asm/svm.h | 2 +
arch/x86/kvm/svm/svm.h | 2 +
drivers/crypto/ccp/sev-dev-tio.h | 105 ++
drivers/crypto/ccp/sev-dev.h | 4 +
drivers/iommu/amd/amd_iommu_types.h | 2 +
drivers/iommu/iommufd/io_pagetable.h | 3 +
drivers/iommu/iommufd/iommufd_private.h | 4 +
drivers/virt/coco/sev-guest/sev-guest.h | 56 +
include/asm-generic/pci_iomap.h | 4 +
include/linux/device.h | 5 +
include/linux/device/bus.h | 3 +
include/linux/dma-direct.h | 4 +
include/linux/iommufd.h | 6 +
include/linux/kvm_host.h | 70 +
include/linux/pci-doe.h | 4 +
include/linux/pci-ide.h | 18 +
include/linux/pci.h | 2 +-
include/linux/psp-sev.h | 116 +-
include/linux/swiotlb.h | 4 +
include/linux/tsm-report.h | 113 ++
include/linux/tsm.h | 337 +++--
include/linux/vfio.h | 1 +
include/uapi/linux/iommufd.h | 1 +
include/uapi/linux/kvm.h | 29 +
include/uapi/linux/pci_regs.h | 77 +-
include/uapi/linux/psp-sev.h | 4 +-
arch/x86/coco/sev/core.c | 11 +
arch/x86/kvm/mmu/mmu.c | 6 +-
arch/x86/kvm/svm/sev.c | 217 +++
arch/x86/kvm/svm/svm.c | 3 +
arch/x86/kvm/x86.c | 12 +
arch/x86/mm/mem_encrypt.c | 5 +
arch/x86/virt/svm/sev.c | 23 +-
drivers/crypto/ccp/sev-dev-tio.c | 1565 ++++++++++++++++++++
drivers/crypto/ccp/sev-dev-tsm.c | 397 +++++
drivers/crypto/ccp/sev-dev.c | 87 +-
drivers/iommu/amd/iommu.c | 20 +-
drivers/iommu/iommufd/hw_pagetable.c | 4 +
drivers/iommu/iommufd/io_pagetable.c | 2 +
drivers/iommu/iommufd/main.c | 21 +
drivers/iommu/iommufd/pages.c | 94 +-
drivers/pci/doe.c | 2 -
drivers/pci/ide.c | 186 +++
drivers/pci/iomap.c | 24 +
drivers/pci/mmap.c | 11 +-
drivers/pci/pci-sysfs.c | 27 +-
drivers/pci/pci.c | 3 +
drivers/pci/proc.c | 2 +-
drivers/vfio/vfio_main.c | 13 +
drivers/virt/coco/sev-guest/{sev-guest.c => sev_guest.c} | 68 +-
drivers/virt/coco/sev-guest/sev_guest_tio.c | 513 +++++++
drivers/virt/coco/tdx-guest/tdx-guest.c | 8 +-
drivers/virt/coco/tsm-report.c | 512 +++++++
drivers/virt/coco/tsm.c | 1542 ++++++++++++++-----
virt/kvm/guest_memfd.c | 40 +
virt/kvm/kvm_main.c | 4 +-
virt/kvm/vfio.c | 197 ++-
Documentation/virt/coco/tsm.rst | 62 +
MAINTAINERS | 4 +-
arch/x86/kvm/Kconfig | 1 +
drivers/pci/Kconfig | 4 +
drivers/virt/coco/Kconfig | 11 +
69 files changed, 6163 insertions(+), 548 deletions(-)
create mode 100644 drivers/crypto/ccp/sev-dev-tio.h
create mode 100644 drivers/virt/coco/sev-guest/sev-guest.h
create mode 100644 include/linux/pci-ide.h
create mode 100644 include/linux/tsm-report.h
create mode 100644 drivers/crypto/ccp/sev-dev-tio.c
create mode 100644 drivers/crypto/ccp/sev-dev-tsm.c
create mode 100644 drivers/pci/ide.c
rename drivers/virt/coco/sev-guest/{sev-guest.c => sev_guest.c} (96%)
create mode 100644 drivers/virt/coco/sev-guest/sev_guest_tio.c
create mode 100644 drivers/virt/coco/tsm-report.c
create mode 100644 Documentation/virt/coco/tsm.rst

--
2.45.2