[PATCH v3 00/30] KVM: s390: enable zPCI for interpretive execution

Matthew Rosato
2022-02-04T21:15:53Z

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci. This is done by introducing a series
of new vfio-pci feature ioctls that are unique vfio-pci-zdev (s390x) and
are used to negotiate the various aspects of zPCI interpretation setup.
By allowing intepretation of zPCI instructions and firmware delivery of
interrupts to guests, we can significantly reduce the frequency of guest
SIE exits for zPCI. We then see additional gains by handling a hot-path
instruction that can still intercept to the hypervisor (RPCIT) directly
in kvm.

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by
default if available in kernel+qemu.

Will reply with a link to the associated QEMU series.

Changelog v2->v3:
- More R-bs / ACKs (Thanks!)
- Re-word patch 6 commit message (Claudio)
- Patch 8 + some later patches: s/gd/gisa/ (Pierre)
- Patch 12: remove !mdd check (Pierre)
- some more virt/phys conversions (Pierre)
- Patch 18: check more sclp bits & facilities during interp probe (Pierre)
- Patch 21: fix fabricated status for some RPCIT intercept errors
- Patch 25-27: remove get/set checks from feature ioctl handlers as they
are already done in vfio core (Pierre)
- Patch 26: s/aif/aif_float/ and s/fhost/aif_fhost/
- remove kvm_s390_pci_attach_kvm and just do the work inline (Pierre)
- Use CONFIG_VFIO_PCI_ZDEV instead of CONFIG_PCI in Makefile and other
code locations (Pierre)
- Due to the above, re-arrange series order so CONFIG_VFIO_PCI_ZDEV is
introduced earlier
- s/aift->lock/aift->aift_lock/ (Pierre)
- Break some AEN init code into local functions zpci_setup_aipb() and
zpci_reset_aipb() (Pierre)
- check for errors on kvm_s390_gisc_register (Pierre)
- handle airq clear errors differently when we know the device is being
removed vs any other reason aif is being disabled (Pierre)
- s/ioat->lock/ioat->ioat_lock/ (Pierre)
- Fix backout case in kvm_s390_pci_ioat_enable, re-arrange rc settings
slightly (Pierre)
- Add a CONFIG_VFIO_PCI_ZDEV check when determining if its safe to allow
KVM_S390_VM_CPU_FEAT_ZPCI_INTERP (need both the facilities and the
kvm/pci.o pieces to allow intepretation)

Matthew Rosato (30):
s390/sclp: detect the zPCI load/store interpretation facility
s390/sclp: detect the AISII facility
s390/sclp: detect the AENI facility
s390/sclp: detect the AISI facility
s390/airq: pass more TPI info to airq handlers
s390/airq: allow for airq structure that uses an input vector
s390/pci: externalize the SIC operation controls and routine
s390/pci: stash associated GISA designation
s390/pci: export some routines related to RPCIT processing
s390/pci: stash dtsm and maxstbl
s390/pci: add helper function to find device by handle
s390/pci: get SHM information from list pci
s390/pci: return status from zpci_refresh_trans
vfio/pci: re-introduce CONFIG_VFIO_PCI_ZDEV
KVM: s390: pci: add basic kvm_zdev structure
KVM: s390: pci: do initial setup for AEN interpretation
KVM: s390: pci: enable host forwarding of Adapter Event Notifications
KVM: s390: mechanism to enable guest zPCI Interpretation
KVM: s390: pci: provide routines for enabling/disabling interpretation
KVM: s390: pci: provide routines for enabling/disabling interrupt
forwarding
KVM: s390: pci: provide routines for enabling/disabling IOAT assist
KVM: s390: pci: handle refresh of PCI translations
KVM: s390: intercept the rpcit instruction
vfio-pci/zdev: wire up group notifier
vfio-pci/zdev: wire up zPCI interpretive execution support
vfio-pci/zdev: wire up zPCI adapter interrupt forwarding support
vfio-pci/zdev: wire up zPCI IOAT assist support
vfio-pci/zdev: add DTSM to clp group capability
KVM: s390: introduce CPU feature for zPCI Interpretation
MAINTAINERS: additional files related kvm s390 pci passthrough

MAINTAINERS | 2 +
arch/s390/include/asm/airq.h | 7 +-
arch/s390/include/asm/kvm_host.h | 5 +
arch/s390/include/asm/kvm_pci.h | 60 +++
arch/s390/include/asm/pci.h | 12 +
arch/s390/include/asm/pci_clp.h | 11 +-
arch/s390/include/asm/pci_dma.h | 3 +
arch/s390/include/asm/pci_insn.h | 31 +-
arch/s390/include/asm/sclp.h | 4 +
arch/s390/include/asm/tpi.h | 13 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/Makefile | 1 +
arch/s390/kvm/interrupt.c | 95 +++-
arch/s390/kvm/kvm-s390.c | 57 ++-
arch/s390/kvm/kvm-s390.h | 10 +
arch/s390/kvm/pci.c | 850 +++++++++++++++++++++++++++++++
arch/s390/kvm/pci.h | 60 +++
arch/s390/kvm/priv.c | 49 ++
arch/s390/pci/pci.c | 31 ++
arch/s390/pci/pci_clp.c | 28 +-
arch/s390/pci/pci_dma.c | 7 +-
arch/s390/pci/pci_insn.c | 15 +-
arch/s390/pci/pci_irq.c | 48 +-
drivers/iommu/s390-iommu.c | 4 +-
drivers/s390/char/sclp_early.c | 4 +
drivers/s390/cio/airq.c | 12 +-
drivers/s390/cio/qdio_thinint.c | 6 +-
drivers/s390/crypto/ap_bus.c | 9 +-
drivers/s390/virtio/virtio_ccw.c | 6 +-
drivers/vfio/pci/Kconfig | 11 +
drivers/vfio/pci/Makefile | 2 +-
drivers/vfio/pci/vfio_pci_core.c | 8 +
drivers/vfio/pci/vfio_pci_zdev.c | 275 +++++++++-
include/linux/vfio_pci_core.h | 42 +-
include/uapi/linux/vfio.h | 22 +
include/uapi/linux/vfio_zdev.h | 51 ++
36 files changed, 1787 insertions(+), 65 deletions(-)
create mode 100644 arch/s390/include/asm/kvm_pci.h
create mode 100644 arch/s390/kvm/pci.c
create mode 100644 arch/s390/kvm/pci.h

--
2.27.0