Re: [Qemu-devel] [PATCH v8 00/19] KVM platform device passthrough

[Eric Auger <eric.auger@linaro.org>]

On 12/08/2014 06:04 PM, Alex Williamson wrote:
> On Sun, 2014-11-30 at 18:35 +0000, Eric Auger wrote:
>> This RFC series aims at enabling KVM platform device passthrough.
>> It implements a VFIO platform device, derived from VFIO PCI device.
>>
>> The VFIO platform device uses the host VFIO platform driver which must
>> be bound to the assigned device prior to the QEMU system start.
>>
>> - the guest can directly access the device register space
>> - assigned device IRQs are transparently routed to the guest by
>>   QEMU/KVM (3 methods currently are supported: user-level eventfd
>>   handling, irqfd, forwarded IRQs)
>> - iommu is transparently programmed to prevent the device from
>>   accessing physical pages outside of the guest address space
>>
>> This patch series is made of the following patch file groups:
>>
>> 1-11) PCI modifications to prepare for platform device introduction
> 
> Hi Eric,
> 
> Sorry for the delay, I've only been able to make a serious review of
> 1-11 and it looks good.  I think we should start trying to pull these in
> after QEMU2.2.

That's a great news!

  Looking beyond patch 11/19, I'm surprised that the very
> first next patch isn't a linux header sync so that the platform
> populate_device routine can test the device info to make sure it's
> operating on a vfio platform device.

Yes you're fully right. Currently I do not test it and it's a mistake. I
will add this linux header sync.

Thanks

Eric

  Thanks,
> 
> Alex
> 
>> 12-15) VFIO & calxeda midway platform device without irqfd support
>> 16) VFIO platform device with irqfd support
>> 17-19) VFIO platform device with IRQ forwarding support
>>
>> Each group is independent and should be separately upstreamable.
>>
>> Dependency List:
>>
>> QEMU dependencies:
>> [1] [PATCH v5] machvirt dynamic sysbus device instantiation
>>     Eric Auger
>> [2] [PATCH v3 0/2] actual checks of KVM_CAP_IRQFD and KVM_CAP_IRQFD_RESAMPLE,
>>     Eric Auger
>>     http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg00589.html
>> [3] [PATCH v2] vfio: migration to trace points
>>     Eric Auger
>>     https://patchwork.ozlabs.org/patch/394785/
>>
>> Kernel Dependencies:
>> [1] [PATCH v10 00/20] VFIO support for platform and AMBA devices on ARM
>>     Antonios Motakis
>>     http://comments.gmane.org/gmane.linux.kernel.iommu/7096
>> [2] [PATCH v3 0/6] vfio: type1: support for ARM SMMUS with VFIO_IOMMU_TYPE1
>>     Antonios Motakis
>>     http://www.spinics.net/lists/kvm-arm/msg11738.html
>> [3] [PATCH v4] ARM: KVM: add irqfd support
>>     Eric Auger
>>     https://lkml.org/lkml/2014/9/1/141
>> [4] [RFC PATCH 0/9] ARM: Forwarding physical interrupts to a guest VM,
>>     Marc Zyngier
>>     http://lwn.net/Articles/603514/
>> [5] [PATCH v3 0/9] KVM-VFIO IRQ forward control
>>     Eric Auger
>>     https://lkml.org/lkml/2014/9/1/344
>>
>> - kernel pieces can be found at:
>>   http://git.linaro.org/people/eric.auger/linux.git (branch 3.18-rc6-v10)
>> - QEMU pieces can be found at:
>>   http://git.linaro.org/people/eric.auger/qemu.git (branch vfio_integ_v8)
>>
>> The patch series was tested on Calxeda Midway (ARMv7) where one xgmac
>> is assigned to KVM host while the second one is assigned to the guest.
>> Reworked PCI device is not tested.
>>
>> Wiki for Calxeda Midway setup:
>> https://wiki.linaro.org/LEG/Engineering/Virtualization/Platform_Device_Passthrough_on_Midway
>>
>> History:
>> v7->v8:
>> - rebase on v2.2.0-rc3 and integrate
>>   "Add skip_dump flag to ignore memory region during dump"
>> - KVM header evolution with subindex addition in kvm_arch_forwarded_irq
>> - split [PATCH v7 03/16] hw/vfio/pci: introduce VFIODevice into 4 patches
>> - vfio_compute_needs_reset does not return bool anymore
>> - add some comments about exposed MMIO region and IRQ in calxeda xgmac
>>   device
>> - vfio_[un]mask_irqindex renamed into vfio_[un]mask_single_irqindex
>> - rework IRQ startup: former machine init done notifier is replaced by a
>>   reset notifier. machine file passes the interrupt controller
>>   DeviceState handle (not the platform bus first irq parameter).
>> - sysbus-fdt:
>>   - move the add_fdt_node_functions array declaration between the device
>>     specific code and the generic code to avoid forward declarations of
>>     decice specific functions
>>   - rename add_basic_vfio_fdt_node into add_calxeda_midway_xgmac_fdt_node
>>     emphasizing the fact it is xgmac specific
>>
>> v6->v7:
>> - fake injection test modality removed
>> - VFIO_DEVICE_TYPE_PLATFORM only introduced with VFIO platform
>> - new helper functions to start VFIO IRQ on machine init done notifier
>>   (introduced in hw/vfio/platform: add vfio-platform support and notifier
>>   registration invoked in hw/arm/virt: add support for VFIO devices).
>>   vfio_start_irq_injection is replaced by vfio_register_irq_starter.
>>
>> v5->v6:
>> - rebase on 2.1rc5 PCI code
>> - forwarded IRQ first integraton
>> - vfio_device property renamed into host property
>> - split IRQ setup in different functions that match the 3 supported
>>   injection techniques (user handled eventfd, irqfd, forwarded IRQ):
>>   removes dynamic switch between injection methods
>> - introduce fake interrupts as a test modality:
>>   x makes possible to test multiple IRQ user-side handling.
>>   x this is a test feature only: enable to trigger a fd as if the
>>     real physical IRQ hit. No virtual IRQ is injected into the guest
>>     but handling is simulated so that the state machine can be tested
>> - user handled eventfd:
>>   x add mutex to protect IRQ state & list manipulation,
>>   x correct misleading comment in vfio_intp_interrupt.
>>   x Fix bugs using fake interrupt modality
>> - irqfd no more advertised in this patchset (handled in [3])
>> - VFIOPlatformDeviceClass becomes abstract and Calxeda xgmac device
>>   and class is re-introduced (as per v4)
>> - all DPRINTF removed in platform and replaced by trace-points
>> - corrects compilation with configure --disable-kvm
>> - simplifies the split for vfio_get_device and introduce a unique
>>   specialized function named vfio_populate_device
>> - group_list renamed into vfio_group_list
>> - hw/arm/dyn_sysbus_devtree.c currently only support vfio-calxeda-xgmac
>>   instantiation. Needs to be specialized for other VFIO devices
>> - fix 2 bugs in dyn_sysbus_devtree(reg_attr index and compat)
>>
>> v4->v5:
>> - rebase on v2.1.0 PCI code
>> - take into account Alex Williamson comments on PCI code rework
>>   - trace updates in vfio_region_write/read
>>   - remove fd from VFIORegion
>>   - get/put ckeanup
>> - bug fix: bar region's vbasedev field duly initialization
>> - misc cleanups in platform device
>> - device tree node generation removed from device and handled in
>>   hw/arm/dyn_sysbus_devtree.c
>> - remove "hw/vfio: add an example calxeda_xgmac": with removal of
>>   device tree node generation we do not have so many things to
>>   implement in that derived device yet. May be re-introduced later
>>   on if needed typically for reset/migration.
>> - no GSI routing table anymore
>>
>> v3->v4 changes (Eric Auger, Alvise Rigo)
>> - rebase on last VFIO PCI code (v2.1.0-rc0)
>> - full git history rework to ease PCI code change review
>> - mv include files in hw/vfio
>> - DPRINTF reformatting temporarily moved out
>> - support of VFIO virq (removal of resamplefd handler on user-side)
>> - integration with sysbus dynamic instantiation framwork
>> - removal of unrealize and cleanup routines until it is better
>>   understood what is really needed
>> - Support of VFIO for Amba devices should be handled in an inherited
>>   device to specialize the device tree generation (clock handle currently
>>   missing in framework however)
>> - "Always use eventfd as notifying mechanism" temporarily moved out
>> - static instantiation is not mainstream (although it remains possible)
>>   note if static instantiation is used, irqfd must be setup in machine file
>>   when virtual IRQ is known
>> - create the GSI routing table on qemu side
>>
>> v2->v3 changes (Alvise Rigo, Eric Auger):
>> - Following Alex W recommandations, further efforts to factorize the
>>   code between PCI:introduction of VFIODevice and VFIORegion
>>   as base classes
>> - unique reset handler for platform and PCI
>> - cleanup following Kim's comments
>> - multiple IRQ support mechanics should be in place although not
>>   tested
>> - Better handling of MMIO multiple regions
>> - New features and fixes by Alvise (multiple compat string, exec
>>   flag, force eventfd usage, amba device tree support)
>> - irqfd support
>>
>> v1->v2 changes (Kim Phillips, Eric Auger):
>> - IRQ initial support (legacy mode where eventfds are handled on
>>   user side)
>> - hacked dynamic instantiation
>>
>> v1 (Kim Phillips):
>> - initial split between PCI and platform
>> - MMIO support only
>> - static instantiation
>>
>> Best Regards
>>
>> Eric
>>
>>
>>
>>
>> Eric Auger (18):
>>   hw/vfio/pci: Rename VFIODevice into VFIOPCIDevice
>>   hw/vfio/pci: generalize mask/unmask to any IRQ index
>>   hw/vfio/pci: introduce minimalist VFIODevice with fd
>>   hw/vfio/pci: add type, name and group fields in VFIODevice
>>   hw/vfio/pci: handle reset at VFIODevice
>>   hw/vfio/pci: Introduce VFIORegion
>>   hw/vfio/pci: split vfio_get_device
>>   hw/vfio/pci: rename group_list into vfio_group_list
>>   hw/vfio/pci: use name field in format strings
>>   hw/vfio: create common module
>>   hw/vfio/platform: add vfio-platform support
>>   hw/vfio: calxeda xgmac device
>>   hw/arm/virt: add support for VFIO devices
>>   hw/arm/sysbus-fdt: enable vfio-calxeda-xgmac dynamic instantiation
>>   hw/vfio/platform: Add irqfd support
>>   linux-headers: Update KVM headers from linux-next tag ToBeFilled
>>   hw/vfio/common: vfio_kvm_device_fd moved in the common header
>>   hw/vfio/platform: add forwarded irq support
>>
>> Kim Phillips (1):
>>   vfio: move hw/misc/vfio.c to hw/vfio/pci.c     Move vfio.h into
>>     include/hw/vfio
>>
>>  LICENSE                              |    2 +-
>>  MAINTAINERS                          |    2 +-
>>  hw/Makefile.objs                     |    1 +
>>  hw/arm/sysbus-fdt.c                  |   88 ++
>>  hw/arm/virt.c                        |   15 +-
>>  hw/misc/Makefile.objs                |    1 -
>>  hw/ppc/spapr_pci_vfio.c              |    2 +-
>>  hw/vfio/Makefile.objs                |    6 +
>>  hw/vfio/calxeda_xgmac.c              |   54 ++
>>  hw/vfio/common.c                     |  960 +++++++++++++++++++
>>  hw/{misc/vfio.c => vfio/pci.c}       | 1671 +++++++---------------------------
>>  hw/vfio/platform.c                   |  777 ++++++++++++++++
>>  include/hw/vfio/vfio-calxeda-xgmac.h |   46 +
>>  include/hw/vfio/vfio-common.h        |  157 ++++
>>  include/hw/vfio/vfio-platform.h      |   88 ++
>>  include/hw/{misc => vfio}/vfio.h     |    0
>>  linux-headers/linux/kvm.h            |   10 +
>>  trace-events                         |  137 +--
>>  18 files changed, 2599 insertions(+), 1418 deletions(-)
>>  create mode 100644 hw/vfio/Makefile.objs
>>  create mode 100644 hw/vfio/calxeda_xgmac.c
>>  create mode 100644 hw/vfio/common.c
>>  rename hw/{misc/vfio.c => vfio/pci.c} (65%)
>>  create mode 100644 hw/vfio/platform.c
>>  create mode 100644 include/hw/vfio/vfio-calxeda-xgmac.h
>>  create mode 100644 include/hw/vfio/vfio-common.h
>>  create mode 100644 include/hw/vfio/vfio-platform.h
>>  rename include/hw/{misc => vfio}/vfio.h (100%)
>>
> 
> 
>