From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:59261) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bAGvc-0002ij-4w for qemu-devel@nongnu.org; Tue, 07 Jun 2016 09:20:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bAGvW-0008DG-2R for qemu-devel@nongnu.org; Tue, 07 Jun 2016 09:20:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:60627) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bAGvV-0008Cy-RT for qemu-devel@nongnu.org; Tue, 07 Jun 2016 09:20:14 -0400 Date: Tue, 7 Jun 2016 21:20:08 +0800 From: Peter Xu Message-ID: <20160607132008.GD1039@pxdev.xzpeter.org> References: <20160528100220.2d48ff2d@ul30vt.home> <20160528113936.48e67fac@ul30vt.home> <20160528134800.1af926d1@ul30vt.home> <20160602073417.43735ed3@ul30vt.home> <20160606080911.GI21254@pxdev.xzpeter.org> <20160606122134.47e6ef39@ul30vt.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20160606122134.47e6ef39@ul30vt.home> Subject: Re: [Qemu-devel] [PATCH v3 3/3] IOMMU: Integrate between VFIO and vIOMMU to support device assignment List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson Cc: "Aviv B.D." , Jan Kiszka , qemu-devel@nongnu.org, "Michael S. Tsirkin" On Mon, Jun 06, 2016 at 12:21:34PM -0600, Alex Williamson wrote: [...] > I'm not sure why you're so eager to avoid implementing a replay > callback for VT-d. What happens when VT-d is not enabled by the > guest? vfio/pci.c calls pci_device_iommu_address_space() to get the > address space for the device, which results in vtd_find_add_as() which > gives us a unique VTDAddressSpace.as for that device. With VT-d not in > use by the guest, when do steps 3-5 occur? I agree with your reasoning > when VT-d is enabled, but the BIOS/UEFI does not enable VT-d, so are we > going to exclude all use cases of an assigned device prior to the guest > enabling VT-d? I think I got the point. I failed to consider the case that we can run IOMMU without enabling it, like BIOS (as you have mentioned), or we can disable IOMMU in kernel boot parameters (though, iommu=pt should still follow the case that IOMMU is enabled). Sounds like a replay callback is a good idea. For Intel version of the callback: when DMAR is enabled, we can just return directly. when DMAR is disabled, we just do whatever we need to do region_add() for the global address_space_memory. > > On that same topic, I'm really dubious that we have the flexibility in > our memory API to really support an IOMMU like VT-d and the layout of > having a VTDAddressSpace per device, rather than exposing shared > address spaces, has implications on the efficiency and locked memory > requirements for vfio. In the above case with VT-d disabled, the > VTDAddressSpace should alias to the system memory AddressSpace and > dynamically switch to a per device address space when VT-d is enabled. > AFAICT, we don't have anything resembling that sort of feature, so we > rely on the IOMMU driver to replay, perhaps even configuring its own > MemoryListener to IOMMU notifier gateway, which is also something that > doesn't currently exist. It sounds more like a notifier for "IOMMU enablement"? The notifier should be triggered when IOMMU switch between enabled <-> disabled? When this happens, we rebuild the mapping in some way. > > Additionally, if there are things unique to VT-d, for instance if VT-d > is enabled and we can rely on the sequence of events you've set forth, > then yes, the replay mechanism should do nothing. But only the VT-d > code can decide that, which implies that vfio always needs to call the > replay function and a new MemoryRegionIOMMUOps callback needs to decide > what to do given the current state of the vIOMMU. Thanks, Right. Thanks. -- peterx