From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752424AbdFWERu (ORCPT ); Fri, 23 Jun 2017 00:17:50 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44100 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750821AbdFWERt (ORCPT ); Fri, 23 Jun 2017 00:17:49 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 1413D19D387 Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=peterx@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 1413D19D387 Date: Fri, 23 Jun 2017 12:17:43 +0800 From: Peter Xu To: Alex Williamson Cc: Nitin Saxena , linux-kernel@vger.kernel.org, qemu-devel Subject: Re: Query on VFIO in Virtual machine Message-ID: <20170623041743.GC3936@pxdev.xzpeter.org> References: <20170622105935.2dbd574d@w520.home> <20170622112709.51bd4885@w520.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20170622112709.51bd4885@w520.home> User-Agent: Mutt/1.5.24 (2015-08-30) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Fri, 23 Jun 2017 04:17:49 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote: > On Thu, 22 Jun 2017 22:42:19 +0530 > Nitin Saxena wrote: > > > Thanks Alex. > > > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace, > > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > > is a legitimate configuration? I did tried this configuration and the > > application (in VM) seems to get container_fd, group_fd, device_fd > > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > > from VM as well as from host. This could be specific to PCI endpoint > > device which I can dig. > > > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > > device is legitimate configuration. > > Using no-iommu in the guest should work in that configuration, however > there's no isolation from the user to the rest of VM memory, so the VM > kernel will be tainted. Host memory does have iommu isolation. Device > reset from VM userspace sounds like another bug to investigate. Thanks, > > Alex Besides what Alex has mentioned, there is a wiki page for the usage. The command line will be slightly different on QEMU side comparing to without vIOMMU: http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices One more thing to mention is that, when vfio-pci devices in the guest are used with emulated VT-d, huge performance degradation will be expected for dynamic allocations at least for now. While for mostly static allocations (like DPDK) the performance should be merely the same as no-IOMMU mode. It's just a hint on performance, and I believe for your own case it should mostly depend on how the application is managing DMA map/unmaps. Thanks, -- Peter Xu From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57622) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dOG2b-00075u-KZ for qemu-devel@nongnu.org; Fri, 23 Jun 2017 00:17:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dOG2Y-0006HK-GJ for qemu-devel@nongnu.org; Fri, 23 Jun 2017 00:17:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:48300) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dOG2Y-0006Gt-9H for qemu-devel@nongnu.org; Fri, 23 Jun 2017 00:17:50 -0400 Date: Fri, 23 Jun 2017 12:17:43 +0800 From: Peter Xu Message-ID: <20170623041743.GC3936@pxdev.xzpeter.org> References: <20170622105935.2dbd574d@w520.home> <20170622112709.51bd4885@w520.home> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20170622112709.51bd4885@w520.home> Subject: Re: [Qemu-devel] Query on VFIO in Virtual machine List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson Cc: Nitin Saxena , linux-kernel@vger.kernel.org, qemu-devel On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote: > On Thu, 22 Jun 2017 22:42:19 +0530 > Nitin Saxena wrote: > > > Thanks Alex. > > > > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace, > > So are you trying to say VFIO NO-IOMMU should work inside VM. Does > > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device > > is a legitimate configuration? I did tried this configuration and the > > application (in VM) seems to get container_fd, group_fd, device_fd > > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks > > from VM as well as from host. This could be specific to PCI endpoint > > device which I can dig. > > > > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same > > device is legitimate configuration. > > Using no-iommu in the guest should work in that configuration, however > there's no isolation from the user to the rest of VM memory, so the VM > kernel will be tainted. Host memory does have iommu isolation. Device > reset from VM userspace sounds like another bug to investigate. Thanks, > > Alex Besides what Alex has mentioned, there is a wiki page for the usage. The command line will be slightly different on QEMU side comparing to without vIOMMU: http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices One more thing to mention is that, when vfio-pci devices in the guest are used with emulated VT-d, huge performance degradation will be expected for dynamic allocations at least for now. While for mostly static allocations (like DPDK) the performance should be merely the same as no-IOMMU mode. It's just a hint on performance, and I believe for your own case it should mostly depend on how the application is managing DMA map/unmaps. Thanks, -- Peter Xu