From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: VFIO no-iommu Date: Tue, 15 Dec 2015 21:38:31 -0700 Message-ID: <1450240711.2674.11.camel@redhat.com> References: <60420822.AbcfvjLZCk@xps13> <566B4A50.9090607@6wind.com> <1449874953.20509.6.camel@redhat.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA6747CE55@IRSMSX108.ger.corp.intel.com> <1450198398.6042.32.camel@redhat.com> <20151216040408.GA18363@sivlogin002.ir.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Cc: "dev@dpdk.org" To: Ferruh Yigit Return-path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by dpdk.org (Postfix) with ESMTP id 531B73005 for ; Wed, 16 Dec 2015 05:38:36 +0100 (CET) In-Reply-To: <20151216040408.GA18363@sivlogin002.ir.intel.com> List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, 2015-12-16 at 04:04 +0000, Ferruh Yigit wrote: > On Tue, Dec 15, 2015 at 09:53:18AM -0700, Alex Williamson wrote: > > On Tue, 2015-12-15 at 13:43 +0000, O'Driscoll, Tim wrote: > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex > > > > Williamson > > > > Sent: Friday, December 11, 2015 11:03 PM > > > > To: Vincent JARDIN; dev@dpdk.org > > > > Subject: Re: [dpdk-dev] VFIO no-iommu > > > >=20 > > > > On Fri, 2015-12-11 at 23:12 +0100, Vincent JARDIN wrote: > > > > > Thanks Thomas for putting back this topic. > > > > >=20 > > > > > Alex, > > > > >=20 > > > > > I'd like to hear more about the impacts of "unsupported": > > > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.g > > > > > it/c > > > > > ommi > > > > > t/?id=3D033291eccbdb1b70ffc02641edae19ac825dc75d > > > > > =C2=A0=C2=A0=C2=A0Use of this mode, specifically binding a devi= ce without a > > > > > native > > > > > =C2=A0=C2=A0=C2=A0IOMMU group to a VFIO bus driver will taint t= he kernel and > > > > > should > > > > > =C2=A0=C2=A0=C2=A0therefore not be considered supported. > > > > >=20 > > > > > It means that we get ride of uio; so it is a nice code > > > > > cleanup: > > > > > but > > > > > why > > > > > would VFIO/NO IOMMU be better if the bottomline is > > > > > "unsupported"? > > > >=20 > > > > How supportable do you think the uio method is? =C2=A0Fundamental= ly > > > > we > > > > have > > > > a userspace driver doing unrestricted DMA; it can access and > > > > modify > > > > any > > > > memory in the system. =C2=A0This is the reason uio won't provide = a > > > > mechanism > > > > to enable MSI and if you ask the uio maintainer, they don't > > > > support > > > > DMA > > > > at all, it's only intended as a programmed IO interface to the > > > > device. > > > > =C2=A0Unless we can sandbox a user owned device within an IOMMU > > > > protected > > > > container, it's not supportable. =C2=A0The VFIO no-iommu mode can > > > > simply > > > > provide you that unsupported mode more easily since it > > > > leverages > > > > code > > > > from the supported mode, which is IOMMU protected. =C2=A0Thanks, > > >=20 > > > Thanks for clarifying. > > >=20 > > > This does seem like it would be useful for DPDK. We're doing some > > > further investigation to see if it works out of the box with DPDK > > > or > > > if we need to make any changes to support it. > >=20 > > The iommu model is different, there's no type1 interface available > > when > > using this mode since we have no ability to provide translation. > > =C2=A0The > > no-iommu iommu model really does nothing, which is a possible issue > > for > > userspace. =C2=A0Is it sufficient? =C2=A0We stopped short of creating= a page > > pinning interface through the no-iommu model because it requires > > code > > and adding piles of new code for an interface we claim is > > unsupported > > doesn't make a lot of sense. =C2=A0The device interface should be > > identical > > to existing vfio support. > >=20 > > > Thomas highlighted that your original commit for this had been > > > reverted. What specifically would you need from us in order to > > > re- > > > submit the VFIO No-IOMMU support? > >=20 > > No API changes should ever go into the kernel without being > > validated > > by a user. =C2=A0Without that we're risking that the kernel interface= is > > broken and we're stuck supporting it. =C2=A0In this case I tried to m= ake > > sure we had a working user before it went it, gambled that it was > > close > > enough to put in anyway, then paid the price when development went > > silent on the user side. =C2=A0To get it back in, I'm going to need a > > working use first. =C2=A0You can re-apply 033291eccbdb or re- > > revert=C2=A0ae5515d66362 for development of that. =C2=A0I need to see= that it > > works and that there's some consensus from the dpdk community that > > it's > > a worthwhile path forward for cases without an iommu. =C2=A0There's n= o > > point > > in merging it if it only becomes a userspace proof of concept. > > =C2=A0Thanks, > >=20 > I tested the DPDK (HEAD of master) with the patch, with help of > Anatoly, > and DPDK works in no-iommu environment with a little modification. >=20 > Basically the only modification is adapt new group naming (noiommu-$)=20 > and Sorry, forgot to mention that one. =C2=A0The intention with the modified group name is that I want to be very certain that a user intending to only support properly iommu isolated devices doesn't accidentally need to deal with these no-iommu mode devices. > disable dma mapping (VFIO_IOMMU_MAP_DMA) >=20 > Also I need to disable VFIO_CHECK_EXTENSION ioctl, because in vfio > module, > container->noiommu is not set before doing a > vfio_group_set_container() > and vfio_for_each_iommu_driver selects wrong driver. Running CHECK_EXTENSION on a container without the group attached is only going to tell you what extensions vfio is capable of, not necessarily what extensions are available to you with that group. =C2=A0I= s this just a general dpdk-vfio ordering bug? > What I test is bind two different type of NICs into VFIO driver, and > use > testpmd to confirm transfer is working.=C2=A0=C2=A0Kernel booted withou= t iommu > enabled, > vfio module inserted with "enable_unsafe_noiommu_support" parameter. So it works. =C2=A0Is it acceptable? =C2=A0Useful? =C2=A0Sufficiently com= plete? =C2=A0Does it imply deprecating the uio interface? =C2=A0I believe the feature that started this discussion was support for MSI/X interrupts so that VFs can support some kind of interrupt (uio only supports INTx since it doesn't allow DMA). =C2=A0Implementing that would be the ultimate test of whether this provides dpdk with not only a more consistent interface, but the feature dpdk wants that's missing in uio. Thanks, Alex