Re: Running DPDK as an unprivileged user

["Walker, Benjamin" <benjamin.walker@intel.com>]

On Thu, 2016-12-29 at 17:14 -0800, Stephen Hemminger wrote:
> If kernel broke pinning of hugepages, then it is an upstream kernel bug.

The kernel, under a myriad of circumstances, will change the mapping of virtual
to physical addresses for hugepages. This behavior began somewhere around kernel
3.16 and with each release more cases where the mapping can change are
introduced. DPDK should not be relying on that mapping staying static, and
instead should be using vfio to explicitly pin the pages. I've consulted the
relevant kernel developers who write the code in this area and they are
universally in agreement that this is not a kernel bug and the mappings will get
less static over time.

On Mon, 2017-01-02 at 11:47 -0800, Stephen Hemminger wrote:
> On Mon, 02 Jan 2017 15:32:08 +0100
> Thomas Monjalon <thomas.monjalon@6wind.com> wrote:
> 
> > 2016-12-29 17:14, Stephen Hemminger:
> > > On Thu, 29 Dec 2016 20:41:21 +0000
> > > "Walker, Benjamin" <benjamin.walker@intel.com> wrote:  
> > > > My second question is whether the user should be allowed to
> > > > mix uio and vfio usage simultaneously. For vfio, the
> > > > physical addresses are really DMA addresses and are best
> > > > when arbitrarily chosen to appear sequential relative to
> > > > their virtual addresses. For uio, they are physical
> > > > addresses and are not chosen at all. It seems that these two
> > > > things are in conflict and that it will be difficult, ugly,
> > > > and maybe impossible to resolve the simultaneous use of
> > > > both.  
> > > 
> > > Unless application is running as privileged user (ie root), UIO
> > > is not going to work. Therefore don't worry about mixed environment.  
> > 
> > Yes, mixing UIO and VFIO is possible only as root.
> > However, what is the benefit of mixing them?
> 
> One possible case where this could be used, Hyper-V/Azure and SR-IOV.
> The VF interface will show up on an isolated PCI bus and the virtual NIC
> is on VMBUS. It is possible to use VFIO on the PCI to get MSI-X per queue
> interrupts, but there is no support for VFIO on VMBUS.

I sent out a patch a little while ago that makes DPDK work when running as an
unprivileged user with an IOMMU. I allow mixing of uio/vfio when root (I choose
the DMA address to be the physical address), but only vfio when unprivileged (I
choose the DMA addresses to start at 0).

Unfortunately, there are a few more wrinkles for systems that do not have an
IOMMU. These systems still need to explicitly pin memory, but they need to use
physical addresses instead of DMA addresses. There are two concerns with this:

1) Physical addresses cannot be exposed to unprivileged users due to security
concerns (the fallout of rowhammer). Therefore, systems without an IOMMU can
only support privileged users. I think this is probably fine.
2) The IOCTL from vfio to pin the memory is tied to specifying the DMA address
and programming the IOMMU. This is unfortunate - systems without an IOMMU still
want to do the pinning, but they need to be given the physical address instead
of specifying a DMA address.
3) Not all device types, particularly in virtualization environments, support
vfio today. These devices have no way to explicitly pin memory.

I think this is going to take a kernel patch or two to resolve, unless someone
has a good idea.