Re: [SPDK] Running nvmf target as non-root

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4082 bytes --]

On Thu, 2019-05-16 at 03:39 +0000, Stojaczyk, Dariusz wrote:
> By default, DPDK tries to retrieve physical addresses from pagemap and that
> obviously doesn't work without root permissions. Since DPDK 18.11 (I think?)
> we can provide --iova-mode=va command line param that should make DPDK rely on
> VFIO IOVAs instead of physical addreses. However, in SPDK we don't provide
> that param today and there's no way to enable it without changing the code.
> That param should be enough to start SPDK framework, but particular modules
> might still malfunction later on. I'm aware of at least vhost PMD that tries
> to read pagemap as well - it just won't work as an unpriviledged user. 
> 
> I think we could make "something" work as an unprivileged user pretty quick,
> but the big blocker towards supporting today it is our testing framework.

Here's a patch that makes the tests run as an unprivileged user for nvmf if an
IOMMU is available. It correctly reproduces the bug on the physical system we
use in the test pool. The patch needs to get rebased on top of a bunch of
refactoring patches that Jim pushed out yesterday, but I think this will be
sufficient to solve the testing problem.

https://review.gerrithub.io/c/spdk/spdk/+/454679

> 
> D.
> 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> > Benjamin
> > Sent: Wednesday, May 15, 2019 8:13 PM
> > To: spdk(a)lists.01.org
> > Subject: Re: [SPDK] Running nvmf target as non-root
> > 
> > On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > > Hello,
> > > 
> > > I was wondering if it is possible to run apps like nvmf target as a
> > > non-root user.
> > > 
> > > I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm
> > not
> > > sure if it is possible or not from the text.
> > > 
> > > In any case, I enabled IOMMU via the grub command line and loaded vfio-
> > pci
> > > but I'm still getting an error:
> > > 
> > > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> > > --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid6059 ]
> > > EAL: VFIO support initialized
> > > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > > error allocating rte services array
> > > EAL: FATAL: rte_service_init() failed
> > > EAL: rte_service_init() failed
> > > Failed to initialize DPDK
> > 
> > In theory, yes it is possible to run the NVMe-oF target as a non-root user.
> > Historically we've even had it working (we contributed the patches to DPDK
> > to
> > make it work). But we don't have automated tests for this, and DPDK doesn't
> > either, so periodically the functionality goes stale. Based on the error
> > message
> > you are seeing, I think that is what happened here - this new
> > rte_service_init
> > code in DPDK is doing something that requires root when it shouldn't be.
> > 
> > I think we need to address the testing issue prior to fixing whatever the
> > bug
> > is. We already have a set of tests running on a physical system with the
> > IOMMU
> > enabled, so it should be as simple as dropping privileges in the test script
> > when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going
> > to
> > give this a shot and see how it works out.
> > 
> > > I'm seeing this message in dmesg after enabling iommu in grub (but
> > nothing
> > > else).
> > > [    0.000000] DMAR: IOMMU enabled
> > > 
> > > Thanks.
> > > 
> > > -- Michael
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk