Re: [SPDK] Running nvmf target as non-root

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4710 bytes --]

On Thu, 2019-05-16 at 16:21 +0000, Walker, Benjamin wrote:
> On Thu, 2019-05-16 at 03:39 +0000, Stojaczyk, Dariusz wrote:
> > By default, DPDK tries to retrieve physical addresses from pagemap and that
> > obviously doesn't work without root permissions. Since DPDK 18.11 (I think?)
> > we can provide --iova-mode=va command line param that should make DPDK rely
> > on
> > VFIO IOVAs instead of physical addreses. However, in SPDK we don't provide
> > that param today and there's no way to enable it without changing the code.
> > That param should be enough to start SPDK framework, but particular modules
> > might still malfunction later on. I'm aware of at least vhost PMD that tries
> > to read pagemap as well - it just won't work as an unpriviledged user. 
> > 
> > I think we could make "something" work as an unprivileged user pretty quick,
> > but the big blocker towards supporting today it is our testing framework.
> 
> Here's a patch that makes the tests run as an unprivileged user for nvmf if an
> IOMMU is available. It correctly reproduces the bug on the physical system we
> use in the test pool. The patch needs to get rebased on top of a bunch of
> refactoring patches that Jim pushed out yesterday, but I think this will be
> sufficient to solve the testing problem.
> 
> https://review.gerrithub.io/c/spdk/spdk/+/454679


And here's a patch to DPDK that fixes the problem. I can run the nvmf target as
an unprivileged user with this change (no SPDK changes were required).

https://review.gerrithub.io/c/spdk/dpdk/+/454926

> 
> > D.
> > 
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> > > Benjamin
> > > Sent: Wednesday, May 15, 2019 8:13 PM
> > > To: spdk(a)lists.01.org
> > > Subject: Re: [SPDK] Running nvmf target as non-root
> > > 
> > > On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > > > Hello,
> > > > 
> > > > I was wondering if it is possible to run apps like nvmf target as a
> > > > non-root user.
> > > > 
> > > > I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm
> > > not
> > > > sure if it is possible or not from the text.
> > > > 
> > > > In any case, I enabled IOMMU via the grub command line and loaded vfio-
> > > pci
> > > > but I'm still getting an error:
> > > > 
> > > > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > > > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> > > > --base-virtaddr=0x200000000000 --match-allocations
> > > > --file-prefix=spdk_pid6059 ]
> > > > EAL: VFIO support initialized
> > > > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > > > error allocating rte services array
> > > > EAL: FATAL: rte_service_init() failed
> > > > EAL: rte_service_init() failed
> > > > Failed to initialize DPDK
> > > 
> > > In theory, yes it is possible to run the NVMe-oF target as a non-root
> > > user.
> > > Historically we've even had it working (we contributed the patches to DPDK
> > > to
> > > make it work). But we don't have automated tests for this, and DPDK
> > > doesn't
> > > either, so periodically the functionality goes stale. Based on the error
> > > message
> > > you are seeing, I think that is what happened here - this new
> > > rte_service_init
> > > code in DPDK is doing something that requires root when it shouldn't be.
> > > 
> > > I think we need to address the testing issue prior to fixing whatever the
> > > bug
> > > is. We already have a set of tests running on a physical system with the
> > > IOMMU
> > > enabled, so it should be as simple as dropping privileges in the test
> > > script
> > > when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going
> > > to
> > > give this a shot and see how it works out.
> > > 
> > > > I'm seeing this message in dmesg after enabling iommu in grub (but
> > > nothing
> > > > else).
> > > > [    0.000000] DMAR: IOMMU enabled
> > > > 
> > > > Thanks.
> > > > 
> > > > -- Michael
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > 
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk