[SPDK] Running nvmf target as non-root

[SPDK] Running nvmf target as non-root

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

Hello,

I was wondering if it is possible to run apps like nvmf target as a
non-root user.

I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm not
sure if it is possible or not from the text.

In any case, I enabled IOMMU via the grub command line and loaded vfio-pci
but I'm still getting an error:

Starting SPDK v19.04 / DPDK 19.02.0 initialization...
[ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
--base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid6059 ]
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
Failed to initialize DPDK

I'm seeing this message in dmesg after enabling iommu in grub (but nothing
else).
[    0.000000] DMAR: IOMMU enabled

Thanks.

-- Michael

Re: [SPDK] Running nvmf target as non-root

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2023 bytes --]

On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> Hello,
> 
> I was wondering if it is possible to run apps like nvmf target as a
> non-root user.
> 
> I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm not
> sure if it is possible or not from the text.
> 
> In any case, I enabled IOMMU via the grub command line and loaded vfio-pci
> but I'm still getting an error:
> 
> Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> --base-virtaddr=0x200000000000 --match-allocations
> --file-prefix=spdk_pid6059 ]
> EAL: VFIO support initialized
> EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> error allocating rte services array
> EAL: FATAL: rte_service_init() failed
> EAL: rte_service_init() failed
> Failed to initialize DPDK

In theory, yes it is possible to run the NVMe-oF target as a non-root user.
Historically we've even had it working (we contributed the patches to DPDK to
make it work). But we don't have automated tests for this, and DPDK doesn't
either, so periodically the functionality goes stale. Based on the error message
you are seeing, I think that is what happened here - this new rte_service_init
code in DPDK is doing something that requires root when it shouldn't be.

I think we need to address the testing issue prior to fixing whatever the bug
is. We already have a set of tests running on a physical system with the IOMMU
enabled, so it should be as simple as dropping privileges in the test script
when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going to
give this a shot and see how it works out.

> 
> I'm seeing this message in dmesg after enabling iommu in grub (but nothing
> else).
> [    0.000000] DMAR: IOMMU enabled
> 
> Thanks.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running nvmf target as non-root

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 3289 bytes --]

By default, DPDK tries to retrieve physical addresses from pagemap and that obviously doesn't work without root permissions. Since DPDK 18.11 (I think?) we can provide --iova-mode=va command line param that should make DPDK rely on VFIO IOVAs instead of physical addreses. However, in SPDK we don't provide that param today and there's no way to enable it without changing the code. That param should be enough to start SPDK framework, but particular modules might still malfunction later on. I'm aware of at least vhost PMD that tries to read pagemap as well - it just won't work as an unpriviledged user. 

I think we could make "something" work as an unprivileged user pretty quick, but the big blocker towards supporting today it is our testing framework.

D.

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> Benjamin
> Sent: Wednesday, May 15, 2019 8:13 PM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] Running nvmf target as non-root
> 
> On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > Hello,
> >
> > I was wondering if it is possible to run apps like nvmf target as a
> > non-root user.
> >
> > I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm
> not
> > sure if it is possible or not from the text.
> >
> > In any case, I enabled IOMMU via the grub command line and loaded vfio-
> pci
> > but I'm still getting an error:
> >
> > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> > --base-virtaddr=0x200000000000 --match-allocations
> > --file-prefix=spdk_pid6059 ]
> > EAL: VFIO support initialized
> > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > error allocating rte services array
> > EAL: FATAL: rte_service_init() failed
> > EAL: rte_service_init() failed
> > Failed to initialize DPDK
> 
> In theory, yes it is possible to run the NVMe-oF target as a non-root user.
> Historically we've even had it working (we contributed the patches to DPDK
> to
> make it work). But we don't have automated tests for this, and DPDK doesn't
> either, so periodically the functionality goes stale. Based on the error
> message
> you are seeing, I think that is what happened here - this new rte_service_init
> code in DPDK is doing something that requires root when it shouldn't be.
> 
> I think we need to address the testing issue prior to fixing whatever the bug
> is. We already have a set of tests running on a physical system with the
> IOMMU
> enabled, so it should be as simple as dropping privileges in the test script
> when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going
> to
> give this a shot and see how it works out.
> 
> >
> > I'm seeing this message in dmesg after enabling iommu in grub (but
> nothing
> > else).
> > [    0.000000] DMAR: IOMMU enabled
> >
> > Thanks.
> >
> > -- Michael
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running nvmf target as non-root

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4082 bytes --]

On Thu, 2019-05-16 at 03:39 +0000, Stojaczyk, Dariusz wrote:
> By default, DPDK tries to retrieve physical addresses from pagemap and that
> obviously doesn't work without root permissions. Since DPDK 18.11 (I think?)
> we can provide --iova-mode=va command line param that should make DPDK rely on
> VFIO IOVAs instead of physical addreses. However, in SPDK we don't provide
> that param today and there's no way to enable it without changing the code.
> That param should be enough to start SPDK framework, but particular modules
> might still malfunction later on. I'm aware of at least vhost PMD that tries
> to read pagemap as well - it just won't work as an unpriviledged user. 
> 
> I think we could make "something" work as an unprivileged user pretty quick,
> but the big blocker towards supporting today it is our testing framework.

Here's a patch that makes the tests run as an unprivileged user for nvmf if an
IOMMU is available. It correctly reproduces the bug on the physical system we
use in the test pool. The patch needs to get rebased on top of a bunch of
refactoring patches that Jim pushed out yesterday, but I think this will be
sufficient to solve the testing problem.

https://review.gerrithub.io/c/spdk/spdk/+/454679

> 
> D.
> 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> > Benjamin
> > Sent: Wednesday, May 15, 2019 8:13 PM
> > To: spdk(a)lists.01.org
> > Subject: Re: [SPDK] Running nvmf target as non-root
> > 
> > On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > > Hello,
> > > 
> > > I was wondering if it is possible to run apps like nvmf target as a
> > > non-root user.
> > > 
> > > I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm
> > not
> > > sure if it is possible or not from the text.
> > > 
> > > In any case, I enabled IOMMU via the grub command line and loaded vfio-
> > pci
> > > but I'm still getting an error:
> > > 
> > > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> > > --base-virtaddr=0x200000000000 --match-allocations
> > > --file-prefix=spdk_pid6059 ]
> > > EAL: VFIO support initialized
> > > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > > error allocating rte services array
> > > EAL: FATAL: rte_service_init() failed
> > > EAL: rte_service_init() failed
> > > Failed to initialize DPDK
> > 
> > In theory, yes it is possible to run the NVMe-oF target as a non-root user.
> > Historically we've even had it working (we contributed the patches to DPDK
> > to
> > make it work). But we don't have automated tests for this, and DPDK doesn't
> > either, so periodically the functionality goes stale. Based on the error
> > message
> > you are seeing, I think that is what happened here - this new
> > rte_service_init
> > code in DPDK is doing something that requires root when it shouldn't be.
> > 
> > I think we need to address the testing issue prior to fixing whatever the
> > bug
> > is. We already have a set of tests running on a physical system with the
> > IOMMU
> > enabled, so it should be as simple as dropping privileges in the test script
> > when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going
> > to
> > give this a shot and see how it works out.
> > 
> > > I'm seeing this message in dmesg after enabling iommu in grub (but
> > nothing
> > > else).
> > > [    0.000000] DMAR: IOMMU enabled
> > > 
> > > Thanks.
> > > 
> > > -- Michael
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running nvmf target as non-root

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4710 bytes --]

On Thu, 2019-05-16 at 16:21 +0000, Walker, Benjamin wrote:
> On Thu, 2019-05-16 at 03:39 +0000, Stojaczyk, Dariusz wrote:
> > By default, DPDK tries to retrieve physical addresses from pagemap and that
> > obviously doesn't work without root permissions. Since DPDK 18.11 (I think?)
> > we can provide --iova-mode=va command line param that should make DPDK rely
> > on
> > VFIO IOVAs instead of physical addreses. However, in SPDK we don't provide
> > that param today and there's no way to enable it without changing the code.
> > That param should be enough to start SPDK framework, but particular modules
> > might still malfunction later on. I'm aware of at least vhost PMD that tries
> > to read pagemap as well - it just won't work as an unpriviledged user. 
> > 
> > I think we could make "something" work as an unprivileged user pretty quick,
> > but the big blocker towards supporting today it is our testing framework.
> 
> Here's a patch that makes the tests run as an unprivileged user for nvmf if an
> IOMMU is available. It correctly reproduces the bug on the physical system we
> use in the test pool. The patch needs to get rebased on top of a bunch of
> refactoring patches that Jim pushed out yesterday, but I think this will be
> sufficient to solve the testing problem.
> 
> https://review.gerrithub.io/c/spdk/spdk/+/454679


And here's a patch to DPDK that fixes the problem. I can run the nvmf target as
an unprivileged user with this change (no SPDK changes were required).

https://review.gerrithub.io/c/spdk/dpdk/+/454926

> 
> > D.
> > 
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> > > Benjamin
> > > Sent: Wednesday, May 15, 2019 8:13 PM
> > > To: spdk(a)lists.01.org
> > > Subject: Re: [SPDK] Running nvmf target as non-root
> > > 
> > > On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > > > Hello,
> > > > 
> > > > I was wondering if it is possible to run apps like nvmf target as a
> > > > non-root user.
> > > > 
> > > > I read the sections about pagemap/IOMMU in both DPDK and SPDK and I'm
> > > not
> > > > sure if it is possible or not from the text.
> > > > 
> > > > In any case, I enabled IOMMU via the grub command line and loaded vfio-
> > > pci
> > > > but I'm still getting an error:
> > > > 
> > > > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > > > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1 --log-level=lib.eal:6
> > > > --base-virtaddr=0x200000000000 --match-allocations
> > > > --file-prefix=spdk_pid6059 ]
> > > > EAL: VFIO support initialized
> > > > EAL: Cannot obtain physical addresses: Success. Only vfio will function.
> > > > error allocating rte services array
> > > > EAL: FATAL: rte_service_init() failed
> > > > EAL: rte_service_init() failed
> > > > Failed to initialize DPDK
> > > 
> > > In theory, yes it is possible to run the NVMe-oF target as a non-root
> > > user.
> > > Historically we've even had it working (we contributed the patches to DPDK
> > > to
> > > make it work). But we don't have automated tests for this, and DPDK
> > > doesn't
> > > either, so periodically the functionality goes stale. Based on the error
> > > message
> > > you are seeing, I think that is what happened here - this new
> > > rte_service_init
> > > code in DPDK is doing something that requires root when it shouldn't be.
> > > 
> > > I think we need to address the testing issue prior to fixing whatever the
> > > bug
> > > is. We already have a set of tests running on a physical system with the
> > > IOMMU
> > > enabled, so it should be as simple as dropping privileges in the test
> > > script
> > > when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm going
> > > to
> > > give this a shot and see how it works out.
> > > 
> > > > I'm seeing this message in dmesg after enabling iommu in grub (but
> > > nothing
> > > > else).
> > > > [    0.000000] DMAR: IOMMU enabled
> > > > 
> > > > Thanks.
> > > > 
> > > > -- Michael
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > 
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Running nvmf target as non-root

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 5273 bytes --]

Awesome. I'll give it a try next week!

-- Michael

On Fri, May 17, 2019 at 11:30 AM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Thu, 2019-05-16 at 16:21 +0000, Walker, Benjamin wrote:
> > On Thu, 2019-05-16 at 03:39 +0000, Stojaczyk, Dariusz wrote:
> > > By default, DPDK tries to retrieve physical addresses from pagemap and
> that
> > > obviously doesn't work without root permissions. Since DPDK 18.11 (I
> think?)
> > > we can provide --iova-mode=va command line param that should make DPDK
> rely
> > > on
> > > VFIO IOVAs instead of physical addreses. However, in SPDK we don't
> provide
> > > that param today and there's no way to enable it without changing the
> code.
> > > That param should be enough to start SPDK framework, but particular
> modules
> > > might still malfunction later on. I'm aware of at least vhost PMD that
> tries
> > > to read pagemap as well - it just won't work as an unpriviledged user.
> > >
> > > I think we could make "something" work as an unprivileged user pretty
> quick,
> > > but the big blocker towards supporting today it is our testing
> framework.
> >
> > Here's a patch that makes the tests run as an unprivileged user for nvmf
> if an
> > IOMMU is available. It correctly reproduces the bug on the physical
> system we
> > use in the test pool. The patch needs to get rebased on top of a bunch of
> > refactoring patches that Jim pushed out yesterday, but I think this will
> be
> > sufficient to solve the testing problem.
> >
> > https://review.gerrithub.io/c/spdk/spdk/+/454679
>
>
> And here's a patch to DPDK that fixes the problem. I can run the nvmf
> target as
> an unprivileged user with this change (no SPDK changes were required).
>
> https://review.gerrithub.io/c/spdk/dpdk/+/454926
>
> >
> > > D.
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> > > > Benjamin
> > > > Sent: Wednesday, May 15, 2019 8:13 PM
> > > > To: spdk(a)lists.01.org
> > > > Subject: Re: [SPDK] Running nvmf target as non-root
> > > >
> > > > On Tue, 2019-05-14 at 13:16 -0600, Michael Haeuptle wrote:
> > > > > Hello,
> > > > >
> > > > > I was wondering if it is possible to run apps like nvmf target as a
> > > > > non-root user.
> > > > >
> > > > > I read the sections about pagemap/IOMMU in both DPDK and SPDK and
> I'm
> > > > not
> > > > > sure if it is possible or not from the text.
> > > > >
> > > > > In any case, I enabled IOMMU via the grub command line and loaded
> vfio-
> > > > pci
> > > > > but I'm still getting an error:
> > > > >
> > > > > Starting SPDK v19.04 / DPDK 19.02.0 initialization...
> > > > > [ DPDK EAL parameters: nvgrid --no-shconf -c 0x1
> --log-level=lib.eal:6
> > > > > --base-virtaddr=0x200000000000 --match-allocations
> > > > > --file-prefix=spdk_pid6059 ]
> > > > > EAL: VFIO support initialized
> > > > > EAL: Cannot obtain physical addresses: Success. Only vfio will
> function.
> > > > > error allocating rte services array
> > > > > EAL: FATAL: rte_service_init() failed
> > > > > EAL: rte_service_init() failed
> > > > > Failed to initialize DPDK
> > > >
> > > > In theory, yes it is possible to run the NVMe-oF target as a non-root
> > > > user.
> > > > Historically we've even had it working (we contributed the patches
> to DPDK
> > > > to
> > > > make it work). But we don't have automated tests for this, and DPDK
> > > > doesn't
> > > > either, so periodically the functionality goes stale. Based on the
> error
> > > > message
> > > > you are seeing, I think that is what happened here - this new
> > > > rte_service_init
> > > > code in DPDK is doing something that requires root when it shouldn't
> be.
> > > >
> > > > I think we need to address the testing issue prior to fixing
> whatever the
> > > > bug
> > > > is. We already have a set of tests running on a physical system with
> the
> > > > IOMMU
> > > > enabled, so it should be as simple as dropping privileges in the test
> > > > script
> > > > when it goes to run the NVMe-oF target, if the IOMMU is enabled. I'm
> going
> > > > to
> > > > give this a shot and see how it works out.
> > > >
> > > > > I'm seeing this message in dmesg after enabling iommu in grub (but
> > > > nothing
> > > > > else).
> > > > > [    0.000000] DMAR: IOMMU enabled
> > > > >
> > > > > Thanks.
> > > > >
> > > > > -- Michael
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>