From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2043904947361188270==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Running SPDK As Non-root User Date: Tue, 27 Dec 2016 21:19:19 +0000 Message-ID: <1482873557.6198.1.camel@intel.com> In-Reply-To: 8046EFA3-9F5C-4BB1-B540-51C0ED39D968@cloudsimple.com List-ID: To: spdk@lists.01.org --===============2043904947361188270== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Fri, 2016-12-23 at 16:46 +0530, karthi wrote: > Hi All, > =C2=A0 > I=E2=80=99m trying to run SPDK (NVMf target) in a non-root user mode. I= =E2=80=99m ending up in > rte_mempool_init always returns 0MB Available memory. But if I run the sa= me as > a root user, everything works fine. =C2=A0Can Someone help me out on this= and I > have come across some DPDK mail thread for the same issue. > =C2=A0 > http://dpdk.org/ml/archives/users/2016-July/000709.html > =C2=A0 > Can DPDK currently run in non-root user mode? Yes, it can, although it isn't well tested or heavily used as far as I can = tell. > =C2=A0 > Is someone tried experimenting this in either DPDK or SPDK? SPDK's scripts/setup.sh will do all of the things necessary to run your application as an unprivileged user if your system has your IOMMU enabled i= n the kernel boot parameters. The script will automatically use vfio instead of u= io and correctly set the permissions on all of the devices and hugepage files = to the user you ran the script as. Note that you do need to run the script as = the user you want to grant permission, but under 'sudo' because it needs root permission to do the initial setup. I'd like to dive into the details here a bit so that everyone knows where we really stand on this. I see running as an unprivileged user a key feature f= or production deployments, so it's one of those things that we really do want = to get functioning as a first class citizen in our test pool and documentation= as soon as possible. DPDK and SPDK (and user space drivers in general) need two features from the kernel to function. First, they must be able to allocate memory that has a fixed virtual to phy= sical address mapping, which is usually called 'pinned' memory. This memory is us= ed for DMA operations that happen asynchronously and off of the CPU, e.g. the = data for reads and writes from the SSD or NIC. DPDK/SPDK accomplish this by allocating the memory from hugepages, which happen to meet this requirement today. Second, they must be able to map the memory region described by the base ad= dress register (BAR) in the PCI header into a virtual address accessible by the process they are running in. DPDK/SPDK can do this using one of two Linux k= ernel mechanisms - uio and vfio - where vfio is the newer of the two. Both mechan= isms present the user with a file in sysfs that can be mmap'd for this purpose. Running SPDK as a non-root user, then, is a matter of finding ways to accom= plish the above two tasks without root. Hugepages are documented here: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt The BAR is harder and it wasn't possible to map this as an unprivileged user until the introduction of vfio. With vfio, a privileged user can simply gra= nt access to the resource file in sysfs to any other unprivileged user. See: https://www.kernel.org/doc/Documentation/vfio.txt Note that SPDK's scripts/setup.sh does all of this automatically if it dete= cts that your system has the IOMMU enabled. The IOMMU is a critical piece secur= ity- wise for unprivileged execution because it will prevent your otherwise unprivileged user from having free reign to DMA to any memory address. The above all sounds great, except I just actually tried this and when I ru= n the NVMe identify example at ./examples/nvme/identify, DPDK gives me an error! = After much debugging, it turns out that in kernel 4.0 the kernel devs removed the ability to get physical addresses for pages from /proc/self/pagemap without elevated privileges in response to the Rowhammer exploit. Everything should= work fine on a 3.x series kernel, but I don't have access to a machine to test i= t out right now. The simplest solution would be to make your program start up with elevated permissions and initialize DPDK. That provides the required virtual to phys= ical memory mappings, which should be static for hugepages. After initialization= , the privilege can be dropped back down. That's the strategy I would pursue for = your application today. We'll keep looking at this and working with the kernel community to come up with a better long term solution. Thanks, Ben > =C2=A0 > =C2=A0 > Regards, > =C2=A0 > Karthi | +91 9036339210 > CloudSimple Inc. > =C2=A0 > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============2043904947361188270==--