From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0920394065516173664==" MIME-Version: 1.0 From: Nabarro, Tom Subject: [SPDK] Re: vfio multiple processes Date: Thu, 06 Feb 2020 14:26:16 +0000 Message-ID: <6fa2b242862342a6899a453fe23b8d96@intel.com> In-Reply-To: 4beda3a3f6e243868d99e4209bab6aab@intel.com List-ID: To: spdk@lists.01.org --===============0920394065516173664== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable That's useful info, we used to have the primary process (which performed de= vice provisioning) launch the secondary I/O processes (after device provisi= oning) therefore we could always do discovery through the primary process o= n devices in use by the secondaries. = Now we want to run our main processes as non-root so switch to vfio and we = do device provisioning through a privileged helper binary (which runs setui= d root and not in multiprocess mode) and call out to it from our main contr= ol process (not a SPDK process). The control process then forks our I/O pro= cesses which run in primary mode (as non-root) so that when we want to inte= rrogate devices being used by our I/O processes we can attach as secondary = rather than have to implement some form of IPC. Regards, Tom Nabarro - DCG/ESAD M: +44 (0)7786 260986 Skype: tom.nabarro -----Original Message----- From: Stojaczyk, Dariusz = Sent: Tuesday, February 4, 2020 12:27 PM To: Storage Performance Development Kit Subject: [SPDK] Re: vfio multiple processes Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK= keeps vfio handles open because they may be required for a secondary proce= ss. There's room for improvement, but the current state is probably going t= o stay for a while. D. > -----Original Message----- > From: Stojaczyk, Dariusz > Sent: Monday, February 3, 2020 10:14 AM > To: Storage Performance Development Kit > Subject: [SPDK] Re: vfio multiple processes > = > TBH I'm not sure the whitelist should be required. Sounds like a bug, = > I'll take a look. > = > D. > = > > -----Original Message----- > > From: Nabarro, Tom > > Sent: Saturday, February 1, 2020 3:00 PM > > To: Storage Performance Development Kit > > Subject: [SPDK] Re: vfio multiple processes > > > > Resolved by adding pci whitelist into spdk_env_opts. > > Adding whitelist into nvme.conf was not sufficient. > > > > Regards, > > Tom Nabarro - DCG/ESAD > > M: +44 (0)7786 260986 > > Skype: tom.nabarro > > > > From: Nabarro, Tom > > Sent: Friday, January 31, 2020 12:42 PM > > To: 'Storage Performance Development Kit' > > Subject: vfio multiple processes > > > > Can't run multiple SPDK processes each accessing disjoint set of = > > NVMe drives using VFIO driver (even as root). Each process seems to = > > try to access all > > /dev/vfio/* device files and get lots of resource busy messages. = > > Sometimes one of the processes complaint it can't find the device = > > it's been assigned in nvme.conf. > > > > Have any ideas? > > (processes are started in normal, non-multiprocess mode) > > > > Thanks in advance > > > > Best regards, > > Tom Nabarro BEng (hons) > > Extreme Storage Architecture & Development Intel Corporation > > E: tom.nabarro(a)intel.com > > M: +44 (0)7786 260986 > > Skype: tom.nabarro > > > > -------------------------------------------------------------------- > > - > > Intel Corporation (UK) Limited > > Registered No. 1134945 (England) > > Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 > > > > This e-mail and any attachments may contain confidential material = > > for the sole use of the intended recipient(s). Any review or = > > distribution by others is strictly prohibited. If you are not the = > > intended recipient, please contact the sender and delete all copies. > > _______________________________________________ > > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email = > > to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to = > spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. --===============0920394065516173664==--