[SPDK] Re: vfio multiple processes

[Nabarro, Tom <tom.nabarro at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4510 bytes --]

That's useful info, we used to have the primary process (which performed device provisioning) launch the secondary I/O processes (after device provisioning) therefore we could always do discovery through the primary process on devices in use by the secondaries. 

Now we want to run our main processes as non-root so switch to vfio and we do device provisioning through a privileged helper binary (which runs setuid root and not in multiprocess mode) and call out to it from our main control process (not a SPDK process). The control process then forks our I/O processes which run in primary mode (as non-root) so that when we want to interrogate devices being used by our I/O processes we can attach as secondary rather than have to implement some form of IPC.

Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarro

-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com> 
Sent: Tuesday, February 4, 2020 12:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processes

Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while.

D.

> -----Original Message-----
> From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
> Sent: Monday, February 3, 2020 10:14 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: vfio multiple processes
> 
> TBH I'm not sure the whitelist should be required. Sounds like a bug, 
> I'll take a look.
> 
> D.
> 
> > -----Original Message-----
> > From: Nabarro, Tom <tom.nabarro(a)intel.com>
> > Sent: Saturday, February 1, 2020 3:00 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Re: vfio multiple processes
> >
> > Resolved by adding pci whitelist into spdk_env_opts.
> > Adding whitelist into nvme.conf was not sufficient.
> >
> > Regards,
> > Tom Nabarro - DCG/ESAD
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > From: Nabarro, Tom
> > Sent: Friday, January 31, 2020 12:42 PM
> > To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> > Subject: vfio multiple processes
> >
> > Can't run multiple SPDK processes each accessing disjoint set of 
> > NVMe drives using VFIO driver (even as root). Each process seems to 
> > try to access all
> > /dev/vfio/* device files and get lots of resource busy messages. 
> > Sometimes one of the processes complaint it can't find the device 
> > it's been assigned in nvme.conf.
> >
> > Have any ideas?
> > (processes are started in normal, non-multiprocess mode)
> >
> > Thanks in advance
> >
> > Best regards,
> > Tom Nabarro BEng (hons)
> > Extreme Storage Architecture & Development Intel Corporation
> > E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > --------------------------------------------------------------------
> > -
> > Intel Corporation (UK) Limited
> > Registered No. 1134945 (England)
> > Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> >
> > This e-mail and any attachments may contain confidential material 
> > for the sole use of the intended recipient(s). Any review or 
> > distribution by others is strictly prohibited. If you are not the 
> > intended recipient, please contact the sender and delete all copies.
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email 
> > to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.