[SPDK] Re: vfio multiple processes

[SPDK] Re: vfio multiple processes

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 1448 bytes --]

Resolved by adding pci whitelist into spdk_env_opts.
Adding whitelist into nvme.conf was not sufficient.

Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarro

From: Nabarro, Tom
Sent: Friday, January 31, 2020 12:42 PM
To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
Subject: vfio multiple processes

Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.

Have any ideas?
(processes are started in normal, non-multiprocess mode)

Thanks in advance

Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[SPDK] Re: vfio multiple processes

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

Excellent!

On 2/1/20, 7:06 AM, "Nabarro, Tom" <tom.nabarro(a)intel.com> wrote:

    Resolved by adding pci whitelist into spdk_env_opts.
    Adding whitelist into nvme.conf was not sufficient.
    
    Regards,
    Tom Nabarro - DCG/ESAD
    M: +44 (0)7786 260986
    Skype: tom.nabarro
    
    From: Nabarro, Tom
    Sent: Friday, January 31, 2020 12:42 PM
    To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
    Subject: vfio multiple processes
    
    Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.
    
    Have any ideas?
    (processes are started in normal, non-multiprocess mode)
    
    Thanks in advance
    
    Best regards,
    Tom Nabarro BEng (hons)
    Extreme Storage Architecture & Development
    Intel Corporation
    E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
    M: +44 (0)7786 260986
    Skype: tom.nabarro
    
    ---------------------------------------------------------------------
    Intel Corporation (UK) Limited
    Registered No. 1134945 (England)
    Registered Office: Pipers Way, Swindon SN3 1RJ
    VAT No: 860 2173 47
    
    This e-mail and any attachments may contain confidential material for
    the sole use of the intended recipient(s). Any review or distribution
    by others is strictly prohibited. If you are not the intended
    recipient, please contact the sender and delete all copies.
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: vfio multiple processes

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 2025 bytes --]

TBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a look.

D.

> -----Original Message-----
> From: Nabarro, Tom <tom.nabarro(a)intel.com>
> Sent: Saturday, February 1, 2020 3:00 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: vfio multiple processes
> 
> Resolved by adding pci whitelist into spdk_env_opts.
> Adding whitelist into nvme.conf was not sufficient.
> 
> Regards,
> Tom Nabarro - DCG/ESAD
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> From: Nabarro, Tom
> Sent: Friday, January 31, 2020 12:42 PM
> To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> Subject: vfio multiple processes
> 
> Can't run multiple SPDK processes each accessing disjoint set of NVMe drives
> using VFIO driver (even as root). Each process seems to try to access all
> /dev/vfio/* device files and get lots of resource busy messages. Sometimes
> one of the processes complaint it can't find the device it's been assigned in
> nvme.conf.
> 
> Have any ideas?
> (processes are started in normal, non-multiprocess mode)
> 
> Thanks in advance
> 
> Best regards,
> Tom Nabarro BEng (hons)
> Extreme Storage Architecture & Development
> Intel Corporation
> E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> M: +44 (0)7786 260986
> Skype: tom.nabarro
> 
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: vfio multiple processes

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 2788 bytes --]

Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while.

D.

> -----Original Message-----
> From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
> Sent: Monday, February 3, 2020 10:14 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: vfio multiple processes
> 
> TBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a
> look.
> 
> D.
> 
> > -----Original Message-----
> > From: Nabarro, Tom <tom.nabarro(a)intel.com>
> > Sent: Saturday, February 1, 2020 3:00 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Re: vfio multiple processes
> >
> > Resolved by adding pci whitelist into spdk_env_opts.
> > Adding whitelist into nvme.conf was not sufficient.
> >
> > Regards,
> > Tom Nabarro - DCG/ESAD
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > From: Nabarro, Tom
> > Sent: Friday, January 31, 2020 12:42 PM
> > To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> > Subject: vfio multiple processes
> >
> > Can't run multiple SPDK processes each accessing disjoint set of NVMe drives
> > using VFIO driver (even as root). Each process seems to try to access all
> > /dev/vfio/* device files and get lots of resource busy messages. Sometimes
> > one of the processes complaint it can't find the device it's been assigned in
> > nvme.conf.
> >
> > Have any ideas?
> > (processes are started in normal, non-multiprocess mode)
> >
> > Thanks in advance
> >
> > Best regards,
> > Tom Nabarro BEng (hons)
> > Extreme Storage Architecture & Development
> > Intel Corporation
> > E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > ---------------------------------------------------------------------
> > Intel Corporation (UK) Limited
> > Registered No. 1134945 (England)
> > Registered Office: Pipers Way, Swindon SN3 1RJ
> > VAT No: 860 2173 47
> >
> > This e-mail and any attachments may contain confidential material for
> > the sole use of the intended recipient(s). Any review or distribution
> > by others is strictly prohibited. If you are not the intended
> > recipient, please contact the sender and delete all copies.
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org
> > To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: vfio multiple processes

[Nabarro, Tom]

[-- Attachment #1: Type: text/plain, Size: 4510 bytes --]

That's useful info, we used to have the primary process (which performed device provisioning) launch the secondary I/O processes (after device provisioning) therefore we could always do discovery through the primary process on devices in use by the secondaries. 

Now we want to run our main processes as non-root so switch to vfio and we do device provisioning through a privileged helper binary (which runs setuid root and not in multiprocess mode) and call out to it from our main control process (not a SPDK process). The control process then forks our I/O processes which run in primary mode (as non-root) so that when we want to interrogate devices being used by our I/O processes we can attach as secondary rather than have to implement some form of IPC.

Regards,
Tom Nabarro - DCG/ESAD
M: +44 (0)7786 260986
Skype: tom.nabarro

-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com> 
Sent: Tuesday, February 4, 2020 12:27 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Re: vfio multiple processes

Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while.

D.

> -----Original Message-----
> From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com>
> Sent: Monday, February 3, 2020 10:14 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: vfio multiple processes
> 
> TBH I'm not sure the whitelist should be required. Sounds like a bug, 
> I'll take a look.
> 
> D.
> 
> > -----Original Message-----
> > From: Nabarro, Tom <tom.nabarro(a)intel.com>
> > Sent: Saturday, February 1, 2020 3:00 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] Re: vfio multiple processes
> >
> > Resolved by adding pci whitelist into spdk_env_opts.
> > Adding whitelist into nvme.conf was not sufficient.
> >
> > Regards,
> > Tom Nabarro - DCG/ESAD
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > From: Nabarro, Tom
> > Sent: Friday, January 31, 2020 12:42 PM
> > To: 'Storage Performance Development Kit' <spdk(a)lists.01.org>
> > Subject: vfio multiple processes
> >
> > Can't run multiple SPDK processes each accessing disjoint set of 
> > NVMe drives using VFIO driver (even as root). Each process seems to 
> > try to access all
> > /dev/vfio/* device files and get lots of resource busy messages. 
> > Sometimes one of the processes complaint it can't find the device 
> > it's been assigned in nvme.conf.
> >
> > Have any ideas?
> > (processes are started in normal, non-multiprocess mode)
> >
> > Thanks in advance
> >
> > Best regards,
> > Tom Nabarro BEng (hons)
> > Extreme Storage Architecture & Development Intel Corporation
> > E: tom.nabarro(a)intel.com<mailto:tom.nabarro(a)intel.com>
> > M: +44 (0)7786 260986
> > Skype: tom.nabarro
> >
> > --------------------------------------------------------------------
> > -
> > Intel Corporation (UK) Limited
> > Registered No. 1134945 (England)
> > Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
> >
> > This e-mail and any attachments may contain confidential material 
> > for the sole use of the intended recipient(s). Any review or 
> > distribution by others is strictly prohibited. If you are not the 
> > intended recipient, please contact the sender and delete all copies.
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email 
> > to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.