Re: [SPDK] SPDK & DPDK in one process

[Gregory Etelson <gregory at weka.io>]

[-- Attachment #1: Type: text/plain, Size: 6297 bytes --]

It would be nice though, if SPDK devices initialization was not bound to eal_rte_init() and could be activated from any place in code
The following 2 patches provide this ability

1 DPDP patch - add  rte_eal_pci_probe_driver() function call
rte_eal_pci_probe_driver() iterates though a list of already discovered PCI devices and matches each device with a driver.
In case of a match - device is activated with the driver 

diff --git a/lib/librte_eal/common/eal_common_pci.c b/lib/librte_eal/common/eal_common_pci.c
index 7248c38..056626d 100644
--- a/lib/librte_eal/common/eal_common_pci.c
+++ b/lib/librte_eal/common/eal_common_pci.c
@@ -427,6 +427,20 @@ rte_eal_pci_probe(void)
        return 0;
 }
 
+int rte_eal_pci_probe_driver(struct rte_pci_driver *dr)
+{
+       int rc = -1;
+       struct rte_pci_device *dev = NULL;
+
+       TAILQ_FOREACH(dev, &pci_device_list, next) {
+               rc = rte_eal_pci_probe_one_driver(dr, dev);
+               if (rc > 0) continue;
+               else break;
+       }
+
+       return rc;
+}
+
 /* dump one device */
 static int
 pci_dump_one_device(FILE *f, struct rte_pci_device *dev)
diff --git a/lib/librte_eal/common/include/rte_pci.h b/lib/librte_eal/common/include/rte_pci.h
index fa74962..e904057 100644
--- a/lib/librte_eal/common/include/rte_pci.h
+++ b/lib/librte_eal/common/include/rte_pci.h
@@ -580,6 +580,8 @@ void rte_eal_pci_ioport_read(struct rte_pci_ioport *p,
 void rte_eal_pci_ioport_write(struct rte_pci_ioport *p,
                              const void *data, size_t len, off_t offset);
 
+int rte_eal_pci_probe_driver(struct rte_pci_driver *dr);
+
 #ifdef __cplusplus
 }
 #endif

2 SPDK patch: 

diff --git a/lib/env/pci.c b/lib/env/pci.c                                                                                                                     
index 65137d0..a663861 100644                                                                                                                                  
--- a/lib/env/pci.c                                                                                                                                            
+++ b/lib/env/pci.c                                                                                                                                            
@@ -192,8 +192,7 @@ spdk_pci_enumerate(enum spdk_pci_device_type type,                                                                                         
 #endif                                                                                                                                                        
                                                                                                                                                               
        rte_eal_pci_register(&ctx.driver);                                                                                                                     
-       rc = rte_eal_pci_probe();                                                                                                                              
-       rte_eal_pci_unregister(&ctx.driver);                                                                                                                   
+       rc = rte_eal_pci_probe_driver(&ctx.driver);                                                                                                            
                                                                                                                                                               
        return rc;                                                                                                                                             
 }                                                                                                                                                             

On Tuesday, 25 October 2016 22:53:53 IDT Thomas Monjalon wrote:
> Hi SPDK
> 
> 2016-10-24 23:30, Walker, Benjamin:
> > On Sat, 2016-10-22 at 21:19 +0300, Gregory Etelson wrote:
> > > Hello
> > > 
> > > I need to run DPDK networking and access NVMe device in the same process.
> > > 
> > > rte_eal_init() initializes network PMD drivers normally
> > > But spdk_nvme_probe() activates network devices probe for the second time and
> > > rte exits with error.
> > > 
> > > To resolve this failure I register NVMe controller driver as DPDK PMD.
> > > Now I can pass NVMe device PCI address to EAL initialization with `-w'
> > > parameter along with network devices
> > > PMD template I use is attached below.
> > > 
> > > Is there another way to run DPDK and SPDK in the same process ? 
> > 
> > You found the best way right now, but I'm working with the DPDK community to
> > improve here. It seems like the rte_pci module doesn't handle multiple calls to
> > rte_eal_pci_probe, which is what SPDK is doing. The most important change to
> > DPDK that needs to happen is that rte_eal_pci_probe needs to not call the driver
> > initialization function for devices that already have a driver loaded. That's a
> > very easy fix that solves your problem and I'll be submitting a patch to the
> > DPDK community.
> > 
> > Longer term I think we really need to work with DPDK to clean up some of these
> > PCI interfaces so they're more dynamic and can handle devices and drivers coming
> > and going at run time.
> 
> Yes you are welcome to contribute to DPDK.
> 
> There is a slow work in progress to better design the EAL in order to ease
> integration of new buses and allow true hotplugging.
> We do these core changes step by step:
> 
> After having better designed the device object, we are changing the bus model.
> Instead of having PCI the default bus, and vdev an exception, we need a
> generic bus object to plug PCI, vdev and other buses in it.
> Then we should design a better app event mechanism, usable for hotplug.
> Another step could be an auto-binding.
> And at the end we could be plugged to hardware events like udev.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>