* NVMe multi-core process @ 2014-09-01 17:10 harryxiyou 2014-09-01 17:22 ` harryxiyou 0 siblings, 1 reply; 6+ messages in thread From: harryxiyou @ 2014-09-01 17:10 UTC (permalink / raw) Dear Keith, I saw following words in the NVMe specification but I cannot see the related codes in the linux NVMe driver. "Typically the number of command queues created is based on the system configuration and anticipated workload. For example, on a four core processor based system, there may be a queue pair per core to avoid locking and ensure data structures are created in the appropriate processor core?s cache." I don't know how to configure NVMe to run it on multiple cores? Thanks, Harry ^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process 2014-09-01 17:10 NVMe multi-core process harryxiyou @ 2014-09-01 17:22 ` harryxiyou 2014-09-01 20:52 ` Angelo Brito 0 siblings, 1 reply; 6+ messages in thread From: harryxiyou @ 2014-09-01 17:22 UTC (permalink / raw) On Mon, Sep 1, 2014@1:10 PM, harryxiyou <harryxiyou@gmail.com> wrote: > Dear Keith, > > I saw following words in the NVMe specification but I cannot see the > related codes > in the linux NVMe driver. > > "Typically the number of command queues created is based on the system > configuration and anticipated workload. For example, on a four core > processor based system, there may be a queue pair per core to avoid > locking and ensure data structures are created in the appropriate > processor core?s cache." > > > I don't know how to configure NVMe to run it on multiple cores? Maybe, I have found the related codes as follows. 1524 static int __devinit nvme_setup_io_queues(struct nvme_dev *dev) 1525 { 1526 struct pci_dev *pdev = dev->pci_dev; 1527 int result, cpu, i, vecs, nr_io_queues, db_bar_size, q_depth; 1528 1529 nr_io_queues = num_online_cpus(); 1530 result = set_queue_count(dev, nr_io_queues); 1531 if (result < 0) 1532 return result; 1533 if (result < nr_io_queues) 1534 nr_io_queues = result; ... 1575 for (i = 0; i < nr_io_queues; i++) { 1576 dev->queues[i + 1] = nvme_create_queue(dev, i + 1, q_depth, i); 1577 if (IS_ERR(dev->queues[i + 1])) { 1578 result = PTR_ERR(dev->queues[i + 1]); 1579 goto free_queues; 1580 } 1581 dev->queue_count++; 1582 } ... } Anyway, thanks very much. Thanks, Harry. ^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process 2014-09-01 17:22 ` harryxiyou @ 2014-09-01 20:52 ` Angelo Brito 2014-09-02 16:44 ` Keith Busch 2014-09-02 23:57 ` harryxiyou 0 siblings, 2 replies; 6+ messages in thread From: Angelo Brito @ 2014-09-01 20:52 UTC (permalink / raw) Hi Harry, In which version of the driver have you seen this code? I am working with the kernels 3.10 and 3.15 and never seen more than the Admin Queue and SQ1. All my machines have multiple cores and one of them has multiple CPU. Regards, Angelo Silva Brito. Digital Systems Engineer http://about.me/angelobrito _________________________________________________ On Mon, Sep 1, 2014@2:22 PM, harryxiyou <harryxiyou@gmail.com> wrote: > On Mon, Sep 1, 2014@1:10 PM, harryxiyou <harryxiyou@gmail.com> wrote: >> Dear Keith, >> >> I saw following words in the NVMe specification but I cannot see the >> related codes >> in the linux NVMe driver. >> >> "Typically the number of command queues created is based on the system >> configuration and anticipated workload. For example, on a four core >> processor based system, there may be a queue pair per core to avoid >> locking and ensure data structures are created in the appropriate >> processor core?s cache." >> >> >> I don't know how to configure NVMe to run it on multiple cores? > > > Maybe, I have found the related codes as follows. > > 1524 static int __devinit nvme_setup_io_queues(struct nvme_dev *dev) > 1525 { > 1526 struct pci_dev *pdev = dev->pci_dev; > 1527 int result, cpu, i, vecs, nr_io_queues, db_bar_size, q_depth; > 1528 > 1529 nr_io_queues = num_online_cpus(); > 1530 result = set_queue_count(dev, nr_io_queues); > 1531 if (result < 0) > 1532 return result; > 1533 if (result < nr_io_queues) > 1534 nr_io_queues = result; > ... > 1575 for (i = 0; i < nr_io_queues; i++) { > 1576 dev->queues[i + 1] = nvme_create_queue(dev, i + 1, q_depth, i); > 1577 if (IS_ERR(dev->queues[i + 1])) { > 1578 result = PTR_ERR(dev->queues[i + 1]); > 1579 goto free_queues; > 1580 } > 1581 dev->queue_count++; > 1582 } > ... > } > > > > Anyway, thanks very much. > > > > Thanks, Harry. > > _______________________________________________ > Linux-nvme mailing list > Linux-nvme at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-nvme ^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process 2014-09-01 20:52 ` Angelo Brito @ 2014-09-02 16:44 ` Keith Busch 2014-09-02 20:24 ` Angelo Brito 2014-09-02 23:57 ` harryxiyou 1 sibling, 1 reply; 6+ messages in thread From: Keith Busch @ 2014-09-02 16:44 UTC (permalink / raw) On Mon, 1 Sep 2014, Angelo Brito wrote: > In which version of the driver have you seen this code? I am working with > the kernels 3.10 and 3.15 and never seen more than the Admin Queue and SQ1. > All my machines have multiple cores and one of them has multiple CPU. The linux-nvme driver from the very beginning has supported an IO queue pair for each CPU if the h/w supports at least as many queues and interrupt vectors. ^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process 2014-09-02 16:44 ` Keith Busch @ 2014-09-02 20:24 ` Angelo Brito 0 siblings, 0 replies; 6+ messages in thread From: Angelo Brito @ 2014-09-02 20:24 UTC (permalink / raw) Thanks for the feedback Keith! I have a device here that supports 9 (1 Admin + 8 I/O) Submission Queues but looking to his doorbell regiters I can see that just SQ1 is in use. My pc is a Power Edge T600 from Dell which has 24 cores (2 CPU whith 12 cores). We are running on Red Hat Enterprise Linux 6.5 with kernels 3.15 and 3.10. I have also checked the Get Features for Number of Queues and it returns 'h00070007 which is zero-based notation. I think something must be wrong at my configurations. Do you have any ideas of the reason why the driver never used more than one submission queue? Some directions on how to debug it would be nice too. ;) Regards, Angelo Silva Brito. Graduate in Engenharia da Computa??o - UFPE http://about.me/angelobrito _________________________________________________ On Tue, Sep 2, 2014@1:44 PM, Keith Busch <keith.busch@intel.com> wrote: > On Mon, 1 Sep 2014, Angelo Brito wrote: >> >> In which version of the driver have you seen this code? I am working with >> the kernels 3.10 and 3.15 and never seen more than the Admin Queue and >> SQ1. >> All my machines have multiple cores and one of them has multiple CPU. > > > The linux-nvme driver from the very beginning has supported an IO > queue pair for each CPU if the h/w supports at least as many queues and > interrupt vectors. ^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process 2014-09-01 20:52 ` Angelo Brito 2014-09-02 16:44 ` Keith Busch @ 2014-09-02 23:57 ` harryxiyou 1 sibling, 0 replies; 6+ messages in thread From: harryxiyou @ 2014-09-02 23:57 UTC (permalink / raw) On Mon, Sep 1, 2014@4:52 PM, Angelo Brito <asb@cin.ufpe.br> wrote: > Hi Harry, Dear Angelo, > > In which version of the driver have you seen this code? I run NVMe upon ESXi 5.x so I use following codes that I changed it a little http://git.infradead.org/users/kbusch/vmklinux-nvme > I am working with the kernels 3.10 and 3.15 and never seen more than > the Admin Queue and SQ1. > All my machines have multiple cores and one of them has multiple CPU. Yeah, it real supports multi-core. Thanks, Harry ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-09-02 23:57 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-09-01 17:10 NVMe multi-core process harryxiyou 2014-09-01 17:22 ` harryxiyou 2014-09-01 20:52 ` Angelo Brito 2014-09-02 16:44 ` Keith Busch 2014-09-02 20:24 ` Angelo Brito 2014-09-02 23:57 ` harryxiyou
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox