* NVMe multi-core process
@ 2014-09-01 17:10 harryxiyou
2014-09-01 17:22 ` harryxiyou
0 siblings, 1 reply; 6+ messages in thread
From: harryxiyou @ 2014-09-01 17:10 UTC (permalink / raw)
Dear Keith,
I saw following words in the NVMe specification but I cannot see the
related codes
in the linux NVMe driver.
"Typically the number of command queues created is based on the system
configuration and anticipated workload. For example, on a four core
processor based system, there may be a queue pair per core to avoid
locking and ensure data structures are created in the appropriate
processor core?s cache."
I don't know how to configure NVMe to run it on multiple cores?
Thanks, Harry
^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process
2014-09-01 17:10 NVMe multi-core process harryxiyou
@ 2014-09-01 17:22 ` harryxiyou
2014-09-01 20:52 ` Angelo Brito
0 siblings, 1 reply; 6+ messages in thread
From: harryxiyou @ 2014-09-01 17:22 UTC (permalink / raw)
On Mon, Sep 1, 2014@1:10 PM, harryxiyou <harryxiyou@gmail.com> wrote:
> Dear Keith,
>
> I saw following words in the NVMe specification but I cannot see the
> related codes
> in the linux NVMe driver.
>
> "Typically the number of command queues created is based on the system
> configuration and anticipated workload. For example, on a four core
> processor based system, there may be a queue pair per core to avoid
> locking and ensure data structures are created in the appropriate
> processor core?s cache."
>
>
> I don't know how to configure NVMe to run it on multiple cores?
Maybe, I have found the related codes as follows.
1524 static int __devinit nvme_setup_io_queues(struct nvme_dev *dev)
1525 {
1526 struct pci_dev *pdev = dev->pci_dev;
1527 int result, cpu, i, vecs, nr_io_queues, db_bar_size, q_depth;
1528
1529 nr_io_queues = num_online_cpus();
1530 result = set_queue_count(dev, nr_io_queues);
1531 if (result < 0)
1532 return result;
1533 if (result < nr_io_queues)
1534 nr_io_queues = result;
...
1575 for (i = 0; i < nr_io_queues; i++) {
1576 dev->queues[i + 1] = nvme_create_queue(dev, i + 1, q_depth, i);
1577 if (IS_ERR(dev->queues[i + 1])) {
1578 result = PTR_ERR(dev->queues[i + 1]);
1579 goto free_queues;
1580 }
1581 dev->queue_count++;
1582 }
...
}
Anyway, thanks very much.
Thanks, Harry.
^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process
2014-09-01 17:22 ` harryxiyou
@ 2014-09-01 20:52 ` Angelo Brito
2014-09-02 16:44 ` Keith Busch
2014-09-02 23:57 ` harryxiyou
0 siblings, 2 replies; 6+ messages in thread
From: Angelo Brito @ 2014-09-01 20:52 UTC (permalink / raw)
Hi Harry,
In which version of the driver have you seen this code?
I am working with the kernels 3.10 and 3.15 and never seen more than
the Admin Queue and SQ1.
All my machines have multiple cores and one of them has multiple CPU.
Regards,
Angelo Silva Brito.
Digital Systems Engineer
http://about.me/angelobrito
_________________________________________________
On Mon, Sep 1, 2014@2:22 PM, harryxiyou <harryxiyou@gmail.com> wrote:
> On Mon, Sep 1, 2014@1:10 PM, harryxiyou <harryxiyou@gmail.com> wrote:
>> Dear Keith,
>>
>> I saw following words in the NVMe specification but I cannot see the
>> related codes
>> in the linux NVMe driver.
>>
>> "Typically the number of command queues created is based on the system
>> configuration and anticipated workload. For example, on a four core
>> processor based system, there may be a queue pair per core to avoid
>> locking and ensure data structures are created in the appropriate
>> processor core?s cache."
>>
>>
>> I don't know how to configure NVMe to run it on multiple cores?
>
>
> Maybe, I have found the related codes as follows.
>
> 1524 static int __devinit nvme_setup_io_queues(struct nvme_dev *dev)
> 1525 {
> 1526 struct pci_dev *pdev = dev->pci_dev;
> 1527 int result, cpu, i, vecs, nr_io_queues, db_bar_size, q_depth;
> 1528
> 1529 nr_io_queues = num_online_cpus();
> 1530 result = set_queue_count(dev, nr_io_queues);
> 1531 if (result < 0)
> 1532 return result;
> 1533 if (result < nr_io_queues)
> 1534 nr_io_queues = result;
> ...
> 1575 for (i = 0; i < nr_io_queues; i++) {
> 1576 dev->queues[i + 1] = nvme_create_queue(dev, i + 1, q_depth, i);
> 1577 if (IS_ERR(dev->queues[i + 1])) {
> 1578 result = PTR_ERR(dev->queues[i + 1]);
> 1579 goto free_queues;
> 1580 }
> 1581 dev->queue_count++;
> 1582 }
> ...
> }
>
>
>
> Anyway, thanks very much.
>
>
>
> Thanks, Harry.
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process
2014-09-01 20:52 ` Angelo Brito
@ 2014-09-02 16:44 ` Keith Busch
2014-09-02 20:24 ` Angelo Brito
2014-09-02 23:57 ` harryxiyou
1 sibling, 1 reply; 6+ messages in thread
From: Keith Busch @ 2014-09-02 16:44 UTC (permalink / raw)
On Mon, 1 Sep 2014, Angelo Brito wrote:
> In which version of the driver have you seen this code? I am working with
> the kernels 3.10 and 3.15 and never seen more than the Admin Queue and SQ1.
> All my machines have multiple cores and one of them has multiple CPU.
The linux-nvme driver from the very beginning has supported an IO
queue pair for each CPU if the h/w supports at least as many queues and
interrupt vectors.
^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process
2014-09-02 16:44 ` Keith Busch
@ 2014-09-02 20:24 ` Angelo Brito
0 siblings, 0 replies; 6+ messages in thread
From: Angelo Brito @ 2014-09-02 20:24 UTC (permalink / raw)
Thanks for the feedback Keith!
I have a device here that supports 9 (1 Admin + 8 I/O) Submission
Queues but looking to his doorbell regiters I can see that just SQ1 is
in use. My pc is a Power Edge T600 from Dell which has 24 cores (2
CPU whith 12 cores). We are running on Red Hat Enterprise Linux 6.5
with kernels 3.15 and 3.10.
I have also checked the Get Features for Number of Queues and it
returns 'h00070007 which is zero-based notation. I think something
must be wrong at my configurations. Do you have any ideas of the
reason why the driver never used more than one submission queue?
Some directions on how to debug it would be nice too. ;)
Regards,
Angelo Silva Brito.
Graduate in Engenharia da Computa??o - UFPE
http://about.me/angelobrito
_________________________________________________
On Tue, Sep 2, 2014@1:44 PM, Keith Busch <keith.busch@intel.com> wrote:
> On Mon, 1 Sep 2014, Angelo Brito wrote:
>>
>> In which version of the driver have you seen this code? I am working with
>> the kernels 3.10 and 3.15 and never seen more than the Admin Queue and
>> SQ1.
>> All my machines have multiple cores and one of them has multiple CPU.
>
>
> The linux-nvme driver from the very beginning has supported an IO
> queue pair for each CPU if the h/w supports at least as many queues and
> interrupt vectors.
^ permalink raw reply [flat|nested] 6+ messages in thread
* NVMe multi-core process
2014-09-01 20:52 ` Angelo Brito
2014-09-02 16:44 ` Keith Busch
@ 2014-09-02 23:57 ` harryxiyou
1 sibling, 0 replies; 6+ messages in thread
From: harryxiyou @ 2014-09-02 23:57 UTC (permalink / raw)
On Mon, Sep 1, 2014@4:52 PM, Angelo Brito <asb@cin.ufpe.br> wrote:
> Hi Harry,
Dear Angelo,
>
> In which version of the driver have you seen this code?
I run NVMe upon ESXi 5.x so I use following codes that I changed it a little
http://git.infradead.org/users/kbusch/vmklinux-nvme
> I am working with the kernels 3.10 and 3.15 and never seen more than
> the Admin Queue and SQ1.
> All my machines have multiple cores and one of them has multiple CPU.
Yeah, it real supports multi-core.
Thanks, Harry
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-09-02 23:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-09-01 17:10 NVMe multi-core process harryxiyou
2014-09-01 17:22 ` harryxiyou
2014-09-01 20:52 ` Angelo Brito
2014-09-02 16:44 ` Keith Busch
2014-09-02 20:24 ` Angelo Brito
2014-09-02 23:57 ` harryxiyou
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox