* Re: [PATCH V2] nvme-pci: assign separate irq vectors for adminq and ioq0 [not found] ` <20180309172445.GC14765@localhost.localdomain> @ 2018-03-12 9:09 ` Ming Lei 2018-10-08 5:05 ` nvme-pci: number of queues off by one Prasun Ratn 0 siblings, 1 reply; 6+ messages in thread From: Ming Lei @ 2018-03-12 9:09 UTC (permalink / raw) To: Keith Busch Cc: Christoph Hellwig, sagi, linux-kernel, linux-nvme, axboe, Jianchao Wang, linux-block, Thomas Gleixner On Fri, Mar 09, 2018 at 10:24:45AM -0700, Keith Busch wrote: > On Thu, Mar 08, 2018 at 08:42:20AM +0100, Christoph Hellwig wrote: > > > > So I suspect we'll need to go with a patch like this, just with a way > > better changelog. > > I have to agree this is required for that use case. I'll run some > quick tests and propose an alternate changelog. > > Longer term, the current way we're including offline present cpus either > (a) has the driver allocate resources it can't use or (b) spreads the > ones it can use thinner than they need to be. Why don't we rerun the > irq spread under a hot cpu notifier for only online CPUs? 4b855ad371 ("blk-mq: Create hctx for each present CPU") removes handling mapping change via hot cpu notifier. Not only code is cleaned up, but also fixes very complicated queue dependency issue: - loop/dm-rq queue depends on underlying queue - for NVMe, IO queue depends on admin queue If freezing queue can be avoided in CPU notifier, it should be fine to do that, otherwise it need to be avoided. Thanks, Ming ^ permalink raw reply [flat|nested] 6+ messages in thread
* nvme-pci: number of queues off by one 2018-03-12 9:09 ` [PATCH V2] nvme-pci: assign separate irq vectors for adminq and ioq0 Ming Lei @ 2018-10-08 5:05 ` Prasun Ratn 2018-10-08 5:59 ` Dongli Zhang 0 siblings, 1 reply; 6+ messages in thread From: Prasun Ratn @ 2018-10-08 5:05 UTC (permalink / raw) To: ming.lei Cc: keith.busch, hch, sagi, linux-nvme, axboe, jianchao.w.wang, linux-block, tglx Hi I have an NVMe SSD that has 8 hw queues and on older kernels I see all 8 show up. However on a recent kernel (I tried 4.18), I only see 7. Is this a known issue? $ uname -r 4.14.1-1.el7.elrepo.x86_64 $ ls /sys/block/nvme*n1/mq/*/cpu_list /sys/block/nvme0n1/mq/0/cpu_list /sys/block/nvme0n1/mq/1/cpu_list /sys/block/nvme0n1/mq/2/cpu_list /sys/block/nvme0n1/mq/3/cpu_list /sys/block/nvme0n1/mq/4/cpu_list /sys/block/nvme0n1/mq/5/cpu_list /sys/block/nvme0n1/mq/6/cpu_list /sys/block/nvme0n1/mq/7/cpu_list /sys/block/nvme1n1/mq/0/cpu_list /sys/block/nvme1n1/mq/1/cpu_list /sys/block/nvme1n1/mq/2/cpu_list /sys/block/nvme1n1/mq/3/cpu_list /sys/block/nvme1n1/mq/4/cpu_list /sys/block/nvme1n1/mq/5/cpu_list /sys/block/nvme1n1/mq/6/cpu_list /sys/block/nvme1n1/mq/7/cpu_list /sys/block/nvme2n1/mq/0/cpu_list /sys/block/nvme2n1/mq/1/cpu_list /sys/block/nvme2n1/mq/2/cpu_list /sys/block/nvme2n1/mq/3/cpu_list /sys/block/nvme2n1/mq/4/cpu_list /sys/block/nvme2n1/mq/5/cpu_list /sys/block/nvme2n1/mq/6/cpu_list /sys/block/nvme2n1/mq/7/cpu_list /sys/block/nvme3n1/mq/0/cpu_list /sys/block/nvme3n1/mq/1/cpu_list /sys/block/nvme3n1/mq/2/cpu_list /sys/block/nvme3n1/mq/3/cpu_list /sys/block/nvme3n1/mq/4/cpu_list /sys/block/nvme3n1/mq/5/cpu_list /sys/block/nvme3n1/mq/6/cpu_list /sys/block/nvme3n1/mq/7/cpu_list $ uname -r 4.18.10-1.el7.elrepo.x86_64 $ ls /sys/block/nvme*n1/mq/*/cpu_list /sys/block/nvme0n1/mq/0/cpu_list /sys/block/nvme0n1/mq/1/cpu_list /sys/block/nvme0n1/mq/2/cpu_list /sys/block/nvme0n1/mq/3/cpu_list /sys/block/nvme0n1/mq/4/cpu_list /sys/block/nvme0n1/mq/5/cpu_list /sys/block/nvme0n1/mq/6/cpu_list /sys/block/nvme1n1/mq/0/cpu_list /sys/block/nvme1n1/mq/1/cpu_list /sys/block/nvme1n1/mq/2/cpu_list /sys/block/nvme1n1/mq/3/cpu_list /sys/block/nvme1n1/mq/4/cpu_list /sys/block/nvme1n1/mq/5/cpu_list /sys/block/nvme1n1/mq/6/cpu_list /sys/block/nvme2n1/mq/0/cpu_list /sys/block/nvme2n1/mq/1/cpu_list /sys/block/nvme2n1/mq/2/cpu_list /sys/block/nvme2n1/mq/3/cpu_list /sys/block/nvme2n1/mq/4/cpu_list /sys/block/nvme2n1/mq/5/cpu_list /sys/block/nvme2n1/mq/6/cpu_list /sys/block/nvme3n1/mq/0/cpu_list /sys/block/nvme3n1/mq/1/cpu_list /sys/block/nvme3n1/mq/2/cpu_list /sys/block/nvme3n1/mq/3/cpu_list /sys/block/nvme3n1/mq/4/cpu_list /sys/block/nvme3n1/mq/5/cpu_list /sys/block/nvme3n1/mq/6/cpu_list ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: nvme-pci: number of queues off by one 2018-10-08 5:05 ` nvme-pci: number of queues off by one Prasun Ratn @ 2018-10-08 5:59 ` Dongli Zhang 2018-10-08 6:58 ` Dongli Zhang 2018-10-08 10:19 ` Ming Lei 0 siblings, 2 replies; 6+ messages in thread From: Dongli Zhang @ 2018-10-08 5:59 UTC (permalink / raw) To: Prasun Ratn, ming.lei Cc: keith.busch, hch, sagi, linux-nvme, axboe, jianchao.w.wang, linux-block, tglx I can reproduce with qemu: # ls /sys/block/nvme*n1/mq/*/cpu_list /sys/block/nvme0n1/mq/0/cpu_list /sys/block/nvme0n1/mq/1/cpu_list /sys/block/nvme0n1/mq/2/cpu_list /sys/block/nvme0n1/mq/3/cpu_list /sys/block/nvme0n1/mq/4/cpu_list /sys/block/nvme0n1/mq/5/cpu_list /sys/block/nvme0n1/mq/6/cpu_list Here is the qemu cmdline emulating 8-queue nvme while the VM has 12 cpu: # qemu-system-x86_64 -m 4096 -smp 12 \ -kernel /path-to-kernel/linux-4.18.10/arch/x86_64/boot/bzImage \ -hda /path-to-img/ubuntu1804.qcow2 \ -append "root=/dev/sda1 init=/sbin/init text" -enable-kvm \ -net nic -net user,hostfwd=tcp::5022-:22 \ -device nvme,drive=nvme1,serial=deadbeaf1,num_queues=8 \ -drive file=/path-to-img/nvme.disk,if=none,id=nvme1 Dongli Zhang On 10/08/2018 01:05 PM, Prasun Ratn wrote: > Hi > > I have an NVMe SSD that has 8 hw queues and on older kernels I see all > 8 show up. However on a recent kernel (I tried 4.18), I only see 7. Is > this a known issue? > > $ uname -r > 4.14.1-1.el7.elrepo.x86_64 > > $ ls /sys/block/nvme*n1/mq/*/cpu_list > /sys/block/nvme0n1/mq/0/cpu_list > /sys/block/nvme0n1/mq/1/cpu_list > /sys/block/nvme0n1/mq/2/cpu_list > /sys/block/nvme0n1/mq/3/cpu_list > /sys/block/nvme0n1/mq/4/cpu_list > /sys/block/nvme0n1/mq/5/cpu_list > /sys/block/nvme0n1/mq/6/cpu_list > /sys/block/nvme0n1/mq/7/cpu_list > /sys/block/nvme1n1/mq/0/cpu_list > /sys/block/nvme1n1/mq/1/cpu_list > /sys/block/nvme1n1/mq/2/cpu_list > /sys/block/nvme1n1/mq/3/cpu_list > /sys/block/nvme1n1/mq/4/cpu_list > /sys/block/nvme1n1/mq/5/cpu_list > /sys/block/nvme1n1/mq/6/cpu_list > /sys/block/nvme1n1/mq/7/cpu_list > /sys/block/nvme2n1/mq/0/cpu_list > /sys/block/nvme2n1/mq/1/cpu_list > /sys/block/nvme2n1/mq/2/cpu_list > /sys/block/nvme2n1/mq/3/cpu_list > /sys/block/nvme2n1/mq/4/cpu_list > /sys/block/nvme2n1/mq/5/cpu_list > /sys/block/nvme2n1/mq/6/cpu_list > /sys/block/nvme2n1/mq/7/cpu_list > /sys/block/nvme3n1/mq/0/cpu_list > /sys/block/nvme3n1/mq/1/cpu_list > /sys/block/nvme3n1/mq/2/cpu_list > /sys/block/nvme3n1/mq/3/cpu_list > /sys/block/nvme3n1/mq/4/cpu_list > /sys/block/nvme3n1/mq/5/cpu_list > /sys/block/nvme3n1/mq/6/cpu_list > /sys/block/nvme3n1/mq/7/cpu_list > > > $ uname -r > 4.18.10-1.el7.elrepo.x86_64 > > $ ls /sys/block/nvme*n1/mq/*/cpu_list > /sys/block/nvme0n1/mq/0/cpu_list > /sys/block/nvme0n1/mq/1/cpu_list > /sys/block/nvme0n1/mq/2/cpu_list > /sys/block/nvme0n1/mq/3/cpu_list > /sys/block/nvme0n1/mq/4/cpu_list > /sys/block/nvme0n1/mq/5/cpu_list > /sys/block/nvme0n1/mq/6/cpu_list > /sys/block/nvme1n1/mq/0/cpu_list > /sys/block/nvme1n1/mq/1/cpu_list > /sys/block/nvme1n1/mq/2/cpu_list > /sys/block/nvme1n1/mq/3/cpu_list > /sys/block/nvme1n1/mq/4/cpu_list > /sys/block/nvme1n1/mq/5/cpu_list > /sys/block/nvme1n1/mq/6/cpu_list > /sys/block/nvme2n1/mq/0/cpu_list > /sys/block/nvme2n1/mq/1/cpu_list > /sys/block/nvme2n1/mq/2/cpu_list > /sys/block/nvme2n1/mq/3/cpu_list > /sys/block/nvme2n1/mq/4/cpu_list > /sys/block/nvme2n1/mq/5/cpu_list > /sys/block/nvme2n1/mq/6/cpu_list > /sys/block/nvme3n1/mq/0/cpu_list > /sys/block/nvme3n1/mq/1/cpu_list > /sys/block/nvme3n1/mq/2/cpu_list > /sys/block/nvme3n1/mq/3/cpu_list > /sys/block/nvme3n1/mq/4/cpu_list > /sys/block/nvme3n1/mq/5/cpu_list > /sys/block/nvme3n1/mq/6/cpu_list > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: nvme-pci: number of queues off by one 2018-10-08 5:59 ` Dongli Zhang @ 2018-10-08 6:58 ` Dongli Zhang 2018-10-08 14:54 ` Keith Busch 2018-10-08 10:19 ` Ming Lei 1 sibling, 1 reply; 6+ messages in thread From: Dongli Zhang @ 2018-10-08 6:58 UTC (permalink / raw) To: Prasun Ratn, ming.lei, jianchao.w.wang Cc: keith.busch, hch, sagi, linux-nvme, axboe, linux-block, tglx I got the same result when emulating nvme with qemu: the VM has 12 cpu, while the num_queues of nvme is 8. # uname -r 4.14.1 # ll /sys/block/nvme*n1/mq/*/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/0/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/1/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/2/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/3/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/4/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/5/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/6/cpu_list # uname -r 4.18.10 # ll /sys/block/nvme*n1/mq/*/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/0/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/1/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/2/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/3/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/4/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/5/cpu_list -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/6/cpu_list >From below qemu source code, when n->num_queues is 8, the handler of NVME_FEAT_NUM_QUEUES returns 0x60006. 719 static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) 720 { 721 uint32_t dw10 = le32_to_cpu(cmd->cdw10); 722 uint32_t dw11 = le32_to_cpu(cmd->cdw11); 723 724 switch (dw10) { 725 case NVME_VOLATILE_WRITE_CACHE: 726 blk_set_enable_write_cache(n->conf.blk, dw11 & 1); 727 break; 728 case NVME_NUMBER_OF_QUEUES: 729 trace_nvme_setfeat_numq((dw11 & 0xFFFF) + 1, 730 ((dw11 >> 16) & 0xFFFF) + 1, 731 n->num_queues - 1, n->num_queues - 1); 732 req->cqe.result = 733 cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); ----> returns 0x60006 when num_queues is 8. Finally, nr_io_queues is set to 6+1=7 in nvme_set_queue_count() in VM kernel. I do not know how to paraphrase this in the world of nvme. Dongli Zhang On 10/08/2018 01:59 PM, Dongli Zhang wrote: > I can reproduce with qemu: > > # ls /sys/block/nvme*n1/mq/*/cpu_list > /sys/block/nvme0n1/mq/0/cpu_list > /sys/block/nvme0n1/mq/1/cpu_list > /sys/block/nvme0n1/mq/2/cpu_list > /sys/block/nvme0n1/mq/3/cpu_list > /sys/block/nvme0n1/mq/4/cpu_list > /sys/block/nvme0n1/mq/5/cpu_list > /sys/block/nvme0n1/mq/6/cpu_list > > Here is the qemu cmdline emulating 8-queue nvme while the VM has 12 cpu: > > # qemu-system-x86_64 -m 4096 -smp 12 \ > -kernel /path-to-kernel/linux-4.18.10/arch/x86_64/boot/bzImage \ > -hda /path-to-img/ubuntu1804.qcow2 \ > -append "root=/dev/sda1 init=/sbin/init text" -enable-kvm \ > -net nic -net user,hostfwd=tcp::5022-:22 \ > -device nvme,drive=nvme1,serial=deadbeaf1,num_queues=8 \ > -drive file=/path-to-img/nvme.disk,if=none,id=nvme1 > > Dongli Zhang > > > On 10/08/2018 01:05 PM, Prasun Ratn wrote: >> Hi >> >> I have an NVMe SSD that has 8 hw queues and on older kernels I see all >> 8 show up. However on a recent kernel (I tried 4.18), I only see 7. Is >> this a known issue? >> >> $ uname -r >> 4.14.1-1.el7.elrepo.x86_64 >> >> $ ls /sys/block/nvme*n1/mq/*/cpu_list >> /sys/block/nvme0n1/mq/0/cpu_list >> /sys/block/nvme0n1/mq/1/cpu_list >> /sys/block/nvme0n1/mq/2/cpu_list >> /sys/block/nvme0n1/mq/3/cpu_list >> /sys/block/nvme0n1/mq/4/cpu_list >> /sys/block/nvme0n1/mq/5/cpu_list >> /sys/block/nvme0n1/mq/6/cpu_list >> /sys/block/nvme0n1/mq/7/cpu_list >> /sys/block/nvme1n1/mq/0/cpu_list >> /sys/block/nvme1n1/mq/1/cpu_list >> /sys/block/nvme1n1/mq/2/cpu_list >> /sys/block/nvme1n1/mq/3/cpu_list >> /sys/block/nvme1n1/mq/4/cpu_list >> /sys/block/nvme1n1/mq/5/cpu_list >> /sys/block/nvme1n1/mq/6/cpu_list >> /sys/block/nvme1n1/mq/7/cpu_list >> /sys/block/nvme2n1/mq/0/cpu_list >> /sys/block/nvme2n1/mq/1/cpu_list >> /sys/block/nvme2n1/mq/2/cpu_list >> /sys/block/nvme2n1/mq/3/cpu_list >> /sys/block/nvme2n1/mq/4/cpu_list >> /sys/block/nvme2n1/mq/5/cpu_list >> /sys/block/nvme2n1/mq/6/cpu_list >> /sys/block/nvme2n1/mq/7/cpu_list >> /sys/block/nvme3n1/mq/0/cpu_list >> /sys/block/nvme3n1/mq/1/cpu_list >> /sys/block/nvme3n1/mq/2/cpu_list >> /sys/block/nvme3n1/mq/3/cpu_list >> /sys/block/nvme3n1/mq/4/cpu_list >> /sys/block/nvme3n1/mq/5/cpu_list >> /sys/block/nvme3n1/mq/6/cpu_list >> /sys/block/nvme3n1/mq/7/cpu_list >> >> >> $ uname -r >> 4.18.10-1.el7.elrepo.x86_64 >> >> $ ls /sys/block/nvme*n1/mq/*/cpu_list >> /sys/block/nvme0n1/mq/0/cpu_list >> /sys/block/nvme0n1/mq/1/cpu_list >> /sys/block/nvme0n1/mq/2/cpu_list >> /sys/block/nvme0n1/mq/3/cpu_list >> /sys/block/nvme0n1/mq/4/cpu_list >> /sys/block/nvme0n1/mq/5/cpu_list >> /sys/block/nvme0n1/mq/6/cpu_list >> /sys/block/nvme1n1/mq/0/cpu_list >> /sys/block/nvme1n1/mq/1/cpu_list >> /sys/block/nvme1n1/mq/2/cpu_list >> /sys/block/nvme1n1/mq/3/cpu_list >> /sys/block/nvme1n1/mq/4/cpu_list >> /sys/block/nvme1n1/mq/5/cpu_list >> /sys/block/nvme1n1/mq/6/cpu_list >> /sys/block/nvme2n1/mq/0/cpu_list >> /sys/block/nvme2n1/mq/1/cpu_list >> /sys/block/nvme2n1/mq/2/cpu_list >> /sys/block/nvme2n1/mq/3/cpu_list >> /sys/block/nvme2n1/mq/4/cpu_list >> /sys/block/nvme2n1/mq/5/cpu_list >> /sys/block/nvme2n1/mq/6/cpu_list >> /sys/block/nvme3n1/mq/0/cpu_list >> /sys/block/nvme3n1/mq/1/cpu_list >> /sys/block/nvme3n1/mq/2/cpu_list >> /sys/block/nvme3n1/mq/3/cpu_list >> /sys/block/nvme3n1/mq/4/cpu_list >> /sys/block/nvme3n1/mq/5/cpu_list >> /sys/block/nvme3n1/mq/6/cpu_list >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: nvme-pci: number of queues off by one 2018-10-08 6:58 ` Dongli Zhang @ 2018-10-08 14:54 ` Keith Busch 0 siblings, 0 replies; 6+ messages in thread From: Keith Busch @ 2018-10-08 14:54 UTC (permalink / raw) To: Dongli Zhang Cc: Prasun Ratn, ming.lei, jianchao.w.wang, hch, sagi, linux-nvme, axboe, linux-block, tglx On Mon, Oct 08, 2018 at 02:58:21PM +0800, Dongli Zhang wrote: > I got the same result when emulating nvme with qemu: the VM has 12 cpu, while > the num_queues of nvme is 8. > > # uname -r > 4.14.1 > # ll /sys/block/nvme*n1/mq/*/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/0/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/1/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/2/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/3/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/4/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/5/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:30 /sys/block/nvme0n1/mq/6/cpu_list > > > # uname -r > 4.18.10 > # ll /sys/block/nvme*n1/mq/*/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/0/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/1/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/2/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/3/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/4/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/5/cpu_list > -r--r--r-- 1 root root 4096 Oct 8 14:34 /sys/block/nvme0n1/mq/6/cpu_list > > From below qemu source code, when n->num_queues is 8, the handler of > NVME_FEAT_NUM_QUEUES returns 0x60006. > > 719 static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeCmd *cmd, NvmeRequest *req) > 720 { > 721 uint32_t dw10 = le32_to_cpu(cmd->cdw10); > 722 uint32_t dw11 = le32_to_cpu(cmd->cdw11); > 723 > 724 switch (dw10) { > 725 case NVME_VOLATILE_WRITE_CACHE: > 726 blk_set_enable_write_cache(n->conf.blk, dw11 & 1); > 727 break; > 728 case NVME_NUMBER_OF_QUEUES: > 729 trace_nvme_setfeat_numq((dw11 & 0xFFFF) + 1, > 730 ((dw11 >> 16) & 0xFFFF) + 1, > 731 n->num_queues - 1, n->num_queues - 1); > 732 req->cqe.result = > 733 cpu_to_le32((n->num_queues - 2) | ((n->num_queues - 2) << 16)); > ----> returns 0x60006 when num_queues is 8. > > > Finally, nr_io_queues is set to 6+1=7 in nvme_set_queue_count() in VM kernel. > > I do not know how to paraphrase this in the world of nvme. > > Dongli Zhang > > On 10/08/2018 01:59 PM, Dongli Zhang wrote: > > I can reproduce with qemu: > > > > # ls /sys/block/nvme*n1/mq/*/cpu_list > > /sys/block/nvme0n1/mq/0/cpu_list > > /sys/block/nvme0n1/mq/1/cpu_list > > /sys/block/nvme0n1/mq/2/cpu_list > > /sys/block/nvme0n1/mq/3/cpu_list > > /sys/block/nvme0n1/mq/4/cpu_list > > /sys/block/nvme0n1/mq/5/cpu_list > > /sys/block/nvme0n1/mq/6/cpu_list > > > > Here is the qemu cmdline emulating 8-queue nvme while the VM has 12 cpu: > > > > # qemu-system-x86_64 -m 4096 -smp 12 \ > > -kernel /path-to-kernel/linux-4.18.10/arch/x86_64/boot/bzImage \ > > -hda /path-to-img/ubuntu1804.qcow2 \ > > -append "root=/dev/sda1 init=/sbin/init text" -enable-kvm \ > > -net nic -net user,hostfwd=tcp::5022-:22 \ > > -device nvme,drive=nvme1,serial=deadbeaf1,num_queues=8 \ > > -drive file=/path-to-img/nvme.disk,if=none,id=nvme1 > > > > Dongli Zhang Qemu counts one of those queues as the admin queue. > > On 10/08/2018 01:05 PM, Prasun Ratn wrote: > >> Hi > >> > >> I have an NVMe SSD that has 8 hw queues and on older kernels I see all > >> 8 show up. However on a recent kernel (I tried 4.18), I only see 7. Is > >> this a known issue? That probably means you only have 8 MSI-x vectors, one of which is reserved for the admin queue. We used to share an IO vector with the admin queue, however some people figured out you can break your controller that way with the linux irq spread. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: nvme-pci: number of queues off by one 2018-10-08 5:59 ` Dongli Zhang 2018-10-08 6:58 ` Dongli Zhang @ 2018-10-08 10:19 ` Ming Lei 1 sibling, 0 replies; 6+ messages in thread From: Ming Lei @ 2018-10-08 10:19 UTC (permalink / raw) To: Dongli Zhang Cc: Prasun Ratn, axboe, sagi, linux-nvme, keith.busch, linux-block, jianchao.w.wang, tglx, hch On Mon, Oct 08, 2018 at 01:59:05PM +0800, Dongli Zhang wrote: > I can reproduce with qemu: > > # ls /sys/block/nvme*n1/mq/*/cpu_list > /sys/block/nvme0n1/mq/0/cpu_list > /sys/block/nvme0n1/mq/1/cpu_list > /sys/block/nvme0n1/mq/2/cpu_list > /sys/block/nvme0n1/mq/3/cpu_list > /sys/block/nvme0n1/mq/4/cpu_list > /sys/block/nvme0n1/mq/5/cpu_list > /sys/block/nvme0n1/mq/6/cpu_list > > Here is the qemu cmdline emulating 8-queue nvme while the VM has 12 cpu: > > # qemu-system-x86_64 -m 4096 -smp 12 \ > -kernel /path-to-kernel/linux-4.18.10/arch/x86_64/boot/bzImage \ > -hda /path-to-img/ubuntu1804.qcow2 \ > -append "root=/dev/sda1 init=/sbin/init text" -enable-kvm \ > -net nic -net user,hostfwd=tcp::5022-:22 \ > -device nvme,drive=nvme1,serial=deadbeaf1,num_queues=8 \ > -drive file=/path-to-img/nvme.disk,if=none,id=nvme1 This 'issue' can be reproduced on v4.14 too. Thanks, Ming ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-10-08 22:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1519832921-13915-1-git-send-email-jianchao.w.wang@oracle.com>
[not found] ` <20180228164726.GB16536@lst.de>
[not found] ` <20180301150329.GB6795@ming.t460p>
[not found] ` <20180301161042.GA14799@localhost.localdomain>
[not found] ` <20180308074220.GC15748@lst.de>
[not found] ` <20180309172445.GC14765@localhost.localdomain>
2018-03-12 9:09 ` [PATCH V2] nvme-pci: assign separate irq vectors for adminq and ioq0 Ming Lei
2018-10-08 5:05 ` nvme-pci: number of queues off by one Prasun Ratn
2018-10-08 5:59 ` Dongli Zhang
2018-10-08 6:58 ` Dongli Zhang
2018-10-08 14:54 ` Keith Busch
2018-10-08 10:19 ` Ming Lei
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox