From mboxrd@z Thu Jan 1 00:00:00 1970 From: keith.busch@intel.com (Keith Busch) Date: Wed, 4 Jun 2014 12:28:22 -0600 (MDT) Subject: [PATCH v5] conversion to blk-mq In-Reply-To: <538EE3E6.60408@bjorling.me> References: <1401742510-10827-1-git-send-email-m@bjorling.me> <538E2C2E.1010500@bjorling.me> <538EE3E6.60408@bjorling.me> Message-ID: On Wed, 4 Jun 2014, Matias Bj?rling wrote: > On 06/04/2014 12:27 AM, Keith Busch wrote: >>> On Tue, 3 Jun 2014, Matias Bjorling wrote: >>>> >>>> Keith, will you take the nvmemq_wip_v6 branch for a spin? Thanks! >> >> BTW, if you want to test this out yourself, it's pretty simple to >> recreate. I just run a simple user admin program sending nvme passthrough >> commands in a tight loop, then run: >> >> # echo 1 > /sys/bus/pci/devices//remove > > I can't recreate- I use the nvme_get_feature program to continuously hit the > ioctl path, testing using your nvme qemu branch. Okay, I'll try to fix it. I think there are multiple problems, but the first is that since there is no gendisk associated with the admin_q, the QUEUE_FLAG_INIT_DONE flag is never set, and blk_mq_queue_enter returns successful whenever this flag is not set even though this queue is dying, so we enter with all its invalid pointers. Here's a couple diff's. The first fixes the kernel oops by not entering a dying queue. The second is just a few unrelated clean-ups in nvme-core.c. I still can't complete my current hot-removal test, though; something appears hung, but haven't nailed that down yet. Please let me know what you think! Thanks. diff --git a/block/blk-mq.c b/block/blk-mq.c index d10013b..5a9ae8a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -105,6 +105,10 @@ static int blk_mq_queue_enter(struct request_queue *q) __percpu_counter_add(&q->mq_usage_counter, 1, 1000000); smp_wmb(); /* we have problems to freeze the queue if it's initializing */ + if (blk_queue_dying(q)) { + __percpu_counter_add(&q->mq_usage_counter, -1, 1000000); + ret = -ENODEV; + } if (!blk_queue_bypass(q) || !blk_queue_init_done(q)) return 0; diff --git a/drivers/block/nvme-core.c b/drivers/block/nvme-core.c index 243a5e6..22e9c82 100644 --- a/drivers/block/nvme-core.c +++ b/drivers/block/nvme-core.c @@ -98,7 +98,6 @@ struct nvme_queue { u8 cq_phase; u8 cqe_seen; u8 q_suspended; - cpumask_var_t cpu_mask; struct async_cmd_info cmdinfo; struct blk_mq_hw_ctx *hctx; }; @@ -1055,8 +1054,6 @@ static void nvme_free_queue(struct nvme_queue *nvmeq) (void *)nvmeq->cqes, nvmeq->cq_dma_addr); dma_free_coherent(nvmeq->q_dmadev, SQ_SIZE(nvmeq->q_depth), nvmeq->sq_cmds, nvmeq->sq_dma_addr); - if (nvmeq->qid) - free_cpumask_var(nvmeq->cpu_mask); kfree(nvmeq); } @@ -1066,9 +1063,9 @@ static void nvme_free_queues(struct nvme_dev *dev, int lowest) for (i = dev->queue_count - 1; i >= lowest; i--) { struct nvme_queue *nvmeq = dev->queues[i]; - nvme_free_queue(nvmeq); dev->queue_count--; dev->queues[i] = NULL; + nvme_free_queue(nvmeq); } } @@ -1142,9 +1139,6 @@ static struct nvme_queue *nvme_alloc_queue(struct nvme_dev *dev, int qid, if (!nvmeq->sq_cmds) goto free_cqdma; - if (qid && !zalloc_cpumask_var(&nvmeq->cpu_mask, GFP_KERNEL)) - goto free_sqdma; - nvmeq->q_dmadev = dmadev; nvmeq->dev = dev; snprintf(nvmeq->irqname, sizeof(nvmeq->irqname), "nvme%dq%d", @@ -1162,9 +1156,6 @@ static struct nvme_queue *nvme_alloc_queue(struct nvme_dev *dev, int qid, return nvmeq; - free_sqdma: - dma_free_coherent(dmadev, SQ_SIZE(depth), (void *)nvmeq->sq_cmds, - nvmeq->sq_dma_addr); free_cqdma: dma_free_coherent(dmadev, CQ_SIZE(depth), (void *)nvmeq->cqes, nvmeq->cq_dma_addr);