Re: how can one drain MQ request queue ?

[Max Gurtovoy <maxg@mellanox.com>]

On 2/22/2018 4:59 AM, Ming Lei wrote:
> Hi Max,

Hi Ming,

> 
> On Tue, Feb 20, 2018 at 11:56:07AM +0200, Max Gurtovoy wrote:
>> hi all,
>> is there a way to drain a blk-mq based request queue (similar to
>> blk_drain_queue for non MQ) ?
> 
> Generally speaking, blk_mq_freeze_queue() should be fine to drain blk-mq
> based request queue, but it may not work well when the hardware is broken.

I tried that, but the path failover takes ~cmd_timeout seconds and this 
is not good enough...

> 
>>
>> I try to fix the following situation:
>> Running DM-multipath over NVMEoF/RDMA block devices, toggling the switch
>> ports during traffic using fio and making sure the traffic never fails.
>>
>> when the switch port goes down the initiator driver start an error recovery
> 
> What is the code you are referring to?

from nvme_rdma driver:

static void nvme_rdma_error_recovery_work(struct work_struct *work)
{
         struct nvme_rdma_ctrl *ctrl = container_of(work,
                         struct nvme_rdma_ctrl, err_work);

         nvme_stop_keep_alive(&ctrl->ctrl);

         if (ctrl->ctrl.queue_count > 1) {
                 nvme_stop_queues(&ctrl->ctrl);
                 blk_mq_tagset_busy_iter(&ctrl->tag_set,
                                         nvme_cancel_request, &ctrl->ctrl);
                 nvme_rdma_destroy_io_queues(ctrl, false);
         }

         blk_mq_quiesce_queue(ctrl->ctrl.admin_q);
         blk_mq_tagset_busy_iter(&ctrl->admin_tag_set,
                                 nvme_cancel_request, &ctrl->ctrl);
         nvme_rdma_destroy_admin_queue(ctrl, false);

         /*
          * queues are not a live anymore, so restart the queues to fail 
fast
          * new IO
          */
         blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
         nvme_start_queues(&ctrl->ctrl);

         if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) {
                 /* state change failure should never happen */
                 WARN_ON_ONCE(1);
                 return;
         }

         nvme_rdma_reconnect_or_remove(ctrl);
}


> 
>> process
>> - blk_mq_quiesce_queue for each namespace request queue
> 
> blk_mq_quiesce_queue() only guarantees that no requests can be dispatched to
> low level driver, and new requests still can be allocated, but can't be
> dispatched until the queue becomes unquiesced.
> 
>> - cancel all requests of the tagset using blk_mq_tagset_busy_iter
> 
> Generally blk_mq_tagset_busy_iter() is used to cancel all in-flight
> requests, and it depends on implementation of the busy_tag_iter_fn, and
> timed-out request can't be covered by blk_mq_tagset_busy_iter().

How can we deal with timed-out commands ?


> 
> So blk_mq_tagset_busy_iter() is often used in error recovery path, such
> as nvme_dev_disable(), which is usually used in resetting PCIe NVMe controller.
> 
>> - destroy the QPs/RDMA connections and MR pools
>> - blk_mq_unquiesce_queue for each namespace request queue
>> - reconnect to the target (after creating RDMA resources again)
>>
>> During the QP destruction, I see a warning that not all the memory regions
>> were back to the mr_pool. For every request we get from the block layer
>> (well, almost every request) we get a MR from the MR pool.
>> So what I see is that, depends on the timing, some requests are
>> dispatched/completed after we blk_mq_unquiesce_queue and after we destroy
>> the QP and the MR pool. Probably these request were inserted during
>> quiescing,
> 
> Yes.

maybe we need to update the nvmf_check_init_req to check that the ctrl 
is in NVME_CTRL_LIVE state (otherwise return IOERR), but I need to think 
about it and test it.

> 
>> and I want to flush/drain them before I destroy the QP.
> 
> As mentioned above, you can't do that by blk_mq_quiesce_queue() &
> blk_mq_tagset_busy_iter().
> 
> The PCIe NVMe driver takes two steps for the error recovery: nvme_dev_disable() &
> nvme_reset_work(), and you may consider the similar approach, but the in-flight
> requests won't be drained in this case because they can be requeued.
> 
> Could you explain a bit what your exact problem is?

The problem is that I assign an MR from QP mr_pool for each call to 
nvme_rdma_queue_rq. During the error recovery I destroy the QP and the 
mr_pool *but* some MR's are missing and not returned to the pool.


> 
> Thanks,
> Ming
> 

Thanks,
Max.