Re: nvme deadlock with ANA

[James Smart <james.smart@broadcom.com>]

On 4/2/2020 8:30 AM, Hannes Reinecke wrote:
> On 4/2/20 5:24 PM, Sagi Grimberg wrote:
>>
>>>> I want to consult with you guys on a deadlock condition I'm able to
>>>> hit with a test that incorporate controller reconnect, ana updates
>>>> and live I/O with timeouts.
>>>>
>>>> This is true for NVMe/TCP, but can also happen in rdma or pci 
>>>> drivers as
>>>> well.
>>>>
>>>> The deadlock combines 4 flows in parallel:
>>>> - ns scanning (triggered from reconnect)
>>>> - request timeout
>>>> - ANA update (triggered from reconnect)
>>>> - FS I/O coming into the mpath device
>>>>
>>>> (1) ns scanning triggers disk revalidation -> update disk info ->
>>>>      freeze queue -> but blocked, why?
>>>
>>> What does -> but blocked mean?
>>
>> It is blocked and cannot complete, because of (2)
>>
>>>> (2) timeout handler reference the g_usage_counter - > but blocks in
>>>>      the timeout handler, why?
>>>
>>> The timeout handler obviously needs to keep the queue alive while
>>> running.  We could think of doing a try_get, though?
>>
>> It is keeping the queue alive, that is not the issue. it is blocked in
>> the driver .timeout() handler (i.e. nvme_tcp_timeout).
>>
>> The reason that it blocked and cannot make forward progress is because
>> the driver timeout handler will call nvme_stop_queues(), which is
>> blocked as this takes namespaces_rwsem...
>>
>> There is a chain of dependency that is deadlocking with circular
>> dependency.
>
> Can't you simply call 'nvme_reset_ctrl()' ?
> Seems to work reasonably well on the fc side, so I wonder what's 
> different for tcp ...

it's no different with FC. We do likely stay in resetting state longer 
as we have to wait for acknowledgement of io abort. But the flows in 
general aren't different.

-- james

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme