Re: nvme deadlock with ANA

[Hannes Reinecke <hare@suse.de>]

On 4/2/20 5:24 PM, Sagi Grimberg wrote:
> 
>>> I want to consult with you guys on a deadlock condition I'm able to
>>> hit with a test that incorporate controller reconnect, ana updates
>>> and live I/O with timeouts.
>>>
>>> This is true for NVMe/TCP, but can also happen in rdma or pci drivers as
>>> well.
>>>
>>> The deadlock combines 4 flows in parallel:
>>> - ns scanning (triggered from reconnect)
>>> - request timeout
>>> - ANA update (triggered from reconnect)
>>> - FS I/O coming into the mpath device
>>>
>>> (1) ns scanning triggers disk revalidation -> update disk info ->
>>>      freeze queue -> but blocked, why?
>>
>> What does -> but blocked mean?
> 
> It is blocked and cannot complete, because of (2)
> 
>>> (2) timeout handler reference the g_usage_counter - > but blocks in
>>>      the timeout handler, why?
>>
>> The timeout handler obviously needs to keep the queue alive while
>> running.  We could think of doing a try_get, though?
> 
> It is keeping the queue alive, that is not the issue. it is blocked in
> the driver .timeout() handler (i.e. nvme_tcp_timeout).
> 
> The reason that it blocked and cannot make forward progress is because
> the driver timeout handler will call nvme_stop_queues(), which is
> blocked as this takes namespaces_rwsem...
> 
> There is a chain of dependency that is deadlocking with circular
> dependency.

Can't you simply call 'nvme_reset_ctrl()' ?
Seems to work reasonably well on the fc side, so I wonder what's 
different for tcp ...

Cheers,

Hannes
-- 
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare@suse.de                               +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme