Re: [PATCH 1/2] nvme-tcp: start error recovery when read_sock fails

[sashiko-bot@kernel.org]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] nvme-tcp: race on queue->rd_enabled after release_sock() in nvme_tcp_try_recv()
--

commit 3915b73f53c8a994bf45e1efec593bc2bb4f9f8b
Author: Martin Belanger <martin.belanger@dell.com>

nvme-tcp: start error recovery when read_sock fails

This commit updates nvme_tcp_try_recv() to log an error, disable the queue,
and trigger controller error recovery when sock->ops->read_sock() returns a
negative error code other than -EAGAIN and queue->rd_enabled is still set.

> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 7190eb10c2481..a03476f0f69d5 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1431,7 +1431,21 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
>  	queue->nr_cqe = 0;
>  	consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
>  	release_sock(sk);
> -	return consumed == -EAGAIN ? 0 : consumed;
> +	if (consumed == -EAGAIN)
> +		return 0;
> +
> +	/* nvme_tcp_recv_skb() does both of these itself, so a queue that is
> +	 * still enabled failed inside read_sock() instead
> +	 */
> +	if (unlikely(consumed < 0 && queue->rd_enabled)) {
> +		dev_err(queue->ctrl->ctrl.device,
> +			"queue %d: receive failed: %d\n",
> +			nvme_tcp_queue_id(queue), consumed);
> +		queue->rd_enabled = false;
> +		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
> +	}

[Severity: Medium]
Should the check and clearing of queue->rd_enabled happen before calling
release_sock(sk) in drivers/nvme/host/tcp.c:nvme_tcp_try_recv()?

In nvme_tcp_recv_skb(), queue->rd_enabled is checked, cleared, and followed
by nvme_tcp_error_recovery() while lock_sock(sk) is held:

drivers/nvme/host/tcp.c:nvme_tcp_recv_skb() {
	if (unlikely(!queue->rd_enabled))
		return -EFAULT;
...
		if (result) {
			dev_err(queue->ctrl->ctrl.device,
				"receive failed:  %d\n", result);
			queue->rd_enabled = false;
			nvme_tcp_error_recovery(&queue->ctrl->ctrl);
			return result;
		}
}

When sock->ops->read_sock() fails directly without hitting the error path in
nvme_tcp_recv_skb() (for example, tls_sw_read_sock() returning -EINVAL on a
non-data TLS record, -EBADMSG, -EKEYEXPIRED, -ENOMEM, or sock_error()),
calling release_sock(sk) first drains sk->sk_backlog via __release_sock(sk)
while queue->rd_enabled is still true.

Any TCP segments that arrived in softirq while the socket was owned will
trigger sk->sk_data_ready:

release_sock()
  -> __release_sock()
    -> nvme_tcp_data_ready()

drivers/nvme/host/tcp.c:nvme_tcp_data_ready() {
	read_lock_bh(&sk->sk_callback_lock);
	queue = sk->sk_user_data;
	if (likely(queue && queue->rd_enabled) &&
	    !test_bit(NVME_TCP_Q_POLLING, &queue->flags))
		queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
	read_unlock_bh(&sk->sk_callback_lock);
}

Because queue->rd_enabled has not been cleared yet during release_sock(sk)
and the subsequent dev_err() call, nvme_tcp_data_ready() sees
queue->rd_enabled as true and re-queues queue->io_work on the failed queue.

In addition, nvme_tcp_try_recv() can be called concurrently from
nvme_tcp_poll() and nvme_tcp_io_work(), relying on lock_sock(sk) for
serialization:

drivers/nvme/host/tcp.c:nvme_tcp_poll() {
	set_bit(NVME_TCP_Q_POLLING, &queue->flags);
	if (sk_can_busy_loop(sk) && skb_queue_empty_lockless(&sk->sk_receive_queue))
		sk_busy_loop(sk, true);
	ret = nvme_tcp_try_recv(queue);
	clear_bit(NVME_TCP_Q_POLLING, &queue->flags);
}

Could releasing lock_sock(sk) before checking and clearing queue->rd_enabled
allow a concurrent nvme_tcp_try_recv() caller to acquire lock_sock(sk) while
queue->rd_enabled is still true, either logging duplicate errors or entering
nvme_tcp_recv_skb() after a transient read_sock() error?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261026-nvme-tcp-ctrlrec-fixes-v1-0-ebf6ff7eff34@gmail.com?part=1