[PATCH] nvmet: fix a possible leak when destroy a ctrl during qp establishment

[PATCH] nvmet: fix a possible leak when destroy a ctrl during qp establishment

[Sagi Grimberg]

In nvmet_sq_destroy we capture sq->ctrl early and if it is non-NULL we
know that a ctrl was allocated (in the admin connect request handler)
and we need to release pending AERs, clear ctrl->sqs and sq->ctrl
(for nvme-loop primarily), and drop the final reference on the ctrl.

However, a small window is possible where nvmet_sq_destroy starts (as
a result of the client giving up and disconnecting) concurrently with
the nvme admin connect cmd (which may be in an early stage). But *before*
kill_and_confirm of sq->ref (i.e. the admin connect managed to get an sq
live reference). In this case, sq->ctrl was allocated however after it was
captured in a local variable in nvmet_sq_destroy.
This prevented the final reference drop on the ctrl.

Solve this by re-capturing the sq->ctrl after all inflight request has
completed, where for sure sq->ctrl reference is final, and move forward
based on that.

This issue was observed in an environment with many hosts connecting
multiple ctrls simoutanuosly, creating a delay in allocating a ctrl
leading up to this race window.

Reported-by: Alex Turin <alex@vastdata.com>
Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/core.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 2fde22323622..06f0c587f343 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -818,6 +818,15 @@ void nvmet_sq_destroy(struct nvmet_sq *sq)
 	percpu_ref_exit(&sq->ref);
 	nvmet_auth_sq_free(sq);
 
+	/*
+	 * we must reference the ctrl again after waiting for inflight IO
+	 * to complete. Because admin connect may have sneaked in after we
+	 * store sq->ctrl locally, but before we killed the percpu_ref. the
+	 * admin connect allocates and assigns sq->ctrl, which now needs a
+	 * final ref put, as this ctrl is going away.
+	 */
+	ctrl = sq->ctrl;
+
 	if (ctrl) {
 		/*
 		 * The teardown flow may take some time, and the host may not
-- 
2.40.1

Re: [PATCH] nvmet: fix a possible leak when destroy a ctrl during qp establishment

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH] nvmet: fix a possible leak when destroy a ctrl during qp establishment

[Keith Busch]

On Mon, May 27, 2024 at 10:38:52PM +0300, Sagi Grimberg wrote:
> In nvmet_sq_destroy we capture sq->ctrl early and if it is non-NULL we
> know that a ctrl was allocated (in the admin connect request handler)
> and we need to release pending AERs, clear ctrl->sqs and sq->ctrl
> (for nvme-loop primarily), and drop the final reference on the ctrl.
> 
> However, a small window is possible where nvmet_sq_destroy starts (as
> a result of the client giving up and disconnecting) concurrently with
> the nvme admin connect cmd (which may be in an early stage). But *before*
> kill_and_confirm of sq->ref (i.e. the admin connect managed to get an sq
> live reference). In this case, sq->ctrl was allocated however after it was
> captured in a local variable in nvmet_sq_destroy.
> This prevented the final reference drop on the ctrl.
> 
> Solve this by re-capturing the sq->ctrl after all inflight request has
> completed, where for sure sq->ctrl reference is final, and move forward
> based on that.
> 
> This issue was observed in an environment with many hosts connecting
> multiple ctrls simoutanuosly, creating a delay in allocating a ctrl
> leading up to this race window.

Thanks, applied to nvme-6.10.

Re: [PATCH] nvmet: fix a possible leak when destroy a ctrl during qp establishment

[Chaitanya Kulkarni]

On 5/27/24 12:38, Sagi Grimberg wrote:
> In nvmet_sq_destroy we capture sq->ctrl early and if it is non-NULL we
> know that a ctrl was allocated (in the admin connect request handler)
> and we need to release pending AERs, clear ctrl->sqs and sq->ctrl
> (for nvme-loop primarily), and drop the final reference on the ctrl.
>
> However, a small window is possible where nvmet_sq_destroy starts (as
> a result of the client giving up and disconnecting) concurrently with
> the nvme admin connect cmd (which may be in an early stage). But *before*
> kill_and_confirm of sq->ref (i.e. the admin connect managed to get an sq
> live reference). In this case, sq->ctrl was allocated however after it was
> captured in a local variable in nvmet_sq_destroy.
> This prevented the final reference drop on the ctrl.
>
> Solve this by re-capturing the sq->ctrl after all inflight request has
> completed, where for sure sq->ctrl reference is final, and move forward
> based on that.
>
> This issue was observed in an environment with many hosts connecting
> multiple ctrls simoutanuosly, creating a delay in allocating a ctrl
> leading up to this race window.
>
> Reported-by: Alex Turin <alex@vastdata.com>
> Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
> ---
>

thanks a lot for fixing this.

FWIW Looks good.

Reviewed-by: Chaitanya Kulkarni <kch@nvidia.com>

-ck