linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
From: Sagi Grimberg <sagi@grimberg.me>
To: linux-nvme@lists.infradead.org, Christoph Hellwig <hch@lst.de>,
	Keith Busch <kbusch@kernel.org>
Cc: Chris Leech <cleech@redhat.com>
Subject: [PATCH v2 1/3] nvme: fix a possible use-after-free in controller reset during load
Date: Tue,  1 Feb 2022 14:54:19 +0200	[thread overview]
Message-ID: <20220201125421.177942-2-sagi@grimberg.me> (raw)
In-Reply-To: <20220201125421.177942-1-sagi@grimberg.me>

Unlike .queue_rq, in .submit_async_event drivers may not check the ctrl
readiness for AER submission. This may lead to a use-after-free
condition that was observed with nvme-tcp.

The race condition may happen in the following scenario:
1. driver executes its reset_ctrl_work
2. -> nvme_stop_ctrl - flushes ctrl async_event_work
3. ctrl sends AEN which is received by the host, which in turn
   schedules AEN handling
4. teardown admin queue (which releases the queue socket)
5. AEN processed, submits another AER, calling the driver to submit
6. driver attempts to send the cmd
==> use-after-free

In order to fix that, add ctrl state check to validate the ctrl
is actually able to accept the AER submission.

This addresses the above race in controller resets because the driver
during teardown should:
1. change ctrl state to RESETTING
2. flush async_event_work (as well as other async work elements)

So after 1,2, any other AER command will find the
ctrl state to be RESETTING and bail out without submitting the AER.

Signed-off-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index dd18861f77c0..c11cd3a814fd 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4251,6 +4251,8 @@ static void nvme_async_event_work(struct work_struct *work)
 		container_of(work, struct nvme_ctrl, async_event_work);
 
 	nvme_aen_uevent(ctrl);
+	if (ctrl->state != NVME_CTRL_LIVE)
+		return;
 	ctrl->ops->submit_async_event(ctrl);
 }
 
-- 
2.30.2



  reply	other threads:[~2022-02-01 12:54 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-02-01 12:54 [PATCH v2 0/3] nvme-[tcp|rdma] fix for possible use-after-free Sagi Grimberg
2022-02-01 12:54 ` Sagi Grimberg [this message]
2022-02-03 14:43   ` [PATCH v2 1/3] nvme: fix a possible use-after-free in controller reset during load Max Gurtovoy
2022-02-03 15:03     ` Sagi Grimberg
2022-02-03 15:47       ` Max Gurtovoy
2022-02-04 12:20   ` Hannes Reinecke
2022-02-01 12:54 ` [PATCH v2 2/3] nvme-tcp: fix possible use-after-free in transport error_recovery work Sagi Grimberg
2022-02-04 12:20   ` Hannes Reinecke
2022-02-01 12:54 ` [PATCH v2 3/3] nvme-rdma: " Sagi Grimberg
2022-02-04 12:21   ` Hannes Reinecke

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220201125421.177942-2-sagi@grimberg.me \
    --to=sagi@grimberg.me \
    --cc=cleech@redhat.com \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).