From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5731CC77B70 for ; Fri, 7 Apr 2023 19:17:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O8LsWxCTGAleK5IzhkYFJhCU1D7RgCZW4eSmA+et/fE=; b=Aj2o+aYLS/1osdAhXEvUgZigSZ 87uWvN5prVucD8ZQAPbhfrfHI2QKZl6kqXwpJH0ZbNSWpafM7LL0/OQF5jJbf6stSmlrL+nRlnBFk VlJy5VdpHpmO4SVf7G6K4XMrcrchWv1oqSBDnavGK01MzGs5cEZ125In9nED1nTEtpO0CVxfrOt7e WOL5OauK+3l+tIXXtDhdbthDn7SjgUcoltQ7fVh56OaPQIGmUdfRLZCYc3A37PF+VPUz/+pB4tvFw PlSWn2b2GGVU/N9Zod+7bMbQ7sWv1je3IeiYdyLtg8xfnqjaJo5WDQ3WgbIwnnsVQsADgLb4S7mUs eRIO/TjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pkra3-00AxJm-2T; Fri, 07 Apr 2023 19:17:03 +0000 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pkra1-00AxIY-0r for linux-nvme@lists.infradead.org; Fri, 07 Apr 2023 19:17:02 +0000 Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337Gh7Lq013448 for ; Fri, 7 Apr 2023 12:16:58 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=O8LsWxCTGAleK5IzhkYFJhCU1D7RgCZW4eSmA+et/fE=; b=UL1+B5lo+dBdK/DUMTOLMIbXazsjDeetXqtNHTWk1yIUOnmK+KVcm3IOGBHI2aKHWtyC WMX9e2NjR9RxwuC2GqHCfWzDCFMW54Y2KZQ2HSlGnpu8hcn0ngzL3/5+GZ32zUE5MrHk rgqWkYwMLaNxiqB39d63+8p1SwL9CyiDIofJJNrydLNFiVQ+3XjeTNYDGyC6gmfex+FM SNQSE2st+tEKbNHslr8F0F2HQ7yWapprd11IQWSpvX34fs2R4hHT3aStOGTCoNSt2ozP 04dFs+fLn1E5ARis4mRqZKBE+7NLyWzkeKHIOuaYSFlBJjtoO5F5GmWjUcoNv3rSe7XL 6Q== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3ptg0a3d87-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:16:58 -0700 Received: from twshared21760.39.frc1.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:16:56 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 9DC37157B5F88; Fri, 7 Apr 2023 12:16:48 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 4/5] nvme: use blk-mq polling for uring commands Date: Fri, 7 Apr 2023 12:16:35 -0700 Message-ID: <20230407191636.2631046-5-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-FB-Internal: Safe Content-Type: text/plain X-Proofpoint-GUID: Pza8pBETsKCSevXVqO0NFPvy4cz_tq9L X-Proofpoint-ORIG-GUID: Pza8pBETsKCSevXVqO0NFPvy4cz_tq9L X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230407_121701_380515_D099CAF1 X-CRM114-Status: GOOD ( 22.42 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org From: Keith Busch Drivers can now poll requests directly, and the nvme uring command unconditionally saves the request now, so use that. We just need to clear it on completion to prevent any races among multiple threads polling the same queues during a tagset teardown. The first advantage is that unshared and multipath namespaces can use the same polling callback, and multipath is guaranteed to get the same queue as the command was submitted on. Previously multipath polling might check a different path and poll the wrong info. The other advantage is that we don't need a bio payload in order to poll, allowing commands like 'flush' and 'write zeroes' to be submitted on the same high priority queue as read and write commands. This can also allow for a future optimization for the driver since we no longer need to create special hidden block devices to back nvme-generic char dev's with unsupported command sets. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 58 +++++------------------------------ drivers/nvme/host/multipath.c | 2 +- drivers/nvme/host/nvme.h | 2 -- 3 files changed, 9 insertions(+), 53 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index a1e0a14dadedc..3604166752e4b 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -478,9 +478,9 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struc= t request *req, { struct io_uring_cmd *ioucmd =3D req->end_io_data; struct nvme_uring_cmd_pdu *pdu =3D nvme_uring_cmd_pdu(ioucmd); - void *cookie =3D READ_ONCE(ioucmd->cookie); int status =3D nvme_req(req)->status; =20 + WRITE_ONCE(pdu->req, NULL); if (nvme_req(req)->flags & NVME_REQ_CANCELLED) status =3D -EINTR; =20 @@ -495,7 +495,7 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struc= t request *req, * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (cookie !=3D NULL && blk_rq_is_poll(req)) + if (blk_rq_is_poll(req)) nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); else io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_cb); @@ -554,7 +554,6 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, = struct nvme_ns *ns, if (issue_flags & IO_URING_F_IOPOLL) rq_flags |=3D REQ_POLLED; =20 -retry: req =3D nvme_alloc_user_request(q, &c, rq_flags, blk_flags); if (IS_ERR(req)) return PTR_ERR(req); @@ -568,19 +567,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl,= struct nvme_ns *ns, return ret; } =20 - if (issue_flags & IO_URING_F_IOPOLL && rq_flags & REQ_POLLED) { - if (unlikely(!req->bio)) { - /* we can't poll this, so alloc regular req instead */ - blk_mq_free_request(req); - rq_flags &=3D ~REQ_POLLED; - goto retry; - } else { - WRITE_ONCE(ioucmd->cookie, req->bio); - req->bio->bi_opf |=3D REQ_POLLED; - } - } - - pdu->req =3D req; + WRITE_ONCE(pdu->req, req); pdu->meta_len =3D d.metadata_len; req->end_io_data =3D ioucmd; req->end_io =3D nvme_uring_cmd_end_io; @@ -735,18 +722,14 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cm= d *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags) { - struct bio *bio; + struct nvme_uring_cmd_pdu *pdu =3D nvme_uring_cmd_pdu(ioucmd); + struct request *req; int ret =3D 0; - struct nvme_ns *ns; - struct request_queue *q; =20 rcu_read_lock(); - bio =3D READ_ONCE(ioucmd->cookie); - ns =3D container_of(file_inode(ioucmd->file)->i_cdev, - struct nvme_ns, cdev); - q =3D ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio && bio->bi_bdev) - ret =3D bio_poll(bio, iob, poll_flags); + req =3D READ_ONCE(pdu->req); + if (req && blk_rq_is_poll(req)) + ret =3D blk_rq_poll(req, iob, poll_flags); rcu_read_unlock(); return ret; } @@ -838,31 +821,6 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *= ioucmd, srcu_read_unlock(&head->srcu, srcu_idx); return ret; } - -int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, - struct io_comp_batch *iob, - unsigned int poll_flags) -{ - struct cdev *cdev =3D file_inode(ioucmd->file)->i_cdev; - struct nvme_ns_head *head =3D container_of(cdev, struct nvme_ns_head, c= dev); - int srcu_idx =3D srcu_read_lock(&head->srcu); - struct nvme_ns *ns =3D nvme_find_path(head); - struct bio *bio; - int ret =3D 0; - struct request_queue *q; - - if (ns) { - rcu_read_lock(); - bio =3D READ_ONCE(ioucmd->cookie); - q =3D ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio - && bio->bi_bdev) - ret =3D bio_poll(bio, iob, poll_flags); - rcu_read_unlock(); - } - srcu_read_unlock(&head->srcu, srcu_idx); - return ret; -} #endif /* CONFIG_NVME_MULTIPATH */ =20 int nvme_dev_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_f= lags) diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.= c index 9171452e2f6d4..f17be1c72f4de 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -470,7 +470,7 @@ static const struct file_operations nvme_ns_head_chr_= fops =3D { .unlocked_ioctl =3D nvme_ns_head_chr_ioctl, .compat_ioctl =3D compat_ptr_ioctl, .uring_cmd =3D nvme_ns_head_chr_uring_cmd, - .uring_cmd_iopoll =3D nvme_ns_head_chr_uring_cmd_iopoll, + .uring_cmd_iopoll =3D nvme_ns_chr_uring_cmd_iopoll, }; =20 static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1e..ca4ea89333660 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -847,8 +847,6 @@ long nvme_dev_ioctl(struct file *file, unsigned int c= md, unsigned long arg); int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags); -int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, - struct io_comp_batch *iob, unsigned int poll_flags); int nvme_ns_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags); int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd, --=20 2.34.1