From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB0C1CCF9E3 for ; Thu, 23 Oct 2025 20:18:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pWqWl9I6OaTJzpoQyIopXGaUVLHhxAn4/wC5EGBqgEo=; b=N/ezve2ROUls5bm7UiJW7q0Gj/ wLdPc3T7famzwdiP23F5rIwp0EOzr8cH8JzznTKRV+k/NK1XQ9hr8oqgyugagx0gecRy5VMpCVwxA tW3LjW/Xr55MRZPeYyipT0QHC11NkZEiYWAfpg8y8OeZMcl51weT8A+eUrNpdjUAlKhG2IIAkzcJr Yj0Hl6+TyobRf76Nrac/xMX6h9rtyXa7OSkYHxrwy6XkIL750s7IjmggqIeoLjpEmaV6He8OS1l0o AaoEL/L9Kjx8HDrd9nAhLSI/7+I7+oJoRLs1AuGqC4QD/+VYgICgUonjgSt2AZJSdvyvTjA1texs5 iCaQirag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1vC1lq-00000007W9m-2QuE; Thu, 23 Oct 2025 20:18:50 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1vC1lo-00000007W81-1OKm for linux-nvme@bombadil.infradead.org; Thu, 23 Oct 2025 20:18:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pWqWl9I6OaTJzpoQyIopXGaUVLHhxAn4/wC5EGBqgEo=; b=JjixAzuUoIEKhlG2KTHmy+VxjD vyLzghMfBadJXSWtL58W3rUGGPeUSPdlurQSBSHQGryiPGgr9mtBxfpqVLg7CDj+NIbgCa+X3qKDv fDXk+393lml4iPwegADi82Ap8alJnSbJuXHQ2vVRdAtHF4BTp0bsc4jZYjlr37IDQLf5t8P7TNcP4 1SwicArm8WoeHjsBVVjLZked19eOCMGYC0nazWFGTl9ecHkj5nubjUOfvM/JxwccEjAznZYJNXRdS jLO5zmp941SW5ItZhTcejCMJg4AwQtdUNr0PrY8djV15WsNUVCZlbUWKaLk8KckDPCo4f1u0i/R9i RkZEYBew==; Received: from mail-wr1-x463.google.com ([2a00:1450:4864:20::463]) by desiato.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1vC0u0-00000001i8J-38ti for linux-nvme@lists.infradead.org; Thu, 23 Oct 2025 19:23:19 +0000 Received: by mail-wr1-x463.google.com with SMTP id ffacd0b85a97d-4270a072a0bso225530f8f.0 for ; Thu, 23 Oct 2025 13:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1761250716; x=1761855516; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pWqWl9I6OaTJzpoQyIopXGaUVLHhxAn4/wC5EGBqgEo=; b=QGUC1Sg3J/zdBX/2Vu2OnKQVpHZ0j620PW8fVie0P0gBDhxTv801uGrpuuQ9DG89VC cWHyXHVr4NmCEqJyRyY62tfOdNi3vMhim9VL7zQNbHLMLFb5Ge3RXNqRTi4+aJmhYzXr ahqbyq7cb2KZNU6zsy0YUlhtc63ea58kmcdCMcloxPzsdD5cjK0MgbJe3BFOtugsLYb3 lDwejnNrZqM6mKNjkEUTp9PHkFey0aSFX3qvS1l8yh6KRRXfIOMDwHMXedKr/hKaM3t5 qqCZLf2EWIGnJ803zNnEcDC9MA4EtG8kbW9jK6YwAfU6txENHtl1I0kKziSkO7bBj2n9 Zt6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761250716; x=1761855516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pWqWl9I6OaTJzpoQyIopXGaUVLHhxAn4/wC5EGBqgEo=; b=b4Hk2UozEBnV9CrKqkfsGt7EledGK+SrxSgWEHgt4wl294XrrN8KJfGVIhcuRx0or8 F6UfBieg8jfBR2u7C84reP26ug1jkBgGDsT1/Fvc0C2xRzpd8n9hw2mbFyg0CAqoEPx4 SHQd7/iQZQq9H98BygZ7VJhFe6tp6NuDEjul5sok/h7yE0LpcwbOxHpWnB8OiB0Yhw7m Qx0Bh1nHlKZTlEWxkqKPEBhN9k3KMiW0f0tKbK9jAXgfadysf81i66dbimDGXVDtkRgF mkNBjwq9Xdz+pNTrEHraowu4UbzpuSXQZMcybIOhI3D47VeFIwyFQItJI11DjK6k9Slv 1pSg== X-Forwarded-Encrypted: i=1; AJvYcCUzr81QJ1nbbE3E4twbLn1c5CLerKGpKSGbL47I6svFMIoZ/DIbk0xNH9tNl5By34JSjAGBUqCERW4O@lists.infradead.org X-Gm-Message-State: AOJu0Yx2yb4PKEjKFy+B7JMgCUpmPqo3WZU4LQESiPSyF15nqcDbLMe3 hTuT2BycyMdLlkmlsZV91WDDKXdSRydgLag0bIoqa45KKbB8jpFLYegsgPfFQNa5fmfo88gM926 ujj700Lb/457DJG2kAcLHCqEo3KVo3xtnmPcI X-Gm-Gg: ASbGncuXavHDydMZzrEJSaxNSz4itUIN7cwTdgEL32k5iTvqgQUisk2+KL/lBi9jDtM 09Qqnk47USBgRXTFnhsa1z5WucG+4sJMD5VVWLfjHke22f9Pr6/VVyXCGYIFx29jCcAri/xEeVA nualv4Whay8NlFnXEvTEgpjUlthYiMBP9e4Z+mH9n+PZ28Tbu9GGZ2lrzFf14nmPdFxIGqBTnah cZ34TZ+6ZtdFoqTBFU9peTL7ml0Upco6w8NjoWTJfHNlrqaqho+G8edCKnTUvrRgrmYt0fR+wAt wtcqg7G10edgwcXUxPFZg54UlM1nwASddqKUJ9nGUGQIF2yVvjXSDcjwNqJXPGY/lYZliHqkpja w8k99hPNUo6OdDnQav8B2iRwJAilHS1c= X-Google-Smtp-Source: AGHT+IG+F7LvVb9q/feNRCz+uAzHxkM23HOpSSc2ZgklRM8Pol3wHaimWcAj/kEpeUsdrKVgqlM3uocuvYeU X-Received: by 2002:a05:600c:1e8b:b0:46e:36f9:c57e with SMTP id 5b1f17b1804b1-47494305991mr53576475e9.5.1761250716185; Thu, 23 Oct 2025 13:18:36 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.129]) by smtp-relay.gmail.com with ESMTPS id 5b1f17b1804b1-475c427ca8bsm10719765e9.3.2025.10.23.13.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Oct 2025 13:18:36 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (unknown [IPv6:2620:125:9007:640:ffff::1199]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id ACE5E340884; Thu, 23 Oct 2025 14:18:34 -0600 (MDT) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id ABEF0E41B1D; Thu, 23 Oct 2025 14:18:34 -0600 (MDT) From: Caleb Sander Mateos To: Jens Axboe , Miklos Szeredi , Ming Lei , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chris Mason , David Sterba Cc: io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-btrfs@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH v2 3/3] io_uring/uring_cmd: avoid double indirect call in task work dispatch Date: Thu, 23 Oct 2025 14:18:30 -0600 Message-ID: <20251023201830.3109805-4-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251023201830.3109805-1-csander@purestorage.com> References: <20251023201830.3109805-1-csander@purestorage.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20251023_202317_577934_099D3EE2 X-CRM114-Status: GOOD ( 21.41 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org io_uring task work dispatch makes an indirect call to struct io_kiocb's io_task_work.func field to allow running arbitrary task work functions. In the uring_cmd case, this calls io_uring_cmd_work(), which immediately makes another indirect call to struct io_uring_cmd's task_work_cb field. Define the uring_cmd task work callbacks as functions whose signatures match io_req_tw_func_t. Define a IO_URING_CMD_TASK_WORK_ISSUE_FLAGS constant in io_uring/cmd.h to avoid manufacturing issue_flags in the uring_cmd task work callbacks. Now uring_cmd task work dispatch makes a single indirect call to the uring_cmd implementation's callback. This also allows removing the task_work_cb field from struct io_uring_cmd, freeing up some additional storage space. Signed-off-by: Caleb Sander Mateos --- block/ioctl.c | 4 +++- drivers/block/ublk_drv.c | 15 +++++++++------ drivers/nvme/host/ioctl.c | 5 +++-- fs/btrfs/ioctl.c | 4 +++- fs/fuse/dev_uring.c | 5 +++-- include/linux/io_uring/cmd.h | 16 +++++++--------- io_uring/uring_cmd.c | 13 ++----------- 7 files changed, 30 insertions(+), 32 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index d7489a56b33c..5c10d48fab27 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -767,13 +767,15 @@ long compat_blkdev_ioctl(struct file *file, unsigned cmd, unsigned long arg) struct blk_iou_cmd { int res; bool nowait; }; -static void blk_cmd_complete(struct io_uring_cmd *cmd, unsigned int issue_flags) +static void blk_cmd_complete(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); struct blk_iou_cmd *bic = io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; if (bic->res == -EAGAIN && bic->nowait) io_uring_cmd_issue_blocking(cmd); else io_uring_cmd_done(cmd, bic->res, issue_flags); diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 0c74a41a6753..00439d1879b0 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1346,13 +1346,14 @@ static void ublk_dispatch_req(struct ublk_queue *ubq, if (ublk_prep_auto_buf_reg(ubq, req, io, issue_flags)) ublk_complete_io_cmd(io, req, UBLK_IO_RES_OK, issue_flags); } -static void ublk_cmd_tw_cb(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void ublk_cmd_tw_cb(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd); struct ublk_queue *ubq = pdu->ubq; ublk_dispatch_req(ubq, pdu->req, issue_flags); } @@ -1364,13 +1365,14 @@ static void ublk_queue_cmd(struct ublk_queue *ubq, struct request *rq) pdu->req = rq; io_uring_cmd_complete_in_task(cmd, ublk_cmd_tw_cb); } -static void ublk_cmd_list_tw_cb(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void ublk_cmd_list_tw_cb(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd); struct request *rq = pdu->req_list; struct request *next; do { @@ -2521,13 +2523,14 @@ static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub, fail_put: ublk_put_req_ref(io, req); return NULL; } -static void ublk_ch_uring_cmd_cb(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void ublk_ch_uring_cmd_cb(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; int ret = ublk_ch_uring_cmd_local(cmd, issue_flags); if (ret != -EIOCBQUEUED) io_uring_cmd_done(cmd, ret, issue_flags); } diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index c212fa952c0f..df39cee94de1 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -396,13 +396,14 @@ static inline struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu( struct io_uring_cmd *ioucmd) { return io_uring_cmd_to_pdu(ioucmd, struct nvme_uring_cmd_pdu); } -static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, - unsigned issue_flags) +static void nvme_uring_task_cb(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); if (pdu->bio) blk_rq_unmap_user(pdu->bio); io_uring_cmd_done32(ioucmd, pdu->status, pdu->result, issue_flags); diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 185bef0df1c2..3b62eb8a50dc 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -4647,13 +4647,15 @@ struct btrfs_uring_priv { struct io_btrfs_cmd { struct btrfs_uring_encoded_data *data; struct btrfs_uring_priv *priv; }; -static void btrfs_uring_read_finished(struct io_uring_cmd *cmd, unsigned int issue_flags) +static void btrfs_uring_read_finished(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); struct io_btrfs_cmd *bc = io_uring_cmd_to_pdu(cmd, struct io_btrfs_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; struct btrfs_uring_priv *priv = bc->priv; struct btrfs_inode *inode = BTRFS_I(file_inode(priv->iocb.ki_filp)); struct extent_io_tree *io_tree = &inode->io_tree; pgoff_t index; u64 cur; diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 71b0c9662716..051136e94a33 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -1207,13 +1207,14 @@ static void fuse_uring_send(struct fuse_ring_ent *ent, struct io_uring_cmd *cmd, /* * This prepares and sends the ring request in fuse-uring task context. * User buffers are not mapped yet - the application does not have permission * to write to it - this has to be executed in ring task context. */ -static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, - unsigned int issue_flags) +static void fuse_uring_send_in_task(struct io_kiocb *req, io_tw_token_t tw) { + struct io_uring_cmd *cmd = io_kiocb_to_cmd(req, struct io_uring_cmd); + unsigned int issue_flags = IO_URING_CMD_TASK_WORK_ISSUE_FLAGS; struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd); struct fuse_ring_queue *queue = ent->queue; int err; if (!io_uring_cmd_should_terminate_tw(cmd)) { diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index b84b97c21b43..3efad93404f9 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -9,18 +9,13 @@ /* only top 8 bits of sqe->uring_cmd_flags for kernel internal use */ #define IORING_URING_CMD_CANCELABLE (1U << 30) /* io_uring_cmd is being issued again */ #define IORING_URING_CMD_REISSUE (1U << 31) -typedef void (*io_uring_cmd_tw_t)(struct io_uring_cmd *cmd, - unsigned issue_flags); - struct io_uring_cmd { struct file *file; const struct io_uring_sqe *sqe; - /* callback to defer completions to task context */ - io_uring_cmd_tw_t task_work_cb; u32 cmd_op; u32 flags; u8 pdu[32]; /* available inline for free use */ }; @@ -58,11 +53,11 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, */ void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32); void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, + io_req_tw_func_t task_work_cb, unsigned flags); /* * Note: the caller should never hard code @issue_flags and only use the * mask provided by the core io_uring code. @@ -107,11 +102,11 @@ static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd, static inline void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret, u64 ret2, unsigned issue_flags, bool is_cqe32) { } static inline void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, unsigned flags) + io_req_tw_func_t task_work_cb, unsigned flags) { } static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { @@ -130,19 +125,22 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct io_uring_cmd *ioucmd, { return true; } #endif +/* task_work executor checks the deferred list completion */ +#define IO_URING_CMD_TASK_WORK_ISSUE_FLAGS IO_URING_F_COMPLETE_DEFER + /* users must follow the IOU_F_TWQ_LAZY_WAKE semantics */ static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb) + io_req_tw_func_t task_work_cb) { __io_uring_cmd_do_in_task(ioucmd, task_work_cb, IOU_F_TWQ_LAZY_WAKE); } static inline void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb) + io_req_tw_func_t task_work_cb) { __io_uring_cmd_do_in_task(ioucmd, task_work_cb, 0); } static inline bool io_uring_cmd_should_terminate_tw(struct io_uring_cmd *cmd) diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 35bdac35cf4d..5a80d35658dc 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -111,29 +111,20 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, io_ring_submit_unlock(ctx, issue_flags); } } EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); -static void io_uring_cmd_work(struct io_kiocb *req, io_tw_token_t tw) -{ - struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd); - - /* task_work executor checks the deffered list completion */ - ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); -} - void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, + io_req_tw_func_t task_work_cb, unsigned flags) { struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); if (WARN_ON_ONCE(req->flags & REQ_F_APOLL_MULTISHOT)) return; - ioucmd->task_work_cb = task_work_cb; - req->io_task_work.func = io_uring_cmd_work; + req->io_task_work.func = task_work_cb; __io_req_task_work_add(req, flags); } EXPORT_SYMBOL_GPL(__io_uring_cmd_do_in_task); static inline void io_req_set_cqe32_extra(struct io_kiocb *req, -- 2.45.2