From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42F251C5F13 for ; Tue, 28 Jan 2025 13:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738071588; cv=none; b=tOukyGPN5myl9NQRRfjl3iHh2gvX745N1vHE2QUetdubih27fPxZEV+riRBnTzmGfWC8hXdM+OiHGu+/WAN7l0wmsyhgsSdxltKEyxN+njMEsd3GPWOClccteQvBSK/3iU+2yNXTdTSQE/a7KBMBf+YShrQNOyjY5zzK6GccHfE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1738071588; c=relaxed/simple; bh=rxwS0rDFX14c0VMUZBaTjErpG3AmBNovvucJwZOZofA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rvRIV2dQvRJFor8tNjnIsblah3c4DVzjGNXa10eocusqjaKX6ZRj3QeoQ0xC2eETlZ8UZtvIPI71pcHBmOjeKJWd37Ecbi0Dubgdvt6P0H/2eIWo4QcObUXQ4x2ZcoKWNoPPSnZxO3Rr1PqVGZlaLQK3tuZFSvEdxtf3p4WClQI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=SIP3kCqd; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="SIP3kCqd" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-43634b570c1so39855905e9.0 for ; Tue, 28 Jan 2025 05:39:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1738071583; x=1738676383; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=96TZy8b5BW2zVkT896YXvpQRJfRndGqutQhUtIiN6/o=; b=SIP3kCqdEaMZT+247qyJh425UPNTX6nRNn7qlEDLoq7ykLkxWKiw1/qrUWW/vLXY9M 7KgfdjNoKDvvrPgQ6D6quMyxB5BXqTdfAvw+e1JVuOjV6aLXoB70lquYjZQTC2GB9pp9 YbS186soaYrRoNiqV6WfJOWrVn45Ebj8IJxDVoG/WrHUVuMx/MmfGLNYjsOdUj2VQQJi vLm7K4slVbb8KWRoDvVHzu03lMPyzy9PTNG2sLYpxoVKP4+Mqr/lOQWmkoHcDa0TPubC k7XAiLWlpeIfR+IYdSAVTSd0Df4oHRsX9gnx4/bOdwQjWRhulgXjVedbCFh/x21/NkSb 1A4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738071583; x=1738676383; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=96TZy8b5BW2zVkT896YXvpQRJfRndGqutQhUtIiN6/o=; b=ruSEg+M4IVLxX0+Gp1ebZrQQb3FI25TUKa/psa2ROPkS09us7MVX4WLgNF89xxVFPf LPk3p6KFMT2NSaDSe5ygudCrMO9nLGwkugOOanL80bxnZiSGmpVl10f9LtWyrtfNqCj+ Slbg/k9nHDYJA8PL1Fi+hnahqQOdlvzQ3X1ijhFLBwZnNIQmCcTkefBiDwGo9qBNt29X CrwKRuWtmzvEa3Vu7CJPRr0EbXHJZSZaTaq8sW3gRSfWE2IAELOKl+oAim8Y5MNe8URe wuvMyDFRLoFj5KNMDLBae6j62or91ANK0IpqbA3BlsvIUwjb0r4JBG5ILzOew8vuVUwX B5ig== X-Forwarded-Encrypted: i=1; AJvYcCV6WKwGpmWkK2BwafWIBHlNpqbr/zVa/YyXhlkPQdiTmciop6TML+w+0wsuE4zkTHA/aQVgTQ8CVJUL1U4=@vger.kernel.org X-Gm-Message-State: AOJu0Yz+yOPvpM+cwFPosURk5S7qjCMeqm3TCb/9X9uQDNvJFrh0RhbU yVhTmzaA3VKE2CsY1lICuZSWwF0gsPV4zXh5ZNqR+hh9IXrZxgc/jo3/6K6SrZ0= X-Gm-Gg: ASbGncuTRubwFUN4XtTdX5uFgl9k1/ZNiWcWAT9cSACV2TyHLDg468dU7I9iz43XPbj U75+KNxdLnCHYF31d3hCm+JtUkEbVcSbeK128hdsWpHvcAEZbGzZUPZ28PgX9C1BJGNcLwsZqp1 t/JWWjuIjdVAkTJQFo15AV2oWP0eqUqNyQIYcmFXKiml72Ei4wmBPvK4ydQ6BvD6UHG42mODj6r 3iP2UqakAxsWzbwzRKoinWJFFqJmVm6Oyvj21I28pVPuILFp1KqzVvdvzmGp89jK+pQkd25wviP YajUp+xLj0gt2Q370aXInK6TGTY2RFYxrjXwpIYiImvrLB6FP6kucQD4koY7usUcWn3CMXlU30U QFp+HMWYyJM6b4kU= X-Google-Smtp-Source: AGHT+IFHR2TSaCX4qA/N4aKyPqiprXuE+wstO94ZvTs6BcxDNv4grOj/aPxscmSz1+8YtYt8k2Ju/g== X-Received: by 2002:a05:6000:18ab:b0:38a:88a0:2234 with SMTP id ffacd0b85a97d-38bf5655328mr34750312f8f.4.1738071583546; Tue, 28 Jan 2025 05:39:43 -0800 (PST) Received: from raven.intern.cm-ag (p200300dc6f2b6900023064fffe740809.dip0.t-ipconnect.de. [2003:dc:6f2b:6900:230:64ff:fe74:809]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a1bb02dsm14160780f8f.70.2025.01.28.05.39.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Jan 2025 05:39:43 -0800 (PST) From: Max Kellermann To: axboe@kernel.dk, asml.silence@gmail.com, io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Max Kellermann Subject: [PATCH 8/8] io_uring: skip redundant poll wakeups Date: Tue, 28 Jan 2025 14:39:27 +0100 Message-ID: <20250128133927.3989681-9-max.kellermann@ionos.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250128133927.3989681-1-max.kellermann@ionos.com> References: <20250128133927.3989681-1-max.kellermann@ionos.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Using io_uring with epoll is very expensive because every completion leads to a __wake_up() call, most of which are unnecessary because the polling process has already been woken up but has not had a chance to process the completions. During this time, wq_has_sleeper() still returns true, therefore this check is not enough. Perf diff for a HTTP server pushing a static file with splice() into the TCP socket: 37.91% -2.00% [kernel.kallsyms] [k] queued_spin_lock_slowpath 1.69% -1.67% [kernel.kallsyms] [k] ep_poll_callback 0.95% +1.64% [kernel.kallsyms] [k] io_wq_free_work 0.88% -0.35% [kernel.kallsyms] [k] _raw_spin_lock_irqsave 1.66% +0.28% [kernel.kallsyms] [k] io_worker_handle_work 1.14% +0.18% [kernel.kallsyms] [k] _raw_spin_lock 0.24% -0.17% [kernel.kallsyms] [k] __wake_up Signed-off-by: Max Kellermann --- include/linux/io_uring_types.h | 10 ++++++++++ io_uring/io_uring.c | 4 ++++ io_uring/io_uring.h | 2 +- 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 623d8e798a11..01514cb76095 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -384,6 +384,16 @@ struct io_ring_ctx { struct wait_queue_head poll_wq; struct io_restriction restrictions; + /** + * Non-zero if a process is waiting for #poll_wq and reset to + * zero when #poll_wq is woken up. This is supposed to + * eliminate redundant wakeup calls. Only checking + * wq_has_sleeper() is not enough because it will return true + * until the sleeper has actually woken up and has been + * scheduled. + */ + atomic_t poll_wq_waiting; + u32 pers_next; struct xarray personalities; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 137c2066c5a3..b65efd07e9f0 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2760,6 +2760,7 @@ static __cold void io_activate_pollwq_cb(struct callback_head *cb) * Wake ups for some events between start of polling and activation * might've been lost due to loose synchronisation. */ + atomic_set_release(&ctx->poll_wq_waiting, 0); wake_up_all(&ctx->poll_wq); percpu_ref_put(&ctx->refs); } @@ -2793,6 +2794,9 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait) if (unlikely(!ctx->poll_activated)) io_activate_pollwq(ctx); + + atomic_set(&ctx->poll_wq_waiting, 1); + /* * provides mb() which pairs with barrier from wq_has_sleeper * call in io_commit_cqring diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index f65e3f3ede51..186cee066f9f 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -287,7 +287,7 @@ static inline void io_commit_cqring(struct io_ring_ctx *ctx) static inline void io_poll_wq_wake(struct io_ring_ctx *ctx) { - if (wq_has_sleeper(&ctx->poll_wq)) + if (wq_has_sleeper(&ctx->poll_wq) && atomic_xchg_release(&ctx->poll_wq_waiting, 0) > 0) __wake_up(&ctx->poll_wq, TASK_NORMAL, 0, poll_to_key(EPOLL_URING_WAKE | EPOLLIN)); } -- 2.45.2