From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1FBC386422 for ; Sat, 11 Jul 2026 09:23:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783761789; cv=none; b=dqocjxmwTvxXw7OSJohDcM+BDobXzpWeoZMytpJ/FKr1DYJfaFux3iNF7Fu+ReNUHb0tzq7qV6nI6GoiZ8EyzKbSd3TEbke/o4cyKMxDiscreVaq2TbgXv6CjW/sqrgO89pqBzGbHMwJvBBrAUIasa9Y4M3zcBPUSeTCiiQxLyo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783761789; c=relaxed/simple; bh=TD0DvdzuAgcOk6opvxNJfsY619z4frlhiDPT+sR+SFo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Sqa7j2V141EJOAvioTAYFtcGbAZRTt/2iXM4Vw/xkrLSxOEhxL3WSUhJW/ODajf0ZwnMrb9102UtNGElkdIrXR/Y394/lNuGwoywj2f3nnWzH9TBvnZZTW+v/FtQluQZ3n9uYkBWPzuSW1EDa+jRXpir+R7plWGBiqjzZ1Yjec4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TzoUuPBu; arc=none smtp.client-ip=209.85.218.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TzoUuPBu" Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-c15ba5b151dso233167266b.3 for ; Sat, 11 Jul 2026 02:23:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783761786; x=1784366586; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Gm+lTnqmWwDxeir2CXikUHR7rj6T9P4bxZUxOe+cAx0=; b=TzoUuPBupoFRicy0XwHTMUPznkBhf4Or+RGhsdXH/1o+iar4MAwGj2lPwFniBzNp/M Q92J0QKrYAucb1p1UpZ0Ot5iyKbz7rS6riZv7X2ePzueSN5EiGHtZf3PEgD1ewCZr+D/ T/Ikt4djBfTmV9Z9qDLIw/DTL4WmJycofrIZk/1dOziqFfIYFMOqbbnc7cFM50sWH99E hYGphIXxUyLx8PhFsPem5o8RwSTkPClbr11zKQfsOhm4fINxKFVVCJBLHUHaMX0EdTs/ NOU+Qt8SyoJJYHvY2AZQmvMSGm6H7kWFwZBYDY6HK1vpT2R6n4D1ejSQL+jJ2EaFEAlE R5ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783761786; x=1784366586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Gm+lTnqmWwDxeir2CXikUHR7rj6T9P4bxZUxOe+cAx0=; b=iGxmk4oK/BaPRonykIU+Y+8YBJD8N6JLwCvlWd/1maLTLzbGR9j7+NAcOJzV01CuHs cS5FCk2QZXsD54Me7oEZO13P/hqyCfm62dcQ6SRXWQUYdl92uLgQjg/Ue7kHbBw03xbj K5bE2+KGaOYS/vWbE/EultIDfxltxOFxa06A4JFia2i6gsGq1v1V70knGIUCL5m8rWMX mkDdkGlI/h6IT4bDAqtgb+hMeKNar19XY1LRhR74rZIgdbRk0bfc2569y/zKF52tKAaX vRv6ReCj20Ia/V1AfU3T7qYQB/cskxZnTZRKAgztqO/I2panBCliTpNBC4bJ9i/qIJMS Wmzg== X-Forwarded-Encrypted: i=1; AHgh+RpJUCPPgwaxbvKn48g4s6XibOfSLep8gP2BhkIYeFBmG+Ra8D8K7/LTXxH1cOvl+PrMR+7jc/k=@vger.kernel.org X-Gm-Message-State: AOJu0YzJ0FCnex9ZskWLpKTOUSdSHDS7qtqccDVSL1dWCbHQzsXtBa6M VZ15slmubG+sJ324YX/k7I4v0LbXMDHuOY+83lg5Omw/8Hm1joi/zq5x X-Gm-Gg: AfdE7ck/Yyd7/SwGGith5UjqktzbG90e+E3dsn0FwYaU6iXthnfHzIT2M+8/8TLbGpN 7zFp6qCLV9LOqNH8t1bUSWFkwFkKrGHoWxOMUoOCeYI1ww1EKydjReIssrsdnvdNeAJfKYpNwqg q2/0nrAQymcOSP9J2VqE+ejFfVclcYxliHszNQss/QWT/Xv2uDM7uA/QE1Aga+OH9o0pLc3JzCL YNrpCP6RdMjYHQDPNIq3RnqLgXyRnQNEgqFoFkDEZvjdwXi08J1F7+nTQekzJVALLdQh85cKse5 bkocNWMZDuZGmNAb6DP6kMB9zbcnrG+/ZKvNu3Fpw6bdnE8cbSwEE1Rpg/VyyqX9dkTZkBSxDHE GHGOnckRsSREdtTE/OGJSkwIGWatAvVXzgRMaAfc0emM+FUgI14YTNxWX2xSbRf9zYHEYu6V67s f9YmWu2hyzXs4StAogE/c7Rb+2fqvmG6ZknmPy8PeyPzlWSP9pmBoyMFaqkNmkYqDgy7viZ3omC Mqddly6hXbGfaly9bYV7AZkr2oQ/OMKqmLiCzm5VzUFHZ8= X-Received: by 2002:a17:907:3f9c:b0:c16:26c7:6ba8 with SMTP id a640c23a62f3a-c1626c78accmr54130766b.6.1783761786090; Sat, 11 Jul 2026 02:23:06 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-221-54.dab.02.net. [82.132.221.54]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-c15d3859f69sm517493566b.27.2026.07.11.02.23.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 11 Jul 2026 02:23:04 -0700 (PDT) From: Pavel Begunkov To: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , netdev@vger.kernel.org Cc: io-uring@vger.kernel.org, asml.silence@gmail.com Subject: [RFC 9/9] io_uring/zcrx: add rq_lock cache of "user" niov refs Date: Sat, 11 Jul 2026 10:22:19 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Now the "user" refs are acquired and released by NAPI/page pool for the optimised path, cache them on the NAPI side and protect it by rq.lock. Store it in net_iov as we'd be touching the cache line by refilling soon anyway. Signed-off-by: Pavel Begunkov --- io_uring/zcrx.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index 23669471a8f0..04a80d1a2b3a 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -393,9 +393,20 @@ static inline atomic_t *io_get_user_counter(struct net_iov *niov) static bool io_zcrx_put_niov_uref(struct net_iov *niov, unsigned refs) { - atomic_t *uref = io_get_user_counter(niov); + unsigned *cached_ref = &niov->mp_private; + atomic_t *uref; int old; + lockdep_assert_held(&io_zcrx_iov_to_area(niov)->ifq->rq.lock); + + if (likely(*cached_ref >= refs)) { + *cached_ref -= refs; + return true; + } + refs -= *cached_ref; + *cached_ref = 0; + + uref = io_get_user_counter(niov); old = atomic_read(uref); do { if (unlikely(old < refs)) @@ -744,6 +755,16 @@ static void io_zcrx_scrub_area(struct io_zcrx_ifq *ifq, struct io_zcrx_area *are { int i; + scoped_guard(spinlock_bh, &ifq->rq.lock) { + for (i = 0; i < area->nia.num_niovs; i++) { + struct net_iov *niov = &area->nia.niovs[i]; + unsigned *ref = &niov->mp_private; + + atomic_add(*ref, &area->user_refs[i]); + *ref = 0; + } + } + /* Reclaim back all buffers given to the user space. */ for (i = 0; i < area->nia.num_niovs; i++) { struct net_iov *niov = &area->nia.niovs[i]; @@ -1327,9 +1348,9 @@ static void zcrx_release_skbs(struct io_zcrx_ifq *ifq) for (i = 0; i < shi->nr_frags; i++) { const skb_frag_t *frag = &shi->frags[i]; struct net_iov *niov = netmem_to_net_iov(frag->netmem); + unsigned *ref = &niov->mp_private; - /* Take niov references the skb holds */ - io_zcrx_get_niov_uref(niov); + *ref += 1; } shi->nr_frags = 0; -- 2.54.0