From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793740472; b=BjAIUVsA7sF1B0X0jiVFDAghEipxYdBEHPUOJktaVY8PBYs4fgu0DMDz7DqSC+lZhfML GrSzlbk+cBmci5I5oKdphGqpbS+f2NRGNulhadc96oG5g5IJHJhpuOH8j8yGLFcaHcOoa gwrQXHJHkPq6VmPaavnKh5Hij3N48bhDNqJ4gTcLi+INBiF34Bx7G3Tav2VGjwh6eXLS2 JDmdEIHrbzqVYrJRUjBUdFy29ipNT3ldYwsLlZS1T7Q6931+zymdXSwT6aP7V5FkRzfoT TmGNJqZPw+biX93gEyAwexi/yknmn5wMXEWpK6BAUfR+yhO5ryRpn2ly5ksmmCzbhyw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793740472; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=1hStyrvOpd1GU3yMJi4xSp99VRdlTs+Kdbc/d6N3APkVIWNeZCVvUFh5bssAoeOiPYKp DOd+4Hroan570EskFPhAqi1ZqMEh8dFy4am0oNaFymeCFibjaYzoPa+N0zEur627kAOOt /X+0de+GwNP9ild9exNIGr7wSjDPFkZ3OaY3970/VWEQW8Km4J+K7k0HD8k8Tj8ZPqNbD E2WNDA0rlgP0DEUni8BwJ6zOlLfc2lzjJTGql1xpd13CIOITPKRFQlwH2fgaajgpjkqX8 4SPfB1j73s9Qwf6wGVAnmWg+Ll1ltZavPAnltW9Aqjm4mwt4JtytJHHT8RNYhvlgUMg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=YRT3mnQc; arc=none smtp.remote-ip=74.125.82.199 Received: from mail-dy1-f199.google.com (mail-dy1-f199.google.com [74.125.82.199]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D3C4606EE90 for ; Tue, 3 Nov 2026 21:14:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 9D3C4606EE90 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=YRT3mnQc Received: by mail-dy1-f199.google.com with SMTP id 5a478bee46e88-30bcb065bfdso8206740eec.0 for ; Tue, 03 Nov 2026 13:14:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1793740471; x=1794345271; darn=lists.linux.dev; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=YRT3mnQcH8BY1g4Nky+c8oF5p9jilC28hZmejMhHIA7puFDDtguHTZY1usJeV0zE0B cDQMEpz5Wsr+Ka2v+9ACUgAwlDgDAVryQfMD91QrIJVMKQtHQCJ9ntoeJcsGlZB5aol0 ezAHI6ZydlQ+dzcKZr0gT/kyjgqzXn1oSO9W68SoRS7icdAgTvh+yAV3Z0OL7AH6OvZl zf/4DbPe5xDCVTkroKGw8lkSjkiywSPplkxFij9ZOvOERvmtdb6aINiB1X1+mJUQZouF NZg1l5A4ai+Kh+yt573766fVn5XhrPCIS2llSL4+2W9QHMTmGvGVgN9oS7zZl02H4gsn 32mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793740471; x=1794345271; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=nYyqphy5sePIA5b9x3XfVLfbYkjwm+2X5hFUMywNiUh3CB8EOmyzBXSgZNxwb3KpQ6 fyBsoegeLsl+BWg6l6Yf5pSAhW9jeNUrc/L1iP2fmKdKr6RuoQvLY/A0h2v2qghqKhdx +RZZt5B2is5NxujLdZ8kEu3cLydR/M73vNDidthMRsv5AKgcyqrIYtSMr9FBM4eJKasg A9WeTKn6ehzrXKIlwSTDko8UbbLfa222A+zNK7Q3PT3eIRVR+4yfcgLTeaSMXckbL3ju sqURKmWnhcs0mzkNy8kbv9b5+OXWy52xTb3UJcAb41InbbCS6FfRili8JrmuOGca0Ef2 LwbA== X-Forwarded-Encrypted: i=1; AKwUvBx4peKDW5zFxtd+BB497zsikGLVxcI4nHW8wNulmAXg1cpkn3MgFHoPOYX/nWqZ50uk2ZMwQ9yrtRfqRgSyWA==@lists.linux.dev X-Gm-Message-State: APuU4Z++3nEqVFoaL/eD2QtZFwD0csHbnYgUen+yMvhrCBhh5EX4eDe7 dbMSqARP5FOClm4pC6kvKaPCeHLkoe8xPuK8sJLfesLtn/R7/WutlVJsb8ih/K8FTRGUvGxL46m bImTtBC97b+lQFUOsDpqFtisZ0w== X-Received: from dyvj18-n1.prod.google.com ([2002:a05:693c:87d2:10b0:368:2c6d:de10]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:693c:8959:20b0:369:f5f6:36a7 with SMTP id 5a478bee46e88-36a423d1cc5mr22095354eec.16.1793740470274; Tue, 03 Nov 2026 13:14:30 -0800 (PST) Date: Tue, 3 Nov 2026 21:14:22 +0000 In-Reply-To: <20261103211424.127627-1-almasrymina@google.com> X-Mailing-List: virtualization@lists.linux.dev List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261103211424.127627-1-almasrymina@google.com> X-Mailer: git-send-email 2.56.0.385.gac671c819a-goog Message-ID: <20261103211424.127627-4-almasrymina@google.com> Subject: [PATCH net-next v1 3/5] net: datagram: refcount page_pool frags in zerocopy_fill_skb_from_iter() From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, bpf@vger.kernel.org, linux-wireless@vger.kernel.org, linux-doc@vger.kernel.org, bcm-kernel-feedback-list@broadcom.com, imx@lists.linux.dev, linux-rdma@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-omap@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert , Herbert Xu , Willem de Bruijn , Stanislav Fomichev , Pavel Begunkov , Bobby Eshleman , Andrew Lunn , "Michael S. Tsirkin" , Jason Wang , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Xuan Zhuo , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jesper Dangaard Brouer , Ilias Apalodimas , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sabrina Dubroca , Johannes Berg , Byungchul Park , Jonathan Corbet , Shuah Khan , Randy Dunlap , Lorenzo Bianconi , Wen Gu , Philo Lu , Justin Chen , Florian Fainelli , Vikas Gupta , Michael Chan , Pavan Chebbi , Doug Berger , Nicolai Buchwitz , Satish Kharat , Wei Fang , Frank Li , Shenwei Wang , Joshua Washington , Harshitha Ramamurthy , Jijie Shao , Jian Shen , Fan Gong , Tony Nguyen , Przemek Kitszel , Marcin Wojtas , Russell King , Sunil Goutham , Geetha sowjanya , Ratheesh Kannoth , Subbaraya Sundeep , Bharat Bhushan , Felix Fietkau , Matthias Brugger , AngeloGioacchino Del Regno , Tariq Toukan , Saeed Mahameed , Mark Bloch , Leon Romanovsky , Ido Schimmel , Petr Machata , Horatiu Vultur , UNGLinuxDriver@microchip.com, Daniel Machon , Steen Hegelund , Haiyang Zhang , Wei Liu , Dexuan Cui , Yibo Dong , Brett Creeley , Justin Lai , Larry Chiu , "=?UTF-8?q?Niklas=20S=C3=B6derlund?=" , Paul Barker , Masahisa Kojima , Maxime Chevallier , Maxime Coquelin , Alexandre Torgue , FUJITA Tomonori , Siddharth Vadapalli , Roger Quadros , MD Danish Anwar , Jiawen Wu , Mengyuan Lou , Ronak Doshi , Ryder Lee , Shayne Chen , Sean Wang , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Jordan Rhee , Ankit Garg , Tim Hostetler , Erni Sri Satya Vennela , Eric Joyner , Mohammad Heib , Prabu Thayalan , Chintan Vankar , Nora Schiffer , Alexander Sverdlin , Kevin Hao , Karl Mehltretter , Aleksandr Loktionov , Meghana Malladi , Jacob Keller , StanleyYP Wang Content-Type: text/plain; charset="UTF-8" When an application maps order-0 page_pool RX fragments into user space via TCP_ZEROCOPY_RECEIVE and forwards that mapped address into TX sendmsg(MSG_ZEROCOPY), zerocopy_fill_skb_from_iter() pins the pages via iov_iter_get_pages2() (which increments page->_refcount) while skb_release_data() releases fragments via netmem_dec_ref(). Acquire each fragment reference via netmem_inc_ref() when attaching a fragment in zerocopy_fill_skb_from_iter() and release the corresponding GUP page->_refcount reference via the existing batched page_ref_sub() path so the TX SKB fragment reference pairs with netmem_dec_ref() in skb_release_data() once netmem_inc_ref() and netmem_dec_ref() handle page_pool netmems. Signed-off-by: Mina Almasry --- net/core/datagram.c | 5 ++ net/core/net_test.c | 146 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+) diff --git a/net/core/datagram.c b/net/core/datagram.c index ecc30a6f83023..f2d6d1d7be2b6 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -34,6 +34,7 @@ * */ +#include #include #include #include @@ -696,12 +697,16 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, refs = 0; } skb_fill_page_desc_noacc(skb, frag++, head, start, size); + netmem_inc_ref(page_to_netmem(head)); + last_head = head; + refs++; } if (refs) page_ref_sub(last_head, refs); } return 0; } +EXPORT_SYMBOL_IF_KUNIT(zerocopy_fill_skb_from_iter); static int zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, diff --git a/net/core/net_test.c b/net/core/net_test.c index e20059a00003c..73da75e6ca2c8 100644 --- a/net/core/net_test.c +++ b/net/core/net_test.c @@ -515,7 +515,9 @@ static void ip_tunnel_flags_test_run(struct kunit *test) /* Netmem & SKB fragment refcounting */ +#include #include +#include #include #include #include @@ -841,6 +843,149 @@ static void netmem_ref_test_pp_page(struct kunit *test) page_pool_put_full_page(pool, page, false); page_pool_destroy(pool); } + +int zerocopy_fill_skb_from_iter(struct sk_buff *skb, + struct iov_iter *from, size_t length); + +static void netmem_ref_test_zerocopy_fill_skb_from_iter(struct kunit *test) +{ + struct page_pool_params pp_params = { + .order = 0, + .pool_size = 4, + .nid = NUMA_NO_NODE, + }; + struct page *pp_page, *comp_page, *reg_page; + struct page_pool *pool, *comp_pool; + int base_ref, base_comp_ref, base_reg_ref; + long base_pp_ref, base_comp_pp_ref; + struct folio_queue folioq; + struct bio_vec bvec[3]; + struct iov_iter iter; + struct sk_buff *skb; + + pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool); + pp_page = page_pool_dev_alloc_pages(pool); + KUNIT_ASSERT_NOT_NULL(test, pp_page); + base_ref = page_ref_count(pp_page); + base_pp_ref = atomic_long_read(&pp_page->pp_ref_count); + + /* 1. Single order-0 page_pool page (simulating TCP_ZEROCOPY_RECEIVE + * page forwarded into MSG_ZEROCOPY TX via zerocopy_fill_skb_from_iter). + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 128, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 1, 128); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 128), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 2. Contiguous segments on the same page_pool page coalesce into a + * single skb_frag_t while a non-contiguous segment creates a second + * skb_frag_t; all 3 GUP _refcounts are dropped. + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 64, 0); + bvec_set_page(&bvec[1], pp_page, 64, 64); + bvec_set_page(&bvec[2], pp_page, 64, 256); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 3, 192); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 192), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), 128U); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[1]), 64U); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 2); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 3. Order-1 compound page_pool page spanning 2 subpages in a single + * iov_iter_get_pages2() batch coalesces into 1 skb_frag_t and drops + * both GUP _refcounts via batched page_ref_sub, alongside a + * non-page_pool page in the same skb. + */ + pp_params.order = 1; + comp_pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, comp_pool); + comp_page = page_pool_dev_alloc_pages(comp_pool); + KUNIT_ASSERT_NOT_NULL(test, comp_page); + base_comp_ref = page_ref_count(comp_page); + base_comp_pp_ref = atomic_long_read(&comp_page->pp_ref_count); + + reg_page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, reg_page); + base_reg_ref = page_ref_count(reg_page); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], comp_page, 2 * PAGE_SIZE, 0); + bvec_set_page(&bvec[1], reg_page, 64, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 2, 2 * PAGE_SIZE + 64); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 2 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), + (unsigned int)(2 * PAGE_SIZE)); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + + /* 4. Single-batch mixed folios via ITER_FOLIOQ: order-1 compound + * page_pool folio (2 subpages) + non-page_pool folio + order-0 + * page_pool folio returned in a single iov_iter_get_pages2() call. + */ + folioq_init(&folioq, 0); + folioq_append(&folioq, page_folio(comp_page)); + folioq_append(&folioq, page_folio(reg_page)); + folioq_append(&folioq, page_folio(pp_page)); + iov_iter_folio_queue(&iter, ITER_SOURCE, &folioq, 0, 0, + 3 * PAGE_SIZE + 64); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 3 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 3); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + __free_page(reg_page); + page_pool_put_full_page(comp_pool, comp_page, false); + page_pool_destroy(comp_pool); + page_pool_put_full_page(pool, pp_page, false); + page_pool_destroy(pool); +} #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) @@ -1738,6 +1883,7 @@ static struct kunit_case net_test_cases[] = { KUNIT_CASE(netmem_ref_test_page), #ifdef CONFIG_PAGE_POOL KUNIT_CASE(netmem_ref_test_pp_page), + KUNIT_CASE(netmem_ref_test_zerocopy_fill_skb_from_iter), #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) KUNIT_CASE(netmem_ref_test_net_iov), -- 2.56.0.385.gac671c819a-goog