From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=74.125.82.197 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793740473; b=5VDdgQSiqr5z/PyeRprKZQy66GEoanwFG2LHNQQKbh1nWICuQtSyMS9wiYuIqv/eBama W4qVNsflUQ5llnHpXp3dqPSBboNxXGcke0Y6NZM6H+kbRiQ77SYvWrEuxNtJ/m3URwdk2 arQPq3kpVbLS0uWrTWCToRQCtPykwnw3I6RhFuRSz2Jsl3/nveMWz2LNzZNxCD01xA81h aoJvt6TZlh+ZANpYVws8pUYC7RvAoKGnu/wpezlY43R5kITGqXv2xQGbQ24hMx8PPoKfO nbIL0E6Ie91yVhxootlh9dnHnCA3QCcgn9mq1f7fmP+c3uEaVPMZQHqsXxzCm4r9TQw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793740473; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Received:Date:In-Reply-To:Mime-Version: References:X-Mailer:Message-ID:Subject:From:To:Cc:Content-Type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=A4IEMrcs2wDchhjx19Rtlw2bFTyrBD9apAWsZGrl5hzvs0xOBtU1LXDv2cyvat/0sEGE bAvFaxMA+BCBg2zCQdsQZ5HKQxOgVMg06OK8APX+fRlsyHLlru95PR4srMw9TFtGrcTRJ 7AvK29OjD+y88TDNE/i8mtMaZ/mMBa9TkzqMd/M/gP2Fquh09JPsUkwaurzfhW3vBx6GQ 4KM//hgfNDfQESYhinh8g7Hf3NCI5ceKl/enWvrQpTYRUoT/XHr//zo1iU+SRRH5yrasO PVUEexy4h6AJ5U5zEu8I+MFcBBr24Bd4B7THq8NRrMKrUy/fyHinOFvfAB66y7xy+jA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=Bn8QTbYn; arc=none smtp.remote-ip=74.125.82.197 Received: from mail-dy1-f197.google.com (mail-dy1-f197.google.com [74.125.82.197]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21777606A32F for ; Tue, 3 Nov 2026 21:14:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 21777606A32F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--almasrymina.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=Bn8QTbYn Received: by mail-dy1-f197.google.com with SMTP id 5a478bee46e88-33c35f5ca6cso9644570eec.1 for ; Tue, 03 Nov 2026 13:14:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1793740472; x=1794345272; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=Bn8QTbYnsNcdzoP1IKBIVuaqKvmxdMuRER/Jd+ExK2ZLaezIC13CGuDvOAVIZCtHro COGpTMlWuNJO44iqiwX/L1i9VjYFoAzbwrbKf8lj0JlZVc7B8kZLUORCXHsfZpadhKVR 1f4jiZF/1OCST/mKuDF1VZ8Sig9vsBAU2Dpbsf1ioH80LhGO1lGbYgkZZGN2Ljb3+El9 h6mej0YlxU6sGHRd/00cCdiKy2KhUTUUp8kYTFueqakyOie6op7ZRw1Wxp+Wz7Vyx5u5 84mSpsUZ2j9C5K4Ftu1vtso8DEHDGLD1xW3u8VQS/Mzd9uUcqPcDWbTtp9fiQBtqzN7m vEIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793740472; x=1794345272; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+2yCJaGEbyCpc4GPm4MdejLrIvWaF/SQGhmWy+Wu8aI=; b=KGxgMJJIYYkiOqPjUBU8v/9pZNJNp1QTyuYmgaqc48RzFsOE/B7h99vewRYVxZaQNX S10iBbCHA42eqsQCUo2XwdOUofxbB7AhtCNUq5og4G12z9y+xPdXdsxwQuaCqE929ddt eEiAJlIk2a4br2Fpu/Eb/T9fpQh1i3QmX6FCEHlfMoS9HJwE9F0KcTgf9qnErl83m+31 CbPeSzZ3beCSMziKkKCQavzO0xTyFdHOJcTAH2XLpX39MI85LGIbVNreCE+7cNPAvprx VKhAaEoWvDlXCeEItPAoN2WPru/k1kad6Eg8Xhppo8c+h6+3pU5SwPbpspVBjcNIrht5 QRaA== X-Gm-Message-State: APuU4Z9REMx9kGFkYZenwog1H2ptUhVq0PPH6LykKJOf7g0pneC572DD XC4FtE2FTli5T0i6JFHR74wh4Er+pogU9BJ44mCMMRMX0zi9girgLlDcAGly4h/2ltmOq2F7b+Y pcyRZLbgHLInqWJnH86Gc9KW3j01bFgdk8AmmN9fObgvnkByp0T2qSeDv6buk0z6P6NdyXHYBx2 PWGX+o9XUazzIZw7rAulhsKcaqHsP05D1wIvkFNSe0q8mdRMgzkfora5xqxZgjvsw= X-Received: from dyvj18-n1.prod.google.com ([2002:a05:693c:87d2:10b0:368:2c6d:de10]) (user=almasrymina job=prod-delivery.src-stubby-dispatcher) by 2002:a05:693c:8959:20b0:369:f5f6:36a7 with SMTP id 5a478bee46e88-36a423d1cc5mr22095354eec.16.1793740470274; Tue, 03 Nov 2026 13:14:30 -0800 (PST) Date: Tue, 3 Nov 2026 21:14:22 +0000 In-Reply-To: <20261103211424.127627-1-almasrymina@google.com> X-Mailing-List: netdev@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261103211424.127627-1-almasrymina@google.com> X-Mailer: git-send-email 2.56.0.385.gac671c819a-goog Message-ID: <20261103211424.127627-4-almasrymina@google.com> Subject: [PATCH net-next v1 3/5] net: datagram: refcount page_pool frags in zerocopy_fill_skb_from_iter() From: Mina Almasry To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux.dev, linux-mm@kvack.org, bpf@vger.kernel.org, linux-wireless@vger.kernel.org, linux-doc@vger.kernel.org, bcm-kernel-feedback-list@broadcom.com, imx@lists.linux.dev, linux-rdma@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-renesas-soc@vger.kernel.org, linux-omap@vger.kernel.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Simon Horman , Steffen Klassert , Herbert Xu , Willem de Bruijn , Stanislav Fomichev , Pavel Begunkov , Bobby Eshleman , Andrew Lunn , "Michael S. Tsirkin" , Jason Wang , "=?UTF-8?q?Eugenio=20P=C3=A9rez?=" , Xuan Zhuo , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Jesper Dangaard Brouer , Ilias Apalodimas , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Sabrina Dubroca , Johannes Berg , Byungchul Park , Jonathan Corbet , Shuah Khan , Randy Dunlap , Lorenzo Bianconi , Wen Gu , Philo Lu , Justin Chen , Florian Fainelli , Vikas Gupta , Michael Chan , Pavan Chebbi , Doug Berger , Nicolai Buchwitz , Satish Kharat , Wei Fang , Frank Li , Shenwei Wang , Joshua Washington , Harshitha Ramamurthy , Jijie Shao , Jian Shen , Fan Gong , Tony Nguyen , Przemek Kitszel , Marcin Wojtas , Russell King , Sunil Goutham , Geetha sowjanya , Ratheesh Kannoth , Subbaraya Sundeep , Bharat Bhushan , Felix Fietkau , Matthias Brugger , AngeloGioacchino Del Regno , Tariq Toukan , Saeed Mahameed , Mark Bloch , Leon Romanovsky , Ido Schimmel , Petr Machata , Horatiu Vultur , UNGLinuxDriver@microchip.com, Daniel Machon , Steen Hegelund , Haiyang Zhang , Wei Liu , Dexuan Cui , Yibo Dong , Brett Creeley , Justin Lai , Larry Chiu , "=?UTF-8?q?Niklas=20S=C3=B6derlund?=" , Paul Barker , Masahisa Kojima , Maxime Chevallier , Maxime Coquelin , Alexandre Torgue , FUJITA Tomonori , Siddharth Vadapalli , Roger Quadros , MD Danish Anwar , Jiawen Wu , Mengyuan Lou , Ronak Doshi , Ryder Lee , Shayne Chen , Sean Wang , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Jordan Rhee , Ankit Garg , Tim Hostetler , Erni Sri Satya Vennela , Eric Joyner , Mohammad Heib , Prabu Thayalan , Chintan Vankar , Nora Schiffer , Alexander Sverdlin , Kevin Hao , Karl Mehltretter , Aleksandr Loktionov , Meghana Malladi , Jacob Keller , StanleyYP Wang Content-Type: text/plain; charset="UTF-8" When an application maps order-0 page_pool RX fragments into user space via TCP_ZEROCOPY_RECEIVE and forwards that mapped address into TX sendmsg(MSG_ZEROCOPY), zerocopy_fill_skb_from_iter() pins the pages via iov_iter_get_pages2() (which increments page->_refcount) while skb_release_data() releases fragments via netmem_dec_ref(). Acquire each fragment reference via netmem_inc_ref() when attaching a fragment in zerocopy_fill_skb_from_iter() and release the corresponding GUP page->_refcount reference via the existing batched page_ref_sub() path so the TX SKB fragment reference pairs with netmem_dec_ref() in skb_release_data() once netmem_inc_ref() and netmem_dec_ref() handle page_pool netmems. Signed-off-by: Mina Almasry --- net/core/datagram.c | 5 ++ net/core/net_test.c | 146 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 151 insertions(+) diff --git a/net/core/datagram.c b/net/core/datagram.c index ecc30a6f83023..f2d6d1d7be2b6 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -34,6 +34,7 @@ * */ +#include #include #include #include @@ -696,12 +697,16 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb, refs = 0; } skb_fill_page_desc_noacc(skb, frag++, head, start, size); + netmem_inc_ref(page_to_netmem(head)); + last_head = head; + refs++; } if (refs) page_ref_sub(last_head, refs); } return 0; } +EXPORT_SYMBOL_IF_KUNIT(zerocopy_fill_skb_from_iter); static int zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from, diff --git a/net/core/net_test.c b/net/core/net_test.c index e20059a00003c..73da75e6ca2c8 100644 --- a/net/core/net_test.c +++ b/net/core/net_test.c @@ -515,7 +515,9 @@ static void ip_tunnel_flags_test_run(struct kunit *test) /* Netmem & SKB fragment refcounting */ +#include #include +#include #include #include #include @@ -841,6 +843,149 @@ static void netmem_ref_test_pp_page(struct kunit *test) page_pool_put_full_page(pool, page, false); page_pool_destroy(pool); } + +int zerocopy_fill_skb_from_iter(struct sk_buff *skb, + struct iov_iter *from, size_t length); + +static void netmem_ref_test_zerocopy_fill_skb_from_iter(struct kunit *test) +{ + struct page_pool_params pp_params = { + .order = 0, + .pool_size = 4, + .nid = NUMA_NO_NODE, + }; + struct page *pp_page, *comp_page, *reg_page; + struct page_pool *pool, *comp_pool; + int base_ref, base_comp_ref, base_reg_ref; + long base_pp_ref, base_comp_pp_ref; + struct folio_queue folioq; + struct bio_vec bvec[3]; + struct iov_iter iter; + struct sk_buff *skb; + + pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, pool); + pp_page = page_pool_dev_alloc_pages(pool); + KUNIT_ASSERT_NOT_NULL(test, pp_page); + base_ref = page_ref_count(pp_page); + base_pp_ref = atomic_long_read(&pp_page->pp_ref_count); + + /* 1. Single order-0 page_pool page (simulating TCP_ZEROCOPY_RECEIVE + * page forwarded into MSG_ZEROCOPY TX via zerocopy_fill_skb_from_iter). + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 128, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 1, 128); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 128), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 2. Contiguous segments on the same page_pool page coalesce into a + * single skb_frag_t while a non-contiguous segment creates a second + * skb_frag_t; all 3 GUP _refcounts are dropped. + */ + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], pp_page, 64, 0); + bvec_set_page(&bvec[1], pp_page, 64, 64); + bvec_set_page(&bvec[2], pp_page, 64, 256); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 3, 192); + KUNIT_ASSERT_EQ(test, zerocopy_fill_skb_from_iter(skb, &iter, 192), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), 128U); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[1]), 64U); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 2); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + /* 3. Order-1 compound page_pool page spanning 2 subpages in a single + * iov_iter_get_pages2() batch coalesces into 1 skb_frag_t and drops + * both GUP _refcounts via batched page_ref_sub, alongside a + * non-page_pool page in the same skb. + */ + pp_params.order = 1; + comp_pool = page_pool_create(&pp_params); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, comp_pool); + comp_page = page_pool_dev_alloc_pages(comp_pool); + KUNIT_ASSERT_NOT_NULL(test, comp_page); + base_comp_ref = page_ref_count(comp_page); + base_comp_pp_ref = atomic_long_read(&comp_page->pp_ref_count); + + reg_page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, reg_page); + base_reg_ref = page_ref_count(reg_page); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + bvec_set_page(&bvec[0], comp_page, 2 * PAGE_SIZE, 0); + bvec_set_page(&bvec[1], reg_page, 64, 0); + iov_iter_bvec(&iter, ITER_SOURCE, bvec, 2, 2 * PAGE_SIZE + 64); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 2 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 2); + KUNIT_EXPECT_EQ(test, skb_frag_size(&skb_shinfo(skb)->frags[0]), + (unsigned int)(2 * PAGE_SIZE)); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + + /* 4. Single-batch mixed folios via ITER_FOLIOQ: order-1 compound + * page_pool folio (2 subpages) + non-page_pool folio + order-0 + * page_pool folio returned in a single iov_iter_get_pages2() call. + */ + folioq_init(&folioq, 0); + folioq_append(&folioq, page_folio(comp_page)); + folioq_append(&folioq, page_folio(reg_page)); + folioq_append(&folioq, page_folio(pp_page)); + iov_iter_folio_queue(&iter, ITER_SOURCE, &folioq, 0, 0, + 3 * PAGE_SIZE + 64); + + skb = alloc_skb(0, GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, skb); + KUNIT_ASSERT_EQ(test, + zerocopy_fill_skb_from_iter(skb, &iter, + 3 * PAGE_SIZE + 64), 0); + KUNIT_EXPECT_EQ(test, skb_shinfo(skb)->nr_frags, 3); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref + 1); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref + 1); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref + 1); + consume_skb(skb); + KUNIT_EXPECT_EQ(test, atomic_long_read(&comp_page->pp_ref_count), + base_comp_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(comp_page), base_comp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(reg_page), base_reg_ref); + KUNIT_EXPECT_EQ(test, atomic_long_read(&pp_page->pp_ref_count), + base_pp_ref); + KUNIT_EXPECT_EQ(test, page_ref_count(pp_page), base_ref); + + __free_page(reg_page); + page_pool_put_full_page(comp_pool, comp_page, false); + page_pool_destroy(comp_pool); + page_pool_put_full_page(pool, pp_page, false); + page_pool_destroy(pool); +} #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) @@ -1738,6 +1883,7 @@ static struct kunit_case net_test_cases[] = { KUNIT_CASE(netmem_ref_test_page), #ifdef CONFIG_PAGE_POOL KUNIT_CASE(netmem_ref_test_pp_page), + KUNIT_CASE(netmem_ref_test_zerocopy_fill_skb_from_iter), #endif #if defined(CONFIG_PAGE_POOL) && defined(CONFIG_NET_DEVMEM) KUNIT_CASE(netmem_ref_test_net_iov), -- 2.56.0.385.gac671c819a-goog