From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.218.71 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795136839; b=GnW60+MEtXDZNnzeGuc5ftKupEd4pK9jUsWuVZehiIiv10+LLy+nFAO0gpH04ADkBcj7 2mQ0UqCTiomLEIxszNkCSuPd5MnPKrt8UibXNzIRkZW33xyOsNuyVz9Z17/Hw75xRZRoB 7JwoTQ+Hat6U75h1sWnvzm6rLTAbJOav/f9vkiGYRGhszn5PTzrmxxjV4PrT9XfEVET6y DP5knRekXZOroqhiuWKab3/Eeew5+zBw+WgnxzFxzzsmWZa80DDxsdgPU54q0j+5YjlTW cH12X70VrgreaytrmGHgzxCE13IKCMrghl2/HoorQ216jTOuU2vDvI/DCv9+YMViNog== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795136839; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=+NFTm4L2VhbLUpPdiBhNTRzw8j2BUIljmUbg5vzbrRk=; b=KeFZ2M8rOiWwIVc1mqbtyuBgowXe0Yf80VZEEJN7++n8dGUqYheNDnbHQqF3XbksMJvQ Yi1US6a9aEKzAJBGzhWc4ZwJgOsposP2F3Ga76ka8A99u30vO2hLvGzZxNyq1a1uJZAXR DPA1lt2kChUGoVxIYlWHjZcHgrPXMjw+a/DJToMJOldqzPzWi74535Dijrgz4bQV22aaK vVL+fNGfifuNQrmxOt72IFTrmYx66k+2OCi9kz3IHKHKTgEPz/RZJzMdSx4OgtAFZuyhu AK3h2HEez20dazml9dyErNPtxgXMfnGwuqkB4W9tEl8/8JHFf4aA9M9jE/RO5vU+HJQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=PpVCnIGb; arc=none smtp.remote-ip=209.85.218.71 Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43A36606A61E for ; Fri, 20 Nov 2026 01:07:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 43A36606A61E Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=PpVCnIGb Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-c50549f6cf1so22010766b.3 for ; Thu, 19 Nov 2026 17:07:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1795136837; x=1795741637; darn=lists.linux.dev; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+NFTm4L2VhbLUpPdiBhNTRzw8j2BUIljmUbg5vzbrRk=; b=PpVCnIGbHJc7A4Ksy9cO8/FbMewzkd4CvocV39iKLWToKM++wamapsCSAQA1cDbr7v eRaap8tAjwcNAKbtPKHO+d4DmT4kl2uy6b3bLWwvEiQEUXI+zscp5d6r0dotVCoO5K8i oWAeOddw6NS/7YRLZOEI2Tk0Hb7h4vsKhnbjD8uU/dCczhC50bqDDTEzHtBEHv6NqOZb 9mh3dG/8TPBT6a+ODABW6gQIRRWZe3gmOx4VGsC3U+MzBxTG5nxeeW4Vw1rzb/ybC2oV DbU8Eam89BL1oU+dsvhfCYaEk9WUttAhtDWpmFFmxTf5QDpBoboBY2/C/Bgt31ZstSYX jIdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795136837; x=1795741637; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+NFTm4L2VhbLUpPdiBhNTRzw8j2BUIljmUbg5vzbrRk=; b=n5HCcSyzAUXs9ZUe1oQcKWg5ki3NqfGlC/wmoCVgQ/zP7e5r6jO+3rF5lpd4X9f2Cb OTikt/HRMHbeJNe50Pb8gcXQEICqf76OFSwHjYPpv1gcFUQh59CiFSV8FvJzL+sm/MTr b6DIZ3cQE7WFRdVqGVSTGVYbplTECeo510mlvpynoChwSSW/hyFTEtc9AWY4x289PfCO W5cfpad13PhABo1L47FBA+kKib/mua+5xUeVv/TGBpSuwiMp3lIZyeaw/ENDuZ1ZOqfG q+UudagPpV1nD9OgeGXBS9T/hRNdXSxHjbiG6FQaeyjEt2ZXZy725TOHZyV4mTEAjKQc OD1Q== X-Forwarded-Encrypted: i=1; AL+hErHWSSceOv89m5U/nPkv/2qRRou7fTn8UOCWkqWd+oE5NO9RQA8Pdyt+rnNrv5MMXzNu3BVeYg==@lists.linux.dev X-Gm-Message-State: APuU4Z9dSeOWdpfARfsVb0nxFTwN4rH5yUfGOP/jEH2YK+1P3wzp3Fz8 B2DKrULXbELBvIO2WhY6AE/IkFAqvTS9E1kXFTc6QaEAmRZ2c6rbds37F3WsDSXpFvEte1VkRo7 aGbQWQg== X-Received: from ejzb4.prod.google.com ([2002:a17:906:d104:b0:c50:fe53:c3ad]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:b02d:b0:c34:1ed:54d9 with SMTP id a640c23a62f3a-c513a52b544mr118213766b.4.1795136836226; Thu, 19 Nov 2026 17:07:16 -0800 (PST) Date: Fri, 20 Nov 2026 01:06:44 +0000 In-Reply-To: <20261120010655.467825-1-lrizzo@google.com> X-Mailing-List: iommu@lists.linux.dev List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261109182533.3594984-1-lrizzo@google.com> <20261120010655.467825-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.1.809.ga0b61035b5-goog Message-ID: <20261120010655.467825-15-lrizzo@google.com> Subject: [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() From: Luigi Rizzo To: Andrew Morton , David Hildenbrand , Johannes Weiner , Vlastimil Babka , Joerg Roedel , Will Deacon , Robin Murphy , Marek Szyprowski , Jens Axboe , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Luigi Rizzo Cc: Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , Harry Yoo , Christoph Lameter , David Rientjes , Roman Gushchin , Hao Li , Matthew Wilcox , Jan Kara , Jann Horn , Catalin Marinas , Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Simon Horman , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Alexander Lobakin , Christoph Hellwig , Jason Gunthorpe , Dragos Tatulea , Naman Gulati , Pranjal Shrivastava , Mostafa Saleh , Randy Dunlap , linux-mm@kvack.org, iommu@lists.linux.dev, netdev@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Transmit socket buffers use memory coming from skb_page_frag_refill(). Sockets whose leaf device has enabled pisa_tx_bufs should use PISA memory. Add a dedicated per-task page_frag, current->task_frag_pisa, populated with read-only PISA blocks via __GFP_PISA, and select it on TX when the socket's (or cork's) route goes out in the clear through a PISA-enabled device. Special paths that hold plaintext before encryption (IPsec, kTLS) or do not use a DMA device keep using regular pages; ESP output uses pisa_tx_bufs_gfp() to opt in when the egress device has pisa_tx_bufs enabled. Signed-off-by: Luigi Rizzo --- include/linux/pisa.h | 15 +++++++++++++++ include/linux/sched.h | 4 ++++ include/net/pisa.h | 33 +++++++++++++++++++++++++++++++++ kernel/exit.c | 2 ++ kernel/fork.c | 2 ++ net/core/sock.c | 10 ++++++++++ net/ipv4/esp4.c | 7 +++++-- net/ipv4/ip_output.c | 12 ++++++++---- net/ipv4/tcp.c | 3 ++- net/ipv6/esp6.c | 7 +++++-- net/ipv6/ip6_output.c | 12 ++++++++---- 11 files changed, 94 insertions(+), 13 deletions(-) diff --git a/include/linux/pisa.h b/include/linux/pisa.h index 6d184e393e297..84692852299f1 100644 --- a/include/linux/pisa.h +++ b/include/linux/pisa.h @@ -202,6 +202,18 @@ static inline bool pisa_page_can_recycle(const struct page *page) __pisa_page_can_recycle(page); } +/* Hooks for dup_task_struct() and do_exit() (guarded by @pisa_key on release). */ +static inline void pisa_task_frag_init(struct task_struct *tsk) +{ + tsk->task_frag_pisa.page = NULL; +} + +static inline void pisa_task_frag_release(struct task_struct *tsk) +{ + if (pisa_enabled() && tsk->task_frag_pisa.page) + put_page(tsk->task_frag_pisa.page); +} + #else /* !CONFIG_PISA */ static inline int pisa_meta_init(void) { return -EOPNOTSUPP; } @@ -240,6 +252,9 @@ static inline bool pisa_page_can_recycle(const struct page *page) return true; } +static inline void pisa_task_frag_init(struct task_struct *tsk) {} +static inline void pisa_task_frag_release(struct task_struct *tsk) {} + static inline bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages, size_t size, size_t offset, diff --git a/include/linux/sched.h b/include/linux/sched.h index e4f6323d78555..a8edeeb2661f4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1455,6 +1455,10 @@ struct task_struct { struct pipe_inode_info *splice_pipe; struct page_frag task_frag; +#ifdef CONFIG_PISA + /* TX frag that may hold PISA pages, see sk_page_frag_dst(). */ + struct page_frag task_frag_pisa; +#endif #ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE struct lazy_mmu_state lazy_mmu_state; diff --git a/include/net/pisa.h b/include/net/pisa.h index 1f95e67e99bf8..8e23d8b01e73d 100644 --- a/include/net/pisa.h +++ b/include/net/pisa.h @@ -4,6 +4,7 @@ #define _NET_PISA_H #include +#include #include /* @@ -72,10 +73,42 @@ static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) __pisa_tx_gfp(skb_dst(skb), PISA_TX_FLAG_BUFS | PISA_TX_XFRM_OK) : 0; } +/* Hook for skb_page_frag_refill() and __ip*_append_data() (@net_pisa_tx_bufs_key). */ +static inline bool is_pisa_pfrag(const struct page_frag *pfrag) +{ + return static_branch_unlikely(&net_pisa_tx_bufs_key) && + pfrag == ¤t->task_frag_pisa; +} + +/* Hook for task/socket TX page_frag selection (@net_pisa_tx_bufs_key). */ +static inline struct page_frag *task_page_frag(const struct dst_entry *dst) +{ + return use_pisa_tx_bufs(dst) ? ¤t->task_frag_pisa : ¤t->task_frag; +} + +static inline struct page_frag *sk_page_frag_dst(struct sock *sk, + const struct dst_entry *dst) +{ + return static_branch_unlikely(&net_pisa_tx_bufs_key) && + sk->sk_use_task_frag ? task_page_frag(dst) : sk_page_frag(sk); +} + #else /* !CONFIG_PISA */ static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) { return false; } static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) { return 0; } +static inline bool is_pisa_pfrag(const struct page_frag *pfrag) { return false; } + +static inline struct page_frag *task_page_frag(const struct dst_entry *dst) +{ + return ¤t->task_frag; +} + +static inline struct page_frag *sk_page_frag_dst(struct sock *sk, + const struct dst_entry *dst) +{ + return sk_page_frag(sk); +} #endif /* CONFIG_PISA */ #endif /* _NET_PISA_H */ diff --git a/kernel/exit.c b/kernel/exit.c index d5e7bc2af804a..ac7ca1fb180c3 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -6,6 +6,7 @@ */ #include +#include #include #include #include @@ -1009,6 +1010,7 @@ void __noreturn do_exit(long code) if (tsk->task_frag.page) put_page(tsk->task_frag.page); + pisa_task_frag_release(tsk); exit_task_stack_account(tsk); diff --git a/kernel/fork.c b/kernel/fork.c index 5efceb8d58a78..d235d7ab04687 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include #include @@ -982,6 +983,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) #endif tsk->splice_pipe = NULL; tsk->task_frag.page = NULL; + pisa_task_frag_init(tsk); tsk->wake_q.next = NULL; tsk->worker_private = NULL; diff --git a/net/core/sock.c b/net/core/sock.c index ad414878dc5ba..56dea49baaec2 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -115,6 +115,7 @@ #include #include #include +#include #include #include #include @@ -147,6 +148,7 @@ #include #include #include +#include #include @@ -3257,13 +3259,21 @@ DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); */ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp) { + /* PISA pages only go into current->task_frag_pisa, see sk_page_frag_dst(). */ + if (is_pisa_pfrag(pfrag)) + gfp |= __GFP_PISA; + if (pfrag->page) { if (page_ref_count(pfrag->page) == 1) { + if (!!(gfp & __GFP_PISA) != is_pisa_page(pfrag->page) || + ((gfp & __GFP_PISA) && !pisa_page_can_recycle(pfrag->page))) + goto drop_page; pfrag->offset = 0; return true; } if (pfrag->offset + sz <= pfrag->size) return true; +drop_page: put_page(pfrag->page); } diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c index ae701561d9f12..8ed713772e4ee 100644 --- a/net/ipv4/esp4.c +++ b/net/ipv4/esp4.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -451,7 +452,8 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info * spin_lock_bh(&x->lock); - if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) { + if (unlikely(!skb_page_frag_refill(allocsize, pfrag, + GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) { spin_unlock_bh(&x->lock); goto cow; } @@ -560,7 +562,8 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info * allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES); spin_lock_bh(&x->lock); - if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) { + if (unlikely(!skb_page_frag_refill(allocsize, pfrag, + GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) { spin_unlock_bh(&x->lock); goto error_free; } diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index d42d776af273e..50b8c92c3cd5e 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -71,6 +71,7 @@ #include #include #include +#include #include #include #include @@ -986,7 +987,10 @@ static int __ip_append_data(struct sock *sk, exthdrlen = !skb ? rt->dst.header_len : 0; mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - paged = !!cork->gso_size; + paged = !!cork->gso_size || + (getfrag == ip_generic_getfrag && + (rt->dst.dev->features & NETIF_F_SG) && + is_pisa_pfrag(pfrag)); hh_len = LL_RESERVED_SPACE(rt->dst.dev); @@ -1383,8 +1387,8 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, } return __ip_append_data(sk, fl4, &sk->sk_write_queue, &inet->cork.base, - sk_page_frag(sk), getfrag, - from, length, transhdrlen, flags); + sk_page_frag_dst(sk, inet->cork.base.dst), + getfrag, from, length, transhdrlen, flags); } static void ip_cork_release(struct inet_cork *cork) @@ -1592,7 +1596,7 @@ struct sk_buff *ip_make_skb(struct sock *sk, return ERR_PTR(err); err = __ip_append_data(sk, fl4, &queue, cork, - ¤t->task_frag, getfrag, + sk_page_frag_dst(sk, cork->dst), getfrag, from, length, transhdrlen, flags); if (err) { __ip_flush_pending_frames(sk, &queue, cork); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 832b25ddd3c2e..3c3edc48705d7 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -273,6 +273,7 @@ #include #include #include +#include #include #include #include @@ -1295,7 +1296,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; - struct page_frag *pfrag = sk_page_frag(sk); + struct page_frag *pfrag = sk_page_frag_dst(sk, __sk_dst_get(sk)); if (!sk_page_frag_refill(sk, pfrag)) goto wait_for_space; diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c index d654f0f442388..98ed9120ae7b4 100644 --- a/net/ipv6/esp6.c +++ b/net/ipv6/esp6.c @@ -36,6 +36,7 @@ #include #include #include +#include #include #include @@ -481,7 +482,8 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info spin_lock_bh(&x->lock); - if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) { + if (unlikely(!skb_page_frag_refill(allocsize, pfrag, + GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) { spin_unlock_bh(&x->lock); goto cow; } @@ -592,7 +594,8 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES); spin_lock_bh(&x->lock); - if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) { + if (unlikely(!skb_page_frag_refill(allocsize, pfrag, + GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) { spin_unlock_bh(&x->lock); goto error_free; } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index cb648c8394310..8f6f0bed15499 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -40,6 +40,7 @@ #include #include +#include #include #include @@ -1489,7 +1490,10 @@ static int __ip6_append_data(struct sock *sk, dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len; } - paged = !!cork->gso_size; + paged = !!cork->gso_size || + (getfrag == ip_generic_getfrag && + (rt->dst.dev->features & NETIF_F_SG) && + is_pisa_pfrag(pfrag)); mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize; orig_mtu = mtu; @@ -1892,8 +1896,8 @@ int ip6_append_data(struct sock *sk, } return __ip6_append_data(sk, &sk->sk_write_queue, &inet->cork, - sk_page_frag(sk), getfrag, - from, length, transhdrlen, flags); + sk_page_frag_dst(sk, inet->cork.base.dst), + getfrag, from, length, transhdrlen, flags); } EXPORT_SYMBOL_GPL(ip6_append_data); @@ -2095,7 +2099,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, } err = __ip6_append_data(sk, &queue, cork, - ¤t->task_frag, getfrag, from, + sk_page_frag_dst(sk, cork->base.dst), getfrag, from, length + exthdrlen, transhdrlen + exthdrlen, flags); if (err) { -- 2.56.1.809.ga0b61035b5-goog