From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.128.70 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795136837; b=H/uchp7dDqHs07kIZygdBSDsDBWi4byO/zUC9DjQavy7HyhOhLDyGwxlLZuj4nY/wKTz eCydH9kdaC0BxVrtFAvgkSiBTtVQX7JPvr6024IfiuA90cMKln/lVIspr+nf5HDYg8VMk 1wZuSesGZwvCsVenfBG8pLgIEwm3+pZdHpqVwensNFI05IkUe94olHn34xi+BQ1Elob4d +3p4KxKAMMyw9Vpr4E5XgSprnQUzkVeMJ9LyHpkC8Nlysxy2nM0Ed75/JZXThnJyPrOWL Nnd8FcvS4sEseuq0Bmm5tQYKbt5GAERC719PI/WeDmkBMvfOB7wdA9aFTNDI4Bky1cw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795136837; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=FkHIJ0AqEEou0V4yLcGWQGEIWwBaAdh9SIA/wm+Eg+o=; b=pwXRSQr0NKQiu0eLJ2Y4RdjT1tqGkXJ2iz6w9ZsgAYsCchwdRFxuufdvuYbz26zssOdB fprcYqTm+qea68ryMOvdAP+gCidFWbCDR+KvqjbRshZ5Ky3kw+jho332WPiGTdG8COcrc GzYMyXLa3HY8cHGeEYsz4M9fmdQTmbNhbbnvoypvzDYz+K8FxgI+E6sf3Hyjbkc/9ysK7 wvYUh+BPZV0K4WFUjYydHB0WC+kFtl+w5ceu7fdVtvoQrALDjTp0sDemZw2BLwWqxJtm6 Iya4IdaB2DyAKzoHOxwVYhLYOzgRk8NiXr/SuOI8x8b3RZEr4W76rW2NoR4G7SBOgyA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=CKlSGMFf; arc=none smtp.remote-ip=209.85.128.70 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F028460A2618 for ; Fri, 20 Nov 2026 01:07:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org F028460A2618 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=CKlSGMFf Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4994d67d0e3so1376405e9.2 for ; Thu, 19 Nov 2026 17:07:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1795136835; x=1795741635; darn=lists.linux.dev; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=FkHIJ0AqEEou0V4yLcGWQGEIWwBaAdh9SIA/wm+Eg+o=; b=CKlSGMFf7YgTXJ+y8RCBNKEYTBb7oEWBgw3uZTjVtrOkETXEAP+MFpeV/XiKz8noqE cd5Uvyvkn5Ows+5914BhuKn9+Z2NGpvIMu/v0uj3VFYh7LhifaFs0Sh6r4fmGTbMFqXn Mo33/NCCKwB+tKoLNq2ChipRUTtvpK3fXU9kb/3dKGxheTKKYSf6KnKdHq+JByiwdYrK G/00xTSSuSZ0RBnu45E0v1BNiXaXpkbJ9U9c/at4BmdFbrgXsI+Zwc2Bs48/GFF4n5SH b3U6ec9SFfwk7D/n7LAQSe1zpft0whZ/5IcJXDkGg9GRPzgVoIOjyAZzeip1u47isO8b /g7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795136835; x=1795741635; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=FkHIJ0AqEEou0V4yLcGWQGEIWwBaAdh9SIA/wm+Eg+o=; b=CkDzc3IstP4OGUayswPxg2FFpKHxPUQcffX9CRu9UH0DW5eX05Ct2bTMVNlTBxPShK 8SASPOMWJsvRd6WjZdxgJMda2crtHcHLylZPv2I5FiP4zBnDfOdsIaY26II18w35vP3l ZlPa9dMLNeTVezCBXYAN1Si1hGT8V+zQPrl8edU9sPDuCc/V8GcRlcm1sG53F+zP9dVH a+rR0q1zol9Gn1pbtZHXp1nqOxdY1Q2oTfk3THFdsj4QJLm1G9XWCNgNUeT7VN6/zRzD O/9GEZKdwV0Bz/AYNGjKVsZSutmJoLhkmeMAslE2OkOFgdWcy5CCqJyXB7U7rzJdA6wD 6jZw== X-Forwarded-Encrypted: i=1; AL+hErHkPfJsY7nW3+T+5GlLSsz063R/OnRHsldK5Xas0N839rkyzhj5vwARYUPXYxXf6yyFnjAHEA==@lists.linux.dev X-Gm-Message-State: APuU4Z/K57UrU4U9Zo1+QBlErs5ohEptrmN0wGx9pCiiOI2B0d8aIXXy 9CbeK+brXS9vUAP5CEfVIP+UWMtBSOR6FoZ0EKGhJU17LregkSyUWFKMe/qOFvDsHHbg6YFL8wG asYuO0A== X-Received: from wmbh22.prod.google.com ([2002:a05:600c:a116:b0:4b6:5d12:e070]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a4b:b0:4a1:dade:f049 with SMTP id 5b1f17b1804b1-4bb88806292mr26486795e9.15.1795136835025; Thu, 19 Nov 2026 17:07:15 -0800 (PST) Date: Fri, 20 Nov 2026 01:06:43 +0000 In-Reply-To: <20261120010655.467825-1-lrizzo@google.com> X-Mailing-List: iommu@lists.linux.dev List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261109182533.3594984-1-lrizzo@google.com> <20261120010655.467825-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.1.809.ga0b61035b5-goog Message-ID: <20261120010655.467825-14-lrizzo@google.com> Subject: [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device From: Luigi Rizzo To: Andrew Morton , David Hildenbrand , Johannes Weiner , Vlastimil Babka , Joerg Roedel , Will Deacon , Robin Murphy , Marek Szyprowski , Jens Axboe , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Luigi Rizzo Cc: Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , Harry Yoo , Christoph Lameter , David Rientjes , Roman Gushchin , Hao Li , Matthew Wilcox , Jan Kara , Jann Horn , Catalin Marinas , Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Simon Horman , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Alexander Lobakin , Christoph Hellwig , Jason Gunthorpe , Dragos Tatulea , Naman Gulati , Pranjal Shrivastava , Mostafa Saleh , Randy Dunlap , linux-mm@kvack.org, iommu@lists.linux.dev, netdev@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the physical DMA parent (dev->dev.parent), whereas a socket route (dst->dev) or skb->dev may point to an upper virtual net_device (bond, VLAN, ipvlan, etc.). Walking lower devices via netdev_walk_all_lower_dev_rcu() on every TX allocation would be too expensive. Add dev->pisa_tx_flags_tstamp on struct net_device to cache the leaf device PISA TX flags and refresh them at most once every 256 jiffies via pisa_tx_flags_refresh(). Also add the net_pisa_tx_bufs_key static key and helpers (use_pisa_tx_bufs(), pisa_tx_bufs_gfp()) for subsequent TX buffer patches. Signed-off-by: Luigi Rizzo --- drivers/base/core.c | 6 ++- include/linux/device.h | 11 ++++ include/linux/netdevice.h | 4 ++ include/net/pisa.h | 81 +++++++++++++++++++++++++++++ mm/pisa-kunit.c | 3 +- net/core/Makefile | 1 + net/core/pisa-net.c | 104 ++++++++++++++++++++++++++++++++++++++ 7 files changed, 207 insertions(+), 3 deletions(-) create mode 100644 include/net/pisa.h create mode 100644 net/core/pisa-net.c diff --git a/drivers/base/core.c b/drivers/base/core.c index df93b6b6e125e..5f1291e644c5e 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -2950,7 +2950,7 @@ static DEVICE_ATTR_RW(name) PISA_BOOL_ATTR(pisa_coherent); PISA_BOOL_ATTR(pisa_rx_bufs); -PISA_BOOL_ATTR(pisa_tx_bufs); +PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set); PISA_BOOL_ATTR(pisa_tx_skb_heads); PISA_BOOL_ATTR(pisa_tx_zerocopy); @@ -3111,8 +3111,10 @@ static void device_remove_attrs(struct device *dev) const struct class *class = dev->class; const struct device_type *type = dev->type; - if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) + if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) { device_remove_group(dev, &dev_attr_pisa_group); + net_pisa_tx_bufs_set(dev, false); + } if (dev->physical_location) { device_remove_group(dev, &dev_attr_physical_location_group); diff --git a/include/linux/device.h b/include/linux/device.h index e705df38e49af..0d867d8255ba5 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -817,6 +817,17 @@ struct device { DECLARE_BITMAP(flags, DEV_FLAG_COUNT); }; +#if defined(CONFIG_NET) && defined(CONFIG_PISA) +int net_pisa_tx_bufs_set(struct device *dev, bool val); +#else +/* Without CONFIG_NET the flag still gates block-device writes (see pisa_bdev_enabled()). */ +static inline int net_pisa_tx_bufs_set(struct device *dev, bool val) +{ + WRITE_ONCE(dev->pisa_tx_bufs, val); + return 0; +} +#endif + #define __create_dev_flag_accessors(accessor_name, flag_name) \ static inline bool dev_##accessor_name(const struct device *dev) \ { \ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 435be7880d7e5..f973faba4402c 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2161,6 +2161,7 @@ enum netdev_reg_state { * @neighbours: List heads pointing to this device's neighbours' * dev_list, one per address-family. * @hwprov: Tracks which PTP performs hardware packet time stamping. + * @pisa_tx_flags_tstamp: Cached leaf pisa flags and timestamp. * * FIXME: cleanup struct net_device such that network protocol info * moves out. @@ -2276,6 +2277,9 @@ struct net_device { } adj_list; /* Read-mostly cache-line for fast-path access */ +#ifdef CONFIG_PISA + u32 pisa_tx_flags_tstamp; +#endif xdp_features_t xdp_features; const struct xdp_metadata_ops *xdp_metadata_ops; const struct xsk_tx_metadata_ops *xsk_tx_metadata_ops; diff --git a/include/net/pisa.h b/include/net/pisa.h new file mode 100644 index 0000000000000..1f95e67e99bf8 --- /dev/null +++ b/include/net/pisa.h @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */ +/* PISA hooks for the networking stack. */ +#ifndef _NET_PISA_H +#define _NET_PISA_H + +#include +#include + +/* + * struct net_device caches the PISA enable flags for tx_bufs and tx_heads used + * by the leaf device, refreshed every 256 jiffies. The helpers below decide + * whether to use PISA memory for a given dst, sk, skb. + */ +#ifdef CONFIG_PISA + +enum pisa_tx_flags { + PISA_TX_FLAG_BUFS = BIT(0), + PISA_TX_XFRM_OK = BIT(2), + PISA_TX_TSTAMP_MASK = ~0xffU, +}; + +/* + * Static keys guarding networking hooks: + * - @net_pisa_tx_bufs_key: enabled while any device has pisa_tx_bufs set. + */ +DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key); + +u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now); + +/* + * May TX data routed through @dst use PISA pages? Those pages stay readable by + * every device that mapped them for as long as they are active, so only data + * that leaves the host in clear qualifies: IPsec payload before encryption has + * dst_xfrm(dst) != NULL and is rejected unless PISA_TX_XFRM_OK is set. + */ +static inline gfp_t __pisa_tx_gfp(const struct dst_entry *dst, + enum pisa_tx_flags flag) +{ + const struct net_device *dev; + u32 stamp, now; + + guard(rcu)(); + if (!dst || (dst_xfrm(dst) && !(flag & PISA_TX_XFRM_OK))) + return 0; + dev = dst_dev_rcu(dst); + if (!dev) + return 0; + stamp = READ_ONCE(dev->pisa_tx_flags_tstamp); + now = (u32)jiffies & PISA_TX_TSTAMP_MASK; + if (unlikely((stamp & PISA_TX_TSTAMP_MASK) != now)) + stamp = pisa_tx_flags_refresh((struct net_device *)dev, now); + return (stamp & flag) ? __GFP_PISA : 0; +} + +/* + * ============================================================================ + * Networking hooks (every hook starts inline with a static-key check) + * ============================================================================ + */ + +/* Hook for route-based TX payload fragment selection (@net_pisa_tx_bufs_key). */ +static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) +{ + return static_branch_unlikely(&net_pisa_tx_bufs_key) && + __pisa_tx_gfp(dst, PISA_TX_FLAG_BUFS); +} + +/* Hook for ESP output skb_page_frag_refill() (@net_pisa_tx_bufs_key). */ +static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) +{ + return static_branch_unlikely(&net_pisa_tx_bufs_key) ? + __pisa_tx_gfp(skb_dst(skb), PISA_TX_FLAG_BUFS | PISA_TX_XFRM_OK) : 0; +} + +#else /* !CONFIG_PISA */ + +static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) { return false; } +static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) { return 0; } + +#endif /* CONFIG_PISA */ +#endif /* _NET_PISA_H */ diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c index 4effbc8610079..d7670c76e98c5 100644 --- a/mm/pisa-kunit.c +++ b/mm/pisa-kunit.c @@ -578,7 +578,8 @@ static void test_dev_pisa_attrs(struct kunit *test) kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent); kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs); kunit_check_pisa_attr(test, dev, "pisa_tx_zerocopy", &dev->pisa_tx_zerocopy); - kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs); + if (IS_ENABLED(CONFIG_NET)) + kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs); kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads); device_unregister(dev); } diff --git a/net/core/Makefile b/net/core/Makefile index c20e520358b8e..3c84358d6b0c0 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -52,3 +52,4 @@ obj-$(CONFIG_NET_TEST) += net_test.o obj-$(CONFIG_NET_DEVMEM) += devmem.o obj-$(CONFIG_DEBUG_NET) += lock_debug.o obj-$(CONFIG_FAIL_SKB_REALLOC) += skb_fault_injection.o +obj-$(CONFIG_PISA) += pisa-net.o diff --git a/net/core/pisa-net.c b/net/core/pisa-net.c new file mode 100644 index 0000000000000..6d4909c82070c --- /dev/null +++ b/net/core/pisa-net.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause +/* + * PISA networking helpers: leaf device TX flag cache, small skb head slab + * cache, and MSG_ZEROCOPY user page upgrade. + * See Documentation/core-api/pisa.rst for the architecture overview. + */ + +#include +#include +#include +#include +#include +#include + +/* Enabled while at least one device has pisa_tx_bufs set. */ +DEFINE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key); +EXPORT_SYMBOL(net_pisa_tx_bufs_key); + +int net_pisa_tx_bufs_set(struct device *dev, bool val) +{ + static DEFINE_MUTEX(lock); + + guard(mutex)(&lock); + if (dev->pisa_tx_bufs != val) { + WRITE_ONCE(dev->pisa_tx_bufs, val); + if (val) + static_branch_inc(&net_pisa_tx_bufs_key); + else + static_branch_dec(&net_pisa_tx_bufs_key); + } + return 0; +} + +/* + * The sysfs attribute pisa_tx_bufs lives on the device that does DMA: the + * closest ancestor of a leaf net_device that has a dma_mask (dev->dev.parent, + * or further up behind bus glue such as virtio). A socket route (dst->dev) may + * instead point to an upper virtual net_device (bond, VLAN, ipvlan, etc.), + * whose leaves are its lower devices. Walking lower devices via + * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too + * expensive, so dev->pisa_tx_flags_tstamp caches the walk result on the net_device + * and refreshes it at most once every 256 jiffies: + * bit 0 (PISA_TX_FLAG_BUFS): pisa_tx_bufs enabled on lower device(s) + * bits 31..8 (PISA_TX_TSTAMP_MASK): upper 24 bits of (u32)jiffies + */ +static u32 pisa_tx_flags(const struct net_device *dev) +{ + const struct device *parent = dev->dev.parent; + u32 flags = 0; + + while (parent && !parent->dma_mask) + parent = parent->parent; + if (parent && READ_ONCE(parent->pisa_tx_bufs)) + flags |= PISA_TX_FLAG_BUFS; + return flags; +} + +/* MACsec encrypts in software above its lower device: its cleartext must stay out of PISA. */ +static bool pisa_tx_dev_excluded(const struct net_device *dev) +{ + return netif_is_macsec(dev); +} + +/* A leaf has no lower devices; what its parent device is does not matter. */ +static bool pisa_tx_dev_is_leaf(const struct net_device *dev) +{ + return list_empty(&dev->adj_list.lower); +} + +/* Every leaf must have opted in; a leaf without a DMA device (veth, tun, ...) has not. */ +static int pisa_tx_flags_walk(struct net_device *dev, struct netdev_nested_priv *priv) +{ + uintptr_t flags = (uintptr_t)priv->data; + + if (pisa_tx_dev_excluded(dev)) + flags = BIT(2); + else if (pisa_tx_dev_is_leaf(dev)) + flags = (flags & pisa_tx_flags(dev)) | BIT(2); + priv->data = (void *)flags; + return 0; +} + +/* Slow path of __pisa_tx_gfp(): refresh dev->pisa_tx_flags_tstamp. */ +u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now) +{ + if (pisa_tx_dev_excluded(dev)) { + /* No flags. */ + } else if (pisa_tx_dev_is_leaf(dev)) { + now |= pisa_tx_flags(dev); + } else { + struct netdev_nested_priv priv = { + .data = (void *)(uintptr_t)PISA_TX_FLAG_BUFS, + }; + + rcu_read_lock(); + netdev_walk_all_lower_dev_rcu(dev, pisa_tx_flags_walk, &priv); + rcu_read_unlock(); + if ((uintptr_t)priv.data & BIT(2)) + now |= (uintptr_t)priv.data & ~BIT(2); + } + WRITE_ONCE(dev->pisa_tx_flags_tstamp, now); + return now; +} +EXPORT_SYMBOL(pisa_tx_flags_refresh); -- 2.56.1.809.ga0b61035b5-goog