From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=209.85.208.69 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1795136832; b=4TddxLMOU5Qjnmmw/msc3sj2NW6ODlBV7jnuOA49ag24/e8Aw3Yj3XE/6Z35fQQ0H2JF WToc1u9se9bGS3A67nTqAXwJ7zydJtJxe/vJu6EFc7jxWcD3SlU8Sbf/twfAOOAFMwExM gh0ki/kZTz5/prk7kInbEmwMvy9JpCTBlH2o3axiCmJFSPtEWO5H8y/GSVDWoCft/Gcvz IbCrYeh3A2dNuYVXCe/dlBesWOIBFN8o4D3cyDOZy31z8FrzCXNAR5nJnpMO3mD3o+MKk I5ldHTF5QnbM56IhjKU+515FLcldnUr8QnPGN5GmDifcTbGzoObjnJqzsgr5UM65t/A== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1795136832; h=DMARC-Filter:Received:DKIM-Signature:X-Google-DKIM-Signature: X-Forwarded-Encrypted:X-Gm-Message-State:X-Received:Date:In-Reply-To: Mime-Version:References:X-Mailer:Message-ID:Subject:From:To:Cc: Content-Type; bh=Ut9IgsodJz9rQhj0smx0VJpNs6AreC5f8aEMQj1btrk=; b=G3HAY6v2MJhy1jSCyy3yAYvaneZVDvefJILHBxZSiARrngUUX6AkPaJZd32jxkhIT84M fdZs03vPbhbXixgvUHNR0OcuvqRcAQukg16p7xMLO0FnIp40iYoBEGnIUFo3uxidKKt+W cM9j4p5/pBh83zx6yeuyFGe+OnHMbzlVjYfd7+8tS65tp+9qUwgvyy0fWE2SG0xuXzkdA 82Xbeb7o6JRA8LiGa/Ett7gdE9X8waLgWjsPlMUp881OZx9UWsseZCXRsxAQP6D7sQBp9 jF3RFah2RgErQFULkTFhV2nauiD3lrAmZEjIAFVDlipNvnVg00Px73rTnDdqawZcB+w== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=google.com; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com; dkim=pass header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=TwURa7PW; arc=none smtp.remote-ip=209.85.208.69 Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E257960A260C for ; Fri, 20 Nov 2026 01:07:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org E257960A260C Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--lrizzo.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20260707 header.b=TwURa7PW Received: by mail-ed1-f69.google.com with SMTP id 4fb4d7f45d1cf-6ba13a9d071so331275a12.3 for ; Thu, 19 Nov 2026 17:07:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20260707; t=1795136829; x=1795741629; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Ut9IgsodJz9rQhj0smx0VJpNs6AreC5f8aEMQj1btrk=; b=TwURa7PWnvIOhsESRKppZ7Lf7w/KPrYo7VR8G5ymN2clf1kQhjk8heaaeZQEgGWx/O /wwUgWgJaFrxASFIT1AQt492bqiwWplgloc2NAXV064BT227vEBEgEho9RQiHzS4eicE uKRCyaqUv07B0CYp29QD4Y+pDkIk8059W91n9a8L3wYF3Q1djro2H58qNj2pmjCNV1RG bdYbLCoEUx1fo0T4HS//CEV0nQJ/HRjripIe3MZf9b4JZLKFHXuezaFOnTYU4nHZR33i 8Fx2C3leNagBZVWKP6evZfdWsFwwBZHDQJUpwkdr5OpWhTLHgiO5ffKh7KJl5VYglZXf je6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1795136829; x=1795741629; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Ut9IgsodJz9rQhj0smx0VJpNs6AreC5f8aEMQj1btrk=; b=uCB2lsdjPsDHV5d62lengAwFXD1bHpQj83Q3AFrIKu8U8cw6J9k1K9Pr+7Np1Mqblr FA8tte3s1YyGFvoytMfKLURpUxrWAOKNAm1kkymhA4VwoC/BedyS6KYf1FYF4w9cUU+c 6TKb9CUFFa2OeRvi4eCKmN0TavXB9fCf3vPkNWCJs3vWBjGgcoLmUc8go0Er+8w0wEcP GDruu1u1/X/tDiWcG8hgBFDptr+fqXIEx+FCkKqZaAkXYqnlZJdPyF4FlNLF3fM0k93W E2kCXHBEYq/t+68uhvRHd0l78N2Zw0N0pyHfma0N1Q4pMkzWyBQ3LjqAvr/igJDA9MuW vHOA== X-Forwarded-Encrypted: i=1; AL+hErEetyABNcFVUbJLn7NLRJdRiRqAeXi2vfMcwZQh2Pw2X0bOhDGaJ370gksyYg08N3vBlanEsst485Zuuw==@vger.kernel.org X-Gm-Message-State: APuU4Z9O6gOJNQHG/hWU9diBFW01l7wlAn8JFKLAcOfAi7yNwrf826Wq GWGCDZQ1pSLnEQtC4/04ojEL+tEW3AYvqWkLb7yRGN+FSFjdHPLeDLR4IrrYcifHOHlH4/N0Y+c RF1rRdw== X-Received: from edrv5.prod.google.com ([2002:aa7:d645:0:b0:6b8:8291:9a7]) (user=lrizzo job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:c4c6:20b0:6b7:8cfe:59b with SMTP id 4fb4d7f45d1cf-6ba37e7a050mr596195a12.15.1795136828958; Thu, 19 Nov 2026 17:07:08 -0800 (PST) Date: Fri, 20 Nov 2026 01:06:37 +0000 In-Reply-To: <20261120010655.467825-1-lrizzo@google.com> X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 References: <20261109182533.3594984-1-lrizzo@google.com> <20261120010655.467825-1-lrizzo@google.com> X-Mailer: git-send-email 2.56.1.809.ga0b61035b5-goog Message-ID: <20261120010655.467825-8-lrizzo@google.com> Subject: [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin From: Luigi Rizzo To: Andrew Morton , David Hildenbrand , Johannes Weiner , Vlastimil Babka , Joerg Roedel , Will Deacon , Robin Murphy , Marek Szyprowski , Jens Axboe , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Luigi Rizzo Cc: Lorenzo Stoakes , "Liam R . Howlett" , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Brendan Jackman , Zi Yan , Harry Yoo , Christoph Lameter , David Rientjes , Roman Gushchin , Hao Li , Matthew Wilcox , Jan Kara , Jann Horn , Catalin Marinas , Greg Kroah-Hartman , "Rafael J . Wysocki" , Danilo Krummrich , Jonathan Corbet , Simon Horman , Jesper Dangaard Brouer , Ilias Apalodimas , Willem de Bruijn , Kuniyuki Iwashima , Joshua Washington , Harshitha Ramamurthy , Alexander Lobakin , Christoph Hellwig , Jason Gunthorpe , Dragos Tatulea , Naman Gulati , Pranjal Shrivastava , Mostafa Saleh , Randy Dunlap , linux-mm@kvack.org, iommu@lists.linux.dev, netdev@vger.kernel.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-api@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Luigi Rizzo Content-Type: text/plain; charset="UTF-8" Because of the performance and security implications, the use of PISA features is opt-in and configurable per device with a fine granularity. Add boolean attributes to struct device exposed under /sys/devices/*/*/pisa_*: - pisa_coherent: enable PISA for coherent allocations - pisa_rx_bufs: enable PISA for RX buffers / page_pool - pisa_tx_bufs: enable PISA for TX socket page frags - pisa_tx_skb_heads: enable PISA slab cache for small TX skb heads - pisa_tx_zerocopy: migrate MSG_ZEROCOPY user pages into PISA Signed-off-by: Luigi Rizzo --- drivers/base/core.c | 75 ++++++++++++++++++++++++++++++++++++++++++ include/linux/device.h | 15 +++++++++ include/linux/pisa.h | 27 +++++++++++++++ mm/pisa-alloc.c | 21 ++++++++++++ mm/pisa-kunit.c | 71 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 209 insertions(+) diff --git a/drivers/base/core.c b/drivers/base/core.c index cbe05197d4a0b..df93b6b6e125e 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -2907,6 +2908,65 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute } static const DEVICE_ATTR_RO(removable); +/* Per-device configuration to selectively enable PISA */ +static ssize_t pisa_attr_store(struct device *dev, const char *buf, size_t count, + bool *field, int (*setter)(struct device *, bool)) +{ + bool val; + int ret = kstrtobool(buf, &val); + + if (ret < 0) + return ret; + if (val) { + ret = pisa_meta_init(); + if (ret < 0) + return ret; + } + if (setter) { + ret = setter(dev, val); + if (ret < 0) + return ret; + } else { + WRITE_ONCE(*field, val); + } + return count; +} + +#define PISA_SET_ATTR(name, setter) \ +static ssize_t name##_show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + return sysfs_emit(buf, "%u\n", READ_ONCE(dev->name)); \ +} \ +static ssize_t name##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t count) \ +{ \ + return pisa_attr_store(dev, buf, count, &dev->name, setter); \ +} \ +static DEVICE_ATTR_RW(name) + +#define PISA_BOOL_ATTR(name) PISA_SET_ATTR(name, NULL) + +PISA_BOOL_ATTR(pisa_coherent); +PISA_BOOL_ATTR(pisa_rx_bufs); +PISA_BOOL_ATTR(pisa_tx_bufs); +PISA_BOOL_ATTR(pisa_tx_skb_heads); +PISA_BOOL_ATTR(pisa_tx_zerocopy); + +static struct attribute *dev_attr_pisa[] = { + &dev_attr_pisa_coherent.attr, + &dev_attr_pisa_rx_bufs.attr, + &dev_attr_pisa_tx_bufs.attr, + &dev_attr_pisa_tx_skb_heads.attr, + &dev_attr_pisa_tx_zerocopy.attr, + NULL, +}; + +static const struct attribute_group dev_attr_pisa_group = { + .attrs = dev_attr_pisa, +}; + int device_add_groups(struct device *dev, const struct attribute_group *const *groups) { @@ -3014,6 +3074,18 @@ static int device_add_attrs(struct device *dev) goto err_remove_dev_removable; } + if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) { + error = device_add_group(dev, &dev_attr_pisa_group); + if (error) { + if (dev->physical_location) { + device_remove_group(dev, &dev_attr_physical_location_group); + kfree(dev->physical_location); + dev->physical_location = NULL; + } + goto err_remove_dev_removable; + } + } + return 0; err_remove_dev_removable: @@ -3039,6 +3111,9 @@ static void device_remove_attrs(struct device *dev) const struct class *class = dev->class; const struct device_type *type = dev->type; + if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) + device_remove_group(dev, &dev_attr_pisa_group); + if (dev->physical_location) { device_remove_group(dev, &dev_attr_physical_location_group); kfree(dev->physical_location); diff --git a/include/linux/device.h b/include/linux/device.h index 429e39f647cf9..e705df38e49af 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -695,6 +695,15 @@ enum struct_device_flags { * @removable: Whether the device can be removed from the system. This * should be set by the subsystem / bus driver that discovered * the device. + * @pisa_coherent: Back coherent dma_alloc_attrs() with PISA + * (sysfs pisa_coherent). + * @pisa_rx_bufs: Use PISA allocators for RX buffers (sysfs pisa_rx_bufs). + * @pisa_tx_bufs: Use PISA allocators for socket TX page fragments + * (sysfs pisa_tx_bufs). + * @pisa_tx_skb_heads: Use PISA slab cache for small TX skb heads + * (sysfs pisa_tx_skb_heads). + * @pisa_tx_zerocopy: Migrate MSG_ZEROCOPY user pages into PISA + * (sysfs pisa_tx_zerocopy). * @flags: DEV_FLAG_XXX flags. Use atomic bitfield operations to modify. * * At the lowest level, every device in a Linux system is represented by an @@ -799,6 +808,12 @@ struct device { enum device_removable removable; + bool pisa_coherent; + bool pisa_rx_bufs; + bool pisa_tx_bufs; + bool pisa_tx_skb_heads; + bool pisa_tx_zerocopy; + DECLARE_BITMAP(flags, DEV_FLAG_COUNT); }; diff --git a/include/linux/pisa.h b/include/linux/pisa.h index cc33e181df097..5222cb207257f 100644 --- a/include/linux/pisa.h +++ b/include/linux/pisa.h @@ -84,6 +84,7 @@ static inline u64 pisa_iova_size(void) } int pisa_meta_init(void); +bool __pisa_has_free_blocks_rx(unsigned int order, int nid); bool __pisa_page_can_recycle(const struct page *page); static inline bool pisa_enabled(void) @@ -142,6 +143,21 @@ static inline struct page *pisa_virt_to_page(const void *addr) return is_pisa_page(page) ? page : NULL; } +/* Hook for __page_pool_alloc_netmems_slow() (guarded by @pisa_key). */ +static inline gfp_t pisa_rx_gfp(const struct device *dev) +{ + return pisa_enabled() && dev && READ_ONCE(dev->pisa_rx_bufs) ? + __GFP_PISA_RW : 0; +} + +/* Hook for NIC driver RX buffer allocation (guarded by @pisa_key). */ +static inline bool pisa_has_free_blocks_rx(const struct device *dev, + unsigned int order, int nid) +{ + return pisa_enabled() && dev && READ_ONCE(dev->pisa_rx_bufs) && + __pisa_has_free_blocks_rx(order, nid); +} + /* Hook for skb_page_frag_refill() and __pisa_free_page() (@pisa_recycle_check_key). */ static inline bool pisa_page_can_recycle(const struct page *page) { @@ -158,6 +174,17 @@ static inline bool is_pisa_page(const struct page *page) { return false; } static inline bool is_pisa_phys(phys_addr_t phys) { return false; } static inline struct page *pisa_virt_to_page(const void *addr) { return NULL; } +static inline gfp_t pisa_rx_gfp(const struct device *dev) +{ + return 0; +} + +static inline bool pisa_has_free_blocks_rx(const struct device *dev, + unsigned int order, int nid) +{ + return false; +} + static inline bool pisa_page_can_recycle(const struct page *page) { return true; diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c index 01d8bbc8d72ff..ff8565c3206ad 100644 --- a/mm/pisa-alloc.c +++ b/mm/pisa-alloc.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -374,6 +375,26 @@ unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen); } +/** + * __pisa_has_free_blocks_rx - Check if per-node RX allocator of @order on @nid has free blocks + * @order: Block order + * @nid: Target NUMA node (or NUMA_NO_NODE) + * + * Return: true if the per-node RX allocator has a free block available, or + * the node's reservoir has a PISA page to give it. + */ +bool __pisa_has_free_blocks_rx(unsigned int order, int nid) +{ + struct pisa_alloc *pa = pisa_node_pa(nid, order, false); + + return pa && + (!list_empty_careful(&pa->partial) || + !list_empty_careful(&pa->partial_dirty) || + !list_empty_careful(&pa->idle) || + READ_ONCE(pisa_nodes[pa->nid].nr_pages)); +} +EXPORT_SYMBOL(__pisa_has_free_blocks_rx); + /* * Split a non-compound page of @old_order into 1 << (@old_order - @new_order) * pages of @new_order (compound if @new_order > 0), all frozen at refcount 0. diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c index eb51007e66de5..2adaeb1ffe43e 100644 --- a/mm/pisa-kunit.c +++ b/mm/pisa-kunit.c @@ -6,6 +6,8 @@ #include #include #include +#include +#include #include #include #include @@ -443,6 +445,74 @@ static void test_map_and_domain_lifecycle(struct kunit *test) } #endif /* CONFIG_IOMMU_DMA */ +static void kunit_check_pisa_attr(struct kunit *test, struct device *dev, + const char *name, bool *field) +{ + const struct sysfs_ops *ops = dev->kobj.ktype->sysfs_ops; + struct kernfs_node *kn; + char *buf; + + buf = (char *)__get_free_page(GFP_KERNEL); + KUNIT_EXPECT_NOT_NULL(test, buf); + if (!buf) + return; + kn = kernfs_find_and_get(dev->kobj.sd, name); + KUNIT_EXPECT_NOT_NULL(test, kn); + if (!kn) + goto out_free; + + KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2); + KUNIT_EXPECT_STREQ(test, buf, "0\n"); + KUNIT_EXPECT_FALSE(test, READ_ONCE(*field)); + + KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "1\n", 2), 2); + KUNIT_EXPECT_TRUE(test, READ_ONCE(*field)); + KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2); + KUNIT_EXPECT_STREQ(test, buf, "1\n"); + + KUNIT_EXPECT_LT(test, ops->store(&dev->kobj, kn->priv, "bad", 3), 0); + KUNIT_EXPECT_TRUE(test, READ_ONCE(*field)); + + KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "0\n", 2), 2); + KUNIT_EXPECT_FALSE(test, READ_ONCE(*field)); + + kernfs_put(kn); +out_free: + free_page((unsigned long)buf); +} + +static void kunit_attr_dev_release(struct device *dev) +{ + kfree(dev); +} + +static void test_dev_pisa_attrs(struct kunit *test) +{ + struct device *dev; + + if (!IS_ENABLED(CONFIG_SYSFS)) + kunit_skip(test, "requires CONFIG_SYSFS"); + + dev = kzalloc(sizeof(*dev), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, dev); + dev->init_name = "pisa_attr_kunit"; + dev->release = kunit_attr_dev_release; + dev->coherent_dma_mask = DMA_BIT_MASK(64); + dev->dma_mask = &dev->coherent_dma_mask; + + if (device_register(dev)) { + put_device(dev); + KUNIT_FAIL(test, "device_register failed"); + return; + } + kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent); + kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs); + kunit_check_pisa_attr(test, dev, "pisa_tx_zerocopy", &dev->pisa_tx_zerocopy); + kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs); + kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads); + device_unregister(dev); +} + static void test_reservoir_and_scrubber(struct kunit *test) { KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0); @@ -459,6 +529,7 @@ static struct kunit_case pisa_meta_test_cases[] = { KUNIT_CASE(test_iova_helpers), KUNIT_CASE(test_map_and_domain_lifecycle), #endif + KUNIT_CASE(test_dev_pisa_attrs), KUNIT_CASE(test_reservoir_and_scrubber), {} }; -- 2.56.1.809.ga0b61035b5-goog