* [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
[not found] <20261003212241.3432303-1-lrizzo@google.com>
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
` (24 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
This is v2 of the RFC previously posted as DMA_PMD [0] (renamed to avoid
confusion with CPU page table PMD levels).
PISA stands for "Pre-mapped IOVA Superpage Allocator" and as the name
says, it is a specialized memory allocator. Invoked via __GFP_PISA* flag
to alloc_pages(), it returns pages ("PISA memory") that make DMA-related
operations on that memory a lot more efficient, both in software and
in hardware.
Specifically, PISA memory has the following properties:
- it is backed by 2MB contiguous physical pages, so the IOMMU can
use large leaf mappings and significantly reduce IOTLB pressure see [1]
- IOMMU mappings are recycled while PISA memory is used for IO buffers,
thus greatly reducing IOVA allocation and dma map/unmap costs.
This includes cases (e.g. overcommitted VMs or Confidential Compute)
where they would trigger expensive VM exits or bounce buffer copies.
- PISA memory is returned to the buddy allocator in background, and
only after all existing IOMMU and IOTLB mappings have been flushed.
This removes a significant security problem in DMA-FQ, which instead returns
pages before flushing the IOTLB, without the performance penalty of
strict iommu (DMA) where the flushes are synchronous.
The pitch line remains:
PISA has the performance of identity mapping, guarantees that only active
I/O buffers have IOMMU/IOTLB mappings (unlike DMA-FQ), removes bounce-buffer
copies in Confidential Computing VMs, and integrates transparently with
existing kernel subsystems and drivers.
=== KEY IDEA: A SPECIALIZED MEMORY ALLOCATOR
For avoidance of doubt, PISA is NOT a new buffer cache, page_pool
replacement, slab allocator, or custom netdev feature. The kernel already
has plenty of specialized caches and pools (page_pool, socket page_frags,
SLUB, page cache) built on top of the buddy allocator.
Instead, PISA sits *below* those existing subsystems as a specialized
memory provider modeled after alloc_pages() and __free_pages_prepare().
It enforces a few properties on the memory it dispenses so that IOMMU
mapping, unmapping, and CoCo page decryption become nearly free:
1. PISA allocates order-N blocks out of physically contiguous 2MB pages that
are mapped once per IOMMU domain using 2MB IOMMU leaf entries in a
reserved per-domain IOVA window (and decrypted once upfront in CoCo
guests). Subsequent dma_map_*() calls resolve the IOVA with a simple
window offset calculation, and dma_unmap_*() is a no-op.
2. Freed blocks are intercepted in __free_pages_prepare() and recycled
back into their PISA allocator while their 2MB IOMMU mappings remain
active. A 2MB page is returned to the buddy allocator only when all
of its blocks are idle, its IOMMU mappings have been removed from all
domains, a synchronous IOTLB flush has completed, and an RCU grace
period has elapsed.
3. Callers allocate from per-NUMA-node PISA allocators using standard page
allocator calls with __GFP_PISA_RW (DMA_BIDIRECTIONAL) or __GFP_PISA_RO
(DMA_TO_DEVICE, mapped IOMMU_READ-only so devices cannot overwrite
shared/adjacent data), falling back to the buddy allocator on failure.
Because PISA operates at the page allocator and DMA mapping boundary,
existing clients require minimal changes. For example, the entire patch
integrating page_pool with PISA (patch 13/25) is just 3 lines:
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
@@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(...)
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
Similarly:
- Coherent rings/queues (dma_alloc_attrs()) are served from the per-node
order-0 PISA allocator when dev->pisa_coherent is enabled; no driver
changes needed.
- Socket TX page fragments (skb_page_frag_refill()) use a per-task
page_frag fed by __GFP_PISA_RO when the egress device has pisa_tx_bufs
enabled; no driver changes needed.
- Small TX skb heads (kmalloc_reserve()) use a parallel
skbuff_small_head_pisa SLUB cache backed by zeroed, read-only PISA
folios (SLAB_CACHE_PISA_RO) when pisa_tx_skb_heads is enabled. This
eliminates the per-driver TX header bounce-buffer patches from v1.
- TX MSG_ZEROCOPY, O_DIRECT, and buffered disk I/O migrates or allocates
order-0 user and page-cache folios from per-node PISA allocators so
zero-copy network sends and storage I/O also use 2MB IOMMU mappings.
All features are strictly opt-in per device via
/sys/devices/.../pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads} (and
pisa.user_max_4K_pages for user/page-cache folios). Until a device opts in,
PISA is gated off by a static key and allocates no metadata.
=== CHANGES SINCE v1 [0]
I have make use and applied several comments and suggestions received on
the previous version. The changes are massive so perhaps it makes sense
to look at the new code in isolation, unless you have already revised
in detail the previous version.
A high level summary is the following:
- Renamed DMA_PMD to PISA (CONFIG_PISA, include/linux/pisa.h,
drivers/iommu/pisa-*.c, /sys/devices/.../pisa_*).
- Hooked __GFP_PISA_RO and __GFP_PISA_RW into __alloc_frozen_pages_noprof()
and alloc_pages_bulk_noprof(), replacing custom pool calls in page_pool
and socket TX refill.
- Added lockless per-CPU block caches (pisa_enable_pcp()) in front of the
per-node order-0 RW and SKB_FRAG_PAGE_ORDER RO allocators.
- Added SLAB_CACHE_PISA_RO in SLUB and skbuff_small_head_pisa for small
TX skb heads, dropping the per-driver TX header bounce-buffer patches.
- Added PISA support for MSG_ZEROCOPY user pages (madvise(MADV_PISA) and
automatic upgrade on send) and block/page-cache I/O (O_DIRECT and
buffered reads/writes), plus age/waste-based compaction and scrub
migration (pisa.wasted_pct, pisa.old_ms, pisa.scrub_migrate).
- Made metadata table initialization and static key activation lazy on
first device opt-in.
=== PERFORMANCE BENEFITS
Your mileage may vary. Enabling the IOMMU may have no throughput impact
until system components (bus, IOTLB, CPU) become overloaded. Aside from
throughput reduction, one key metric is IOTLB effectiveness. Below are SMMU
performance counters on a large ARM64 system with 2x200G NICs running
bidirectional traffic:
=== DMA-FQ MODE (total throughput ~440Gbps)
25,757,488 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
16,634,941 smmuv3_pmcg_*/event=0x81/ IOTLB misses
=== PISA on top of strict DMA (total throughput ~745Gbps)
22,693,218 smmuv3_pmcg_*/event=0x80/ IOTLB lookups
30,536 smmuv3_pmcg_*/event=0x81/ IOTLB misses
The 2MB mappings make IOTLB misses almost non-existent because the working
set of IOTLB entries is reduced by a factor of ~512.
=== NOTES
[0] v1 (DMA_PMD):
https://lore.kernel.org/all/20261003212241.3432303-1-lrizzo@google.com/
[1] IOMMU IMPACT AND IOTLB THRASHING
Documented in detail in Documentation/core-api/pisa.rst. In short, there
are three main costs with the IOMMU:
- CPU cost for dma_map/unmap
- CPU cost and latency for synchronous IOTLB flush (required for security)
- IOTLB thrashing when the working set of 4KB pages exceeds IOTLB capacity,
causing IOMMU page walks that backpressure PCIe/bus traffic (30..60%
throughput drop on high-speed NICs). See also:
https://lore.kernel.org/all/4b42f2eb-dc29-153e-ace9-5584ea2e5070@redhat.com/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10280580/
[2] WHY NOT USE PAGE_POOL FOR EVERYTHING
page_pool is a consumer of PISA (via __GFP_PISA_RW), not a replacement:
- page_pool does not split 2MB pages into order-N blocks or manage shared
2MB IOMMU mappings across domains;
- page_pool pages cannot be managed by arbitrary get_page()/put_page()
callers (socket TX fragments, SLUB slabs, user/page-cache folios);
- page_pool requires a single NAPI consumer with BH disabled and a known
device at allocation time, whereas socket TX, SLUB, and dma_alloc_attrs()
run in process/preemptible context across arbitrary devices.
Luigi Rizzo (25):
iommu/dma: Introduce CONFIG_PISA and metadata table
iommu/dma: Add PISA allocator lifecycle and page recycle hook
iommu/dma: Add per-NUMA-node PISA page reservoir
iommu/dma: Add PISA block allocation
iommu/dma: Add global cap and shrinker for PISA pages
iommu/dma: Reserve a per-domain IOVA window for PISA pages
iommu/dma: Release PISA domain mappings on domain teardown
iommu/dma: Use per-domain IOVA window to map PISA memory
driver core: Add per-device pisa_* sysfs attributes for PISA optin
iommu/dma: Use PISA for coherent DMA allocations
iommu/dma: Add per-CPU block caches to PISA allocators
net/core: Use per-node PISA allocators for skb_page_frag_refill()
net/core: Use PISA for page_pool memory
iommu/dma: Support decrypted PISA pages
iommu/dma: Add background page scrubber for PISA allocators
mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
net: Allocate small skb heads from PISA
iommu/dma: Migrate user pages into PISA for TX zerocopy
iommu/dma: Stop recycling old PISA pages when allocator waste is high
net/libeth: Use GFP_KERNEL when pre-filling RX rings
net/gve: Use GFP_KERNEL when pre-filling RX rings
net/gve: Use PISA memory for RX buffers
block, mm, iommu/dma: Support PISA for disk I/O and scrub migration
Documentation: Add PISA architecture documentation
iommu/dma: Add PISA statistics and debugfs
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/pisa.rst | 426 ++++
block/bio.c | 46 +
drivers/base/core.c | 78 +
drivers/iommu/Kconfig | 34 +
drivers/iommu/Makefile | 5 +
drivers/iommu/dma-iommu.c | 48 +-
drivers/iommu/dma-iommu.h | 16 +
drivers/iommu/iommu.c | 10 +-
drivers/iommu/pisa-alloc.c | 2063 +++++++++++++++++
drivers/iommu/pisa-kunit.c | 669 ++++++
drivers/iommu/pisa-map.c | 553 +++++
drivers/iommu/pisa-meta.c | 300 +++
drivers/iommu/pisa-node.c | 378 +++
drivers/iommu/pisa-priv.h | 511 ++++
drivers/iommu/pisa-user.c | 662 ++++++
.../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 7 +-
drivers/net/ethernet/google/gve/gve_rx.c | 40 +-
include/linux/device.h | 27 +
include/linux/gfp_types.h | 17 +-
include/linux/netdevice.h | 4 +
include/linux/pisa.h | 367 +++
include/linux/sched.h | 4 +
include/linux/skbuff.h | 2 +
include/linux/slab.h | 10 +
include/linux/swap.h | 1 +
include/net/libeth/rx.h | 11 +-
include/net/sock.h | 143 ++
include/trace/events/mmflags.h | 11 +-
include/uapi/asm-generic/mman-common.h | 2 +
kernel/dma/direct.c | 14 +-
kernel/exit.c | 2 +
kernel/fork.c | 2 +
mm/filemap.c | 15 +-
mm/madvise.c | 11 +
mm/page-writeback.c | 8 +
mm/page_alloc.c | 59 +
mm/readahead.c | 5 +-
mm/slab.h | 3 +-
mm/slab_common.c | 5 +-
mm/slub.c | 14 +-
net/core/datagram.c | 85 +-
net/core/page_pool.c | 3 +
net/core/skbuff.c | 42 +-
net/core/sock.c | 62 +-
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 19 +-
net/ipv4/tcp.c | 17 +-
net/ipv4/tcp_output.c | 12 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 19 +-
52 files changed, 6784 insertions(+), 73 deletions(-)
create mode 100644 Documentation/core-api/pisa.rst
create mode 100644 drivers/iommu/pisa-alloc.c
create mode 100644 drivers/iommu/pisa-kunit.c
create mode 100644 drivers/iommu/pisa-map.c
create mode 100644 drivers/iommu/pisa-meta.c
create mode 100644 drivers/iommu/pisa-node.c
create mode 100644 drivers/iommu/pisa-priv.h
create mode 100644 drivers/iommu/pisa-user.c
create mode 100644 include/linux/pisa.h
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply [flat|nested] 26+ messages in thread* [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table
[not found] <20261003212241.3432303-1-lrizzo@google.com>
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
` (23 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA ("Pre-mapped IOVA Superpage Allocator") is a memory allocator
that provides devices which opt in with DMA-able memory backed by large
physically contiguous pages (typically 2MB, but fully configurable),
and mapped by IOMMU leaves of the same size.
PISA has the following features:
- greatly reduces IOTLB pressure by using fewer, much larger IOMMU
entries.
- greatly reduces DMA map/unmap cost. Mapping only happens when the
first block from a 2MB page is used. Unmapping is a no-op for the
caller, and happens in a separate task when no blocks from the 2MB
page are in use and the page should be returned to the buddy
allocator.
- much better security than DMA-FQ, because buffers are not returned to
the buddy allocator until the mappings have been removed and the IOTLB
flushed.
- removes the cost of copying into bounce buffers, e.g. for
overcommitted or Confidential Computing VMs.
- fine-grained opt-in selection (each device can opt in selectively for
descriptors, transmit and/or receive data buffers and headers) permits
choosing the correct security and performance tradeoffs.
- default is to use the base IOMMU mode (DMA or DMA-FQ) of the device.
PISA is integrated with the main IO APIs (dma_alloc_attrs,
dma_map/unmap, page_pool, TX socket buffers and disk IO buffers) so that
no device driver modifications should be necessary.
Add a CONFIG_PISA option, enabled only on supported kernels (64bit,
!PREEMPT_RT) and introduce a sparse metadata table to hold per-2MB buffer
state (similar to pageblock_flags).
The metadata table is initialized on first use, consumes 4KB of RAM
and 2MB of virtual address space for each 16GB of physical RAM.
Also add a built-in KUnit test suite (CONFIG_PISA_META_KUNIT_TEST)
exercising initialization idempotency, PFN/physical round-trip lookups,
out-of-bounds physical address handling, and PISA membership toggling.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Kconfig | 33 +++++
drivers/iommu/Makefile | 2 +
drivers/iommu/pisa-kunit.c | 102 +++++++++++++
drivers/iommu/pisa-meta.c | 286 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 31 ++++
include/linux/pisa.h | 79 ++++++++++
6 files changed, 533 insertions(+)
create mode 100644 drivers/iommu/pisa-kunit.c
create mode 100644 drivers/iommu/pisa-meta.c
create mode 100644 drivers/iommu/pisa-priv.h
create mode 100644 include/linux/pisa.h
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 85a8d7fdfa30b..15e06eaed0924 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -423,4 +423,37 @@ config IOMMU_DEBUG_PAGEALLOC
If unsure, say N.
endif # IOMMU_SUPPORT
+# Pre-mapped IOVA Superpage Allocator
+config PISA
+ bool "Pre-mapped IOVA Superpage Allocator"
+ depends on X86_64 || ARM64
+ depends on !PREEMPT_RT
+ default y
+ help
+ PISA is a specialized memory allocator with an API similar to
+ alloc_pages(). For devices that opt-in, it provides DMA-able
+ pages backed by 2MB physically contiguous pages, and mapped in
+ the IOMMU maps with a single leaf PTE of the same size. Mapping,
+ unmapping and IOTLB flushing are done lazily to reduce CPU overhead.
+ The large mappings greatly reduce IOTLB usage. As in strict iommu
+ mode, pages are returned to the buddy allocator only when any existing
+ mappings have been removed and IOTLB flushed.
+ Uses 4KB of metadata for each 32MB of PISA buffers, allocated on demand.
+
+ The config option enforces requirements (2MB large pages and !PREEMPT_RT) at compile
+ time. Other constraints (e.g. dma-coherent devices) are verified at runtime.
+
+ If unsure, say Y.
+
+config PISA_META_KUNIT_TEST
+ bool "KUnit test for PISA metadata table (built-in)" if !KUNIT_ALL_TESTS
+ depends on PISA && KUNIT=y
+ default KUNIT_ALL_TESTS
+ help
+ Builds KUnit unit tests for the sparse, per-PISA-frame metadata table,
+ and PISA page allocators in drivers/iommu/pisa-*.c, verifying metadata
+ lookup, allocator recycling, and coherent block allocation.
+
+ If unsure, say N.
+
source "drivers/iommu/generic_pt/Kconfig"
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 29a26d2a3af12..b5d3be1df3eda 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,6 +11,8 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+obj-$(CONFIG_PISA) += pisa-meta.o
+obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
new file mode 100644
index 0000000000000..0633f3238e012
--- /dev/null
+++ b/drivers/iommu/pisa-kunit.c
@@ -0,0 +1,102 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * KUnit tests for the opaque struct pisa_meta table API.
+ */
+#include <kunit/test.h>
+#include <linux/gfp.h>
+#include <linux/pisa.h>
+#include <linux/mm.h>
+
+#include "pisa-priv.h"
+
+static void test_meta_init_and_roundtrip(struct kunit *test)
+{
+ unsigned long pfn, base_pfn;
+ struct pisa_meta *m_pfn;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+ /* Second call must be idempotent. */
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ page = alloc_page(GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PISA_ORDER);
+ __free_page(page);
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_ensure_pfn(pfn, true), 0);
+ m_pfn = pisa_meta_of_pfn(pfn);
+ KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
+}
+
+static void test_meta_invalid_phys(struct kunit *test)
+{
+ struct pisa_meta *meta;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ meta = pisa_meta_of_pfn(PHYS_PFN(PHYS_ADDR_MAX));
+ KUNIT_ASSERT_NOT_NULL(test, meta);
+ KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(meta), ULONG_MAX);
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(ULONG_MAX >> PAGE_SHIFT));
+ KUNIT_EXPECT_NE(test, pisa_meta_ensure_pfn(ULONG_MAX >> PAGE_SHIFT, false), 0);
+}
+
+static void pisa_meta_set_active(struct pisa_meta *meta, bool active)
+{
+ WRITE_ONCE(meta->flags, active ? PISA_ACTIVE : 0);
+}
+
+static void test_meta_active_toggle(struct kunit *test)
+{
+ unsigned long pfn, base_pfn;
+ struct pisa_meta *meta;
+ struct page *page;
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ page = alloc_pages(GFP_KERNEL, PISA_ORDER);
+ KUNIT_ASSERT_NOT_NULL(test, page);
+
+ pfn = page_to_pfn(page);
+ base_pfn = ALIGN_DOWN(pfn, 1UL << PISA_ORDER);
+ /* The buddy allocator returns naturally aligned blocks. */
+ KUNIT_EXPECT_EQ(test, pfn, base_pfn);
+ if (pisa_meta_ensure_pfn(pfn, true)) {
+ __free_pages(page, PISA_ORDER);
+ KUNIT_FAIL(test, "pisa_meta_ensure_pfn(%lu) failed", pfn);
+ return;
+ }
+ KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
+ meta = pisa_meta_of_pfn(pfn);
+
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+ pisa_meta_set_active(meta, true);
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+
+ pisa_meta_set_active(meta, false);
+ KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+ __free_pages(page, PISA_ORDER);
+}
+
+static struct kunit_case pisa_meta_test_cases[] = {
+ KUNIT_CASE(test_meta_init_and_roundtrip),
+ KUNIT_CASE(test_meta_invalid_phys),
+ KUNIT_CASE(test_meta_active_toggle),
+ {}
+};
+
+static struct kunit_suite pisa_meta_test_suite = {
+ .name = "pisa_meta",
+ .test_cases = pisa_meta_test_cases,
+};
+
+kunit_test_suite(pisa_meta_test_suite);
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
new file mode 100644
index 0000000000000..5ceea91e4d5e8
--- /dev/null
+++ b/drivers/iommu/pisa-meta.c
@@ -0,0 +1,286 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA sparse per-PISA-frame metadata table.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/bitmap.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/memblock.h>
+#include <linux/memory.h>
+#include <linux/moduleparam.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/pgtable.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/vmalloc.h>
+
+#include "pisa-priv.h"
+
+static_assert(PISA_META_SIZE <= PAGE_SIZE);
+static_assert(PISA_SIZE <= SZ_2M);
+static_assert(PISA_BLOCKS(0) <= PISA_MAX_BLOCKS);
+static_assert(PISA_ORDER <= MAX_PAGE_ORDER);
+static_assert(PISA_SHIFT <= SUBSECTION_SHIFT);
+
+/* Enabled by pisa_meta_init() to activate fast-path hooks. */
+DEFINE_STATIC_KEY_FALSE(pisa_key);
+EXPORT_SYMBOL(pisa_key);
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Physical address bits covered by @pisa_meta_array (0 = max_possible_pfn). */
+static unsigned int pisa_max_bits __read_mostly;
+module_param_named(max_bits, pisa_max_bits, uint, 0444);
+
+/*
+ * Metadata is in a sparse @pisa_meta_array[pfn >> PISA_ORDER] in vmalloc space
+ * (256B per 2MB frame). Each 4KB backing page covers a chunk of 16 entries (32MB)
+ * and is tracked in @pisa_chunk_bitmap, 1 bit per chunk, 4KB per 1TB, 1MB for 48bit
+ * of address space. The total address space is @pisa_meta_nframes, all the variables
+ * are set and memory allocated before enabling @pisa_key.
+ */
+
+/* Sparse per-PISA-frame metadata table, indexed by pfn >> PISA_ORDER */
+struct pisa_meta *pisa_meta_array __read_mostly;
+EXPORT_SYMBOL(pisa_meta_array);
+
+/* Number of PISA frames covered by @pisa_meta_array. */
+unsigned long pisa_meta_nframes __read_mostly;
+EXPORT_SYMBOL(pisa_meta_nframes);
+
+/* One bit per 32MB chunk, set once its metadata page is backed. */
+unsigned long *pisa_chunk_bitmap __read_mostly;
+EXPORT_SYMBOL(pisa_chunk_bitmap);
+
+/* Shared sink entry returned for unbacked or out-of-range PFNs */
+static struct pisa_meta pisa_meta_nil __read_mostly;
+
+/* Protects init and page-table allocation */
+static DEFINE_MUTEX(pisa_meta_mutex);
+
+/* Return metadata for @pfn, or shared sink @pisa_meta_nil (flags == 0) if unbacked. */
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
+{
+ unsigned long frame = pfn >> PISA_ORDER;
+
+ if (unlikely(!static_branch_unlikely(&pisa_key) || frame >= pisa_meta_nframes ||
+ !test_bit_acquire(pfn >> PISA_CHUNK_ORDER, pisa_chunk_bitmap)))
+ return &pisa_meta_nil;
+
+ return &pisa_meta_array[frame];
+}
+
+/* Base PFN of @meta's PISA frame, or ULONG_MAX if out-of-range/sink. */
+unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
+{
+ unsigned long frame = meta - pisa_meta_array;
+
+ if (unlikely(frame >= pisa_meta_nframes))
+ return ULONG_MAX;
+
+ return frame << PISA_ORDER;
+}
+
+/* PTE for @addr in the array, or NULL if its page table was not allocated. */
+static pte_t *pisa_meta_pte(unsigned long addr)
+{
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ if (pgd_none(pgdp_get(pgd)))
+ return NULL;
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(p4dp_get(p4d)))
+ return NULL;
+ pud = pud_offset(p4d, addr);
+ if (pud_none(pudp_get(pud)))
+ return NULL;
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(pmdp_get(pmd)))
+ return NULL;
+ return pte_offset_kernel(pmd, addr);
+}
+
+/* Back chunk of @pfn in @pisa_meta_array with a zeroed page and set its bitmap bit. */
+static int pisa_meta_back(unsigned long pfn, gfp_t gfp)
+{
+ /* Protects lazy installation of backing pages into preallocated page tables. */
+ static DEFINE_SPINLOCK(pisa_meta_pte_lock);
+ unsigned long idx = pfn >> PISA_CHUNK_ORDER;
+ unsigned long addr = (unsigned long)pisa_meta_array + (idx << PAGE_SHIFT), flags;
+ struct page *online = pfn_to_online_page(pfn);
+ int nid = online ? page_to_nid(online) : NUMA_NO_NODE;
+ pte_t *ptep = pisa_meta_pte(addr);
+ struct page *page;
+
+ if (!ptep)
+ return -EINVAL;
+
+ page = alloc_pages_node(nid, gfp | __GFP_ZERO | __GFP_NOWARN, 0);
+ if (!page)
+ return -ENOMEM;
+
+ spin_lock_irqsave(&pisa_meta_pte_lock, flags);
+ if (pte_none(ptep_get(ptep))) {
+ set_pte_at(&init_mm, addr, ptep, pfn_pte(page_to_pfn(page), PAGE_KERNEL));
+ flush_cache_vmap(addr, addr + PAGE_SIZE);
+ page = NULL;
+ /* Pair with test_bit_acquire() in readers. */
+ smp_mb__before_atomic();
+ set_bit(idx, pisa_chunk_bitmap);
+ }
+ spin_unlock_irqrestore(&pisa_meta_pte_lock, flags);
+
+ if (page)
+ __free_page(page);
+ return 0;
+}
+
+/* Allocate the page tables for the entries of [@start_pfn, @end_pfn). */
+static int pisa_meta_prealloc(struct pisa_meta *array, unsigned long nframes,
+ unsigned long start_pfn, unsigned long end_pfn)
+{
+ unsigned long start, end;
+
+ end_pfn = min(end_pfn, nframes << PISA_ORDER);
+ if (start_pfn >= end_pfn)
+ return 0;
+
+ start = round_down((unsigned long)&array[start_pfn >> PISA_ORDER], PAGE_SIZE);
+ end = round_up((unsigned long)&array[((end_pfn - 1) >> PISA_ORDER) + 1], PAGE_SIZE);
+ return apply_to_page_range(&init_mm, start, end - start, NULL, NULL);
+}
+
+struct pisa_meta_range {
+ struct pisa_meta *array;
+ unsigned long nframes;
+};
+
+static int pisa_meta_prealloc_ram(unsigned long start_pfn, unsigned long nr_pages, void *arg)
+{
+ struct pisa_meta_range *r = arg;
+
+ return pisa_meta_prealloc(r->array, r->nframes, start_pfn, start_pfn + nr_pages);
+}
+
+/* Page tables for memory about to go online; failure keeps it out of the allocators. */
+static int pisa_meta_memory_cb(struct notifier_block *nb, unsigned long action, void *arg)
+{
+ struct memory_notify *mn = arg;
+
+ if (action != MEM_GOING_ONLINE)
+ return NOTIFY_OK;
+
+ mutex_lock(&pisa_meta_mutex);
+ if (pisa_meta_array)
+ pisa_meta_prealloc(pisa_meta_array, pisa_meta_nframes, mn->start_pfn,
+ mn->start_pfn + mn->nr_pages);
+ mutex_unlock(&pisa_meta_mutex);
+ return NOTIFY_OK;
+}
+
+/* Memory hotplug notifier for preallocating metadata page tables. */
+static struct notifier_block pisa_meta_memory_nb = {
+ .notifier_call = pisa_meta_memory_cb,
+};
+
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block)
+{
+ unsigned long idx = pfn >> PISA_CHUNK_ORDER;
+
+ if (!static_branch_unlikely(&pisa_key) || (pfn >> PISA_ORDER) >= pisa_meta_nframes)
+ return -ERANGE;
+
+ if (likely(test_bit_acquire(idx, pisa_chunk_bitmap)))
+ return 0;
+
+ /* Memory onlined before the notifier was registered has no page table. */
+ if (can_block && !pisa_meta_pte((unsigned long)pisa_meta_array + (idx << PAGE_SHIFT))) {
+ int err;
+
+ mutex_lock(&pisa_meta_mutex);
+ err = pisa_meta_prealloc(pisa_meta_array, pisa_meta_nframes, pfn, pfn + 1);
+ mutex_unlock(&pisa_meta_mutex);
+ if (err)
+ return err;
+ }
+
+ return pisa_meta_back(pfn, can_block ? GFP_KERNEL : GFP_ATOMIC);
+}
+
+/* Reserve the sparse @pisa_meta_array and preallocate page tables for present RAM. */
+int pisa_meta_init(void)
+{
+ unsigned long top_pfn, nframes, nchunks, size;
+ struct pisa_meta_range range;
+ struct pisa_meta *array;
+ struct vm_struct *vm;
+ int ret = 0;
+
+ if (static_branch_unlikely(&pisa_key))
+ return 0;
+
+ mutex_lock(&pisa_meta_mutex);
+ if (pisa_meta_array)
+ goto out_unlock;
+
+ if (pisa_max_bits)
+ top_pfn = 1UL << (clamp_t(unsigned int, pisa_max_bits, PISA_SHIFT,
+ MAX_PHYSMEM_BITS) - PAGE_SHIFT);
+ else
+ top_pfn = max_t(unsigned long, max_pfn, max_possible_pfn);
+ nframes = DIV_ROUND_UP(top_pfn, 1UL << PISA_ORDER);
+ size = PAGE_ALIGN(nframes << PISA_META_SHIFT);
+ nchunks = size >> PAGE_SHIFT;
+
+ /* kvalloc() to avoid kmalloc limitations on large layouts */
+ pisa_chunk_bitmap = kvcalloc(BITS_TO_LONGS(nchunks), sizeof(unsigned long), GFP_KERNEL);
+ if (!pisa_chunk_bitmap) {
+ ret = -ENOMEM;
+ goto out_unlock;
+ }
+
+ vm = get_vm_area(size, VM_MAP);
+ if (!vm) {
+ ret = -ENOMEM;
+ goto out_free_bitmap;
+ }
+ array = vm->addr;
+
+ range.array = array;
+ range.nframes = nframes;
+ ret = walk_system_ram_range(0, nframes << PISA_ORDER, &range, pisa_meta_prealloc_ram);
+ if (ret)
+ goto out_free_vm;
+
+ pisa_meta_nframes = nframes;
+ pisa_meta_array = array;
+ static_branch_enable(&pisa_key);
+
+ pr_info("pisa: %lu frames, %lu KB of KVA\n", nframes, size / 1024);
+ mutex_unlock(&pisa_meta_mutex);
+
+ /* Register outside pisa_meta_mutex (taken by the notifier under chain lock). */
+ register_memory_notifier(&pisa_meta_memory_nb);
+ return 0;
+
+out_free_vm:
+ free_vm_area(vm);
+out_free_bitmap:
+ kvfree(pisa_chunk_bitmap);
+ pisa_chunk_bitmap = NULL;
+out_unlock:
+ mutex_unlock(&pisa_meta_mutex);
+ return ret;
+}
+EXPORT_SYMBOL(pisa_meta_init);
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
new file mode 100644
index 0000000000000..c873970f6a670
--- /dev/null
+++ b/drivers/iommu/pisa-priv.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
+#define _DRIVERS_IOMMU_PISA_PRIV_H
+
+#include <linux/pisa.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_PISA
+
+#define PISA_MAX_BLOCKS 512U
+
+#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+
+/**
+ * struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
+ * @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ */
+struct pisa_meta {
+ u8 flags;
+} __aligned(PISA_META_SIZE);
+
+static_assert(offsetof(struct pisa_meta, flags) == 0);
+static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
+unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+
+#endif /* CONFIG_PISA */
+#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
new file mode 100644
index 0000000000000..acf92ced245d8
--- /dev/null
+++ b/include/linux/pisa.h
@@ -0,0 +1,79 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* See Documentation/core-api/pisa.rst for the architecture overview. */
+#ifndef _LINUX_PISA_H
+#define _LINUX_PISA_H
+
+#include <linux/bitops.h>
+#include <linux/jump_label.h>
+#include <linux/mm.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_PISA
+
+#define PISA_SHIFT 21U
+#define PISA_SIZE (1UL << PISA_SHIFT)
+#define PISA_MASK (~(PISA_SIZE - 1))
+#define PISA_ORDER (PISA_SHIFT - PAGE_SHIFT)
+
+/*
+ * Each struct pisa_meta entry in pisa_meta_array has size PISA_META_SIZE
+ * (256 B) and starts with a u8 flags byte.
+ */
+#define PISA_META_SHIFT 8
+#define PISA_META_SIZE (1U << PISA_META_SHIFT)
+#define PISA_CHUNK_ORDER (PISA_ORDER + PAGE_SHIFT - PISA_META_SHIFT)
+
+enum {
+ PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
+};
+
+/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
+DECLARE_STATIC_KEY_FALSE(pisa_key);
+
+struct pisa_meta;
+extern struct pisa_meta *pisa_meta_array;
+extern unsigned long pisa_meta_nframes;
+extern unsigned long *pisa_chunk_bitmap;
+
+int pisa_meta_init(void);
+
+/* fast membership check for pisa pfn/pages */
+static inline bool is_pisa_pfn(unsigned long pfn)
+{
+ unsigned long frame = pfn >> PISA_ORDER;
+ const u8 *flags;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+ if (unlikely(frame >= pisa_meta_nframes ||
+ !test_bit_acquire(pfn >> PISA_CHUNK_ORDER, pisa_chunk_bitmap)))
+ return false;
+
+ flags = (const u8 *)pisa_meta_array + (frame << PISA_META_SHIFT);
+ return READ_ONCE(*flags) & PISA_ACTIVE;
+}
+
+#else /* !CONFIG_PISA */
+
+static inline int pisa_meta_init(void)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline bool is_pisa_pfn(unsigned long pfn)
+{
+ return false;
+}
+
+#endif /* CONFIG_PISA */
+
+static inline bool is_pisa_phys(phys_addr_t phys)
+{
+ return is_pisa_pfn(phys >> PAGE_SHIFT);
+}
+
+static inline bool is_pisa_page(const struct page *page)
+{
+ return is_pisa_pfn(page_to_pfn(page));
+}
+#endif /* _LINUX_PISA_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook
[not found] <20261003212241.3432303-1-lrizzo@google.com>
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
` (22 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA allocators use 2MB physically contiguous memory pages from which IO
buffers are allocated. Recycling memory to the PISA allocator amortizes the
cost of mapping, unmapping and IOTLB flush. Memory is returned to the buddy
allocator only when all active IOMMU/IOTLB mappings have been flushed,
so sensitive data or code will not ever be accessible to IO devices.
Add the allocator creation (struct pisa_alloc, pisa_create()) and the
page release side, hooking into __free_pages_prepare() so that PISA
pages are returned to the allocator.
Also implement asynchronous PISA page retirement and release to the
buddy allocator via a WQ_MEM_RECLAIM worker. Retirement removes PISA
pages from allocators and destroys the mappings flushing the IOTLB, but
defers the buddy release by an RCU grace period.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 369 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 10 +-
drivers/iommu/pisa-priv.h | 85 +++++++++
include/linux/pisa.h | 28 ++-
mm/page_alloc.c | 21 +++
6 files changed, 512 insertions(+), 3 deletions(-)
create mode 100644 drivers/iommu/pisa-alloc.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index b5d3be1df3eda..bda0b102dd3ba 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
new file mode 100644
index 0000000000000..68f5d333ae15c
--- /dev/null
+++ b/drivers/iommu/pisa-alloc.c
@@ -0,0 +1,369 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA streaming page allocator, async page reclaim, shrinker, and debugfs.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-mapping.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/workqueue.h>
+
+#include "pisa-priv.h"
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Idle PISA pages over the allocator watermark queued locklessly for release. */
+static LLIST_HEAD(pisa_free_list);
+
+/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
+static struct workqueue_struct *pisa_wq __ro_after_init;
+
+/* Max completely idle PISA pages kept per allocator before async release. */
+static unsigned int pisa_max_idle __read_mostly = 1;
+module_param_named(max_idle, pisa_max_idle, uint, 0644);
+
+/* Grace period in ms before releasing idle PISA pages above @pisa_max_idle (0 = immediate). */
+static unsigned int pisa_idle_ms __read_mostly = 200;
+module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
+
+/* All live allocators, protected by pisa_allocs_lock. */
+static LIST_HEAD(pisa_allocs);
+static DEFINE_RWLOCK(pisa_allocs_lock);
+
+static void pisa_schedule_reclaim(void);
+
+/*
+ * Idle PISA page release proceeds in three stages:
+ * 1. Detach onto lockless pisa_free_list and schedule pisa_reclaim_work.
+ * 2. In process context (pisa_release_page()), unmap from all IOMMU domains,
+ * clear PISA_ACTIVE, and queue pisa_release_page_rcu() via call_rcu().
+ * 3. After an RCU grace period, pisa_release_page_rcu() unfreezes all blocks
+ * and returns them to the buddy allocator.
+ */
+
+/* RCU callback: unfreeze and return a retired PISA page's blocks to the buddy allocator. */
+static void pisa_release_page_rcu(struct rcu_head *head)
+{
+ /* Read @meta before freeing blocks; the frame can be reused immediately. */
+ struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
+ struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
+ unsigned int i, order = meta->pa->order;
+
+ WRITE_ONCE(meta->pa, NULL);
+
+ for (i = 0; i < PISA_BLOCKS(order); i++) {
+ block = page + (i << order);
+ page_ref_unfreeze(block, 1);
+ __free_pages(block, order);
+ }
+}
+
+/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
+static void pisa_release_page(struct pisa_meta *meta)
+{
+ /* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
+
+ /* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
+ WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
+ call_rcu(&meta->rcu, pisa_release_page_rcu);
+}
+
+/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
+enum pisa_meta_list {
+ PISA_LIST_NONE,
+ PISA_LIST_FULL,
+ PISA_LIST_IDLE_CLEAN, /* head of @idle */
+ PISA_LIST_IDLE_DIRTY, /* tail of @idle */
+ PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
+ PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
+ PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
+ PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
+};
+
+static void pisa_reclaim_work_fn(struct work_struct *work);
+
+/* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
+static DECLARE_DELAYED_WORK(pisa_reclaim_work, pisa_reclaim_work_fn);
+
+static void pisa_reclaim_work_fn(struct work_struct *work)
+{
+ unsigned long grace = msecs_to_jiffies(READ_ONCE(pisa_idle_ms));
+ unsigned long next_delay = MAX_JIFFY_OFFSET;
+ struct pisa_meta *meta, *tmp;
+ struct llist_node *node;
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ unsigned long expire = READ_ONCE(pa->idle_jiffies) + grace;
+ unsigned long now = jiffies, flags;
+
+ if (READ_ONCE(pa->num_idle_pages) <= pa->max_idle_pages)
+ continue;
+ if (grace && time_before(now, expire)) {
+ next_delay = min(next_delay, expire - now);
+ continue;
+ }
+ spin_lock_irqsave(&pa->lock, flags);
+ while (pa->num_idle_pages > pa->max_idle_pages && !list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ list_del_init(&meta->list);
+ WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+ llist_add(&meta->llnode, &pisa_free_list);
+ pa->num_idle_pages--;
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ node = llist_del_all(&pisa_free_list);
+ llist_for_each_entry_safe(meta, tmp, node, llnode)
+ pisa_release_page(meta);
+
+ if (next_delay != MAX_JIFFY_OFFSET)
+ queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
+}
+
+static void pisa_schedule_reclaim(void)
+{
+ struct workqueue_struct *wq = pisa_wq ?: system_wq;
+ unsigned int idle_ms = READ_ONCE(pisa_idle_ms);
+
+ if (!llist_empty(&pisa_free_list) || !idle_ms)
+ mod_delayed_work(wq, &pisa_reclaim_work, 0);
+ else
+ queue_delayed_work(wq, &pisa_reclaim_work, msecs_to_jiffies(idle_ms));
+}
+/*
+ * Update @meta's position on @pa's lists under @pa->lock. When all blocks
+ * become available (avail == bpp), either queues @meta to @pisa_free_list (and
+ * returns true so the caller schedules reclaim) or moves it to @pa->idle.
+ */
+static bool pisa_place_meta(struct pisa_meta *meta)
+{
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int avail = pisa_meta_avail(meta), bpp = PISA_BLOCKS(pa->order);
+ u8 on_list;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (avail == bpp) {
+ if (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms)) {
+ list_del_init(&meta->list);
+ WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+ llist_add(&meta->llnode, &pisa_free_list);
+ return true;
+ }
+ if (pa->num_idle_pages >= pa->max_idle_pages)
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ pa->num_idle_pages++;
+ on_list = meta->nr_dirty ? PISA_LIST_IDLE_DIRTY : PISA_LIST_IDLE_CLEAN;
+ } else if (!avail) {
+ on_list = PISA_LIST_FULL;
+ } else {
+ bool sparse;
+
+ if (avail >= bpp - (bpp >> 2))
+ sparse = true;
+ else if (avail <= (bpp >> 1))
+ sparse = false;
+ else
+ sparse = meta->on_list == PISA_LIST_PARTIAL_CLEAN_SPARSE ||
+ meta->on_list == PISA_LIST_PARTIAL_DIRTY_SPARSE;
+
+ if (meta->nr_clean)
+ on_list = sparse ? PISA_LIST_PARTIAL_CLEAN_SPARSE :
+ PISA_LIST_PARTIAL_CLEAN_DENSE;
+ else
+ on_list = sparse ? PISA_LIST_PARTIAL_DIRTY_SPARSE :
+ PISA_LIST_PARTIAL_DIRTY_DENSE;
+ }
+
+ if (on_list == meta->on_list)
+ return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
+ WRITE_ONCE(meta->on_list, on_list);
+
+ switch (on_list) {
+ case PISA_LIST_FULL:
+ list_move(&meta->list, &pa->full);
+ break;
+ case PISA_LIST_IDLE_CLEAN:
+ list_move(&meta->list, &pa->idle);
+ break;
+ case PISA_LIST_IDLE_DIRTY:
+ list_move_tail(&meta->list, &pa->idle);
+ break;
+ case PISA_LIST_PARTIAL_CLEAN_DENSE:
+ list_move(&meta->list, &pa->partial);
+ break;
+ case PISA_LIST_PARTIAL_CLEAN_SPARSE:
+ list_move_tail(&meta->list, &pa->partial);
+ break;
+ case PISA_LIST_PARTIAL_DIRTY_DENSE:
+ list_move(&meta->list, &pa->partial_dirty);
+ break;
+ default:
+ list_move_tail(&meta->list, &pa->partial_dirty);
+ }
+ return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
+}
+
+/* Check for hwpoisoned block; if poisoned, warn once and leak it to keep @pa intact. */
+static bool pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
+{
+ unsigned int i, nr = 1U << pa->order;
+
+ for (i = 0; i < nr; i++) {
+ if (unlikely(PageHWPoison(block + i))) {
+ pr_err_once("pisa: poisoned block at pfn %lu, leaking it to keep allocator %p intact\n",
+ page_to_pfn(block), pa);
+ return true;
+ }
+ }
+ return false;
+}
+
+/**
+ * pisa_create - Create a PISA page allocator
+ * @nid: Home NUMA node for this allocator
+ * @order: Block order to dispense (<= PISA_ORDER)
+ * @dir: Supported DMA direction (DMA_BIDIRECTIONAL, DMA_TO_DEVICE, or
+ * DMA_FROM_DEVICE); other mappings fall back to per-buffer IOMMU mappings
+ *
+ * Return: Pointer to the created allocator, or NULL on failure.
+ */
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir)
+{
+ struct pisa_alloc *pa;
+
+ if (order > PISA_ORDER || !valid_dma_direction(dir))
+ return NULL;
+
+ pa = kzalloc(sizeof(*pa), GFP_KERNEL);
+ if (!pa)
+ return NULL;
+
+ spin_lock_init(&pa->lock);
+ pa->order = order;
+ pa->nid = pisa_sanitize_nid(nid);
+ pa->max_idle_pages = READ_ONCE(pisa_max_idle) ? : 1;
+ pisa_set_name(pa, "%ps", (void *)_RET_IP_);
+ INIT_LIST_HEAD(&pa->partial);
+ INIT_LIST_HEAD(&pa->partial_dirty);
+ INIT_LIST_HEAD(&pa->idle);
+ INIT_LIST_HEAD(&pa->full);
+
+ write_lock(&pisa_allocs_lock);
+ list_add_tail(&pa->node, &pisa_allocs);
+ write_unlock(&pisa_allocs_lock);
+
+ return pa;
+}
+
+/**
+ * pisa_set_name - Name a PISA allocator for debugfs
+ * @pa: Allocator to name (may be NULL)
+ * @fmt: printf-style format, truncated to fit @pa->name
+ */
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...)
+{
+ va_list args;
+
+ if (!pa)
+ return;
+ va_start(args, fmt);
+ vsnprintf(pa->name, sizeof(pa->name), fmt, args);
+ va_end(args);
+}
+
+/* Return freed @block (at refcount 0) to @pa's bitmaps under @pa->lock. */
+static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block, bool *release)
+{
+ unsigned long pfn = page_to_pfn(block);
+ struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+ unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (WARN_ON_ONCE(test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+ return;
+
+ /* __pisa_free_page() zeroed it if init_on_free is set. */
+ if (want_init_on_free()) {
+ __set_bit(idx, meta->clean_bitmap);
+ meta->nr_clean++;
+ } else {
+ __set_bit(idx, meta->dirty_bitmap);
+ meta->nr_dirty++;
+ }
+ *release |= pisa_place_meta(meta);
+}
+
+/**
+ * __pisa_free_page - Recycle a freed block back into its PISA allocator
+ * @page: Block being freed (called from __free_pages_prepare() under RCU)
+ *
+ * Return: true if the page belonged to a PISA allocator and was recycled.
+ */
+bool __pisa_free_page(struct page *page)
+{
+ bool should_release = false;
+ unsigned long pfn = page_to_pfn(page), flags;
+ struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+ /* Reject double frees, mismatched orders, or unaligned blocks. */
+ if (WARN_ON_ONCE(!IS_ALIGNED(pfn, 1UL << pa->order) ||
+ compound_order(page) != pa->order ||
+ test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+ return true;
+
+ /* Never reuse or return a hwpoisoned block to the buddy allocator. */
+ if (pisa_block_poisoned(pa, page))
+ return true;
+
+ /* Checks skipped by the early return in __free_pages_prepare(). */
+ if (unlikely(!free_page_prepare_pisa(page, pa->order)))
+ return true;
+
+ /* KASAN/KMSAN/pgalloc_tag_sub run when pisa_release_page_rcu() frees to buddy. */
+ if (want_init_on_free())
+ memset(page_address(page), 0, PAGE_SIZE << pa->order);
+
+ spin_lock_irqsave(&pa->lock, flags);
+ pisa_return_block_locked(pa, page, &should_release);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return true;
+}
+
+static int __init pisa_init(void)
+{
+ /* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
+ pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
+ if (!pisa_wq)
+ pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+
+ return 0;
+}
+subsys_initcall(pisa_init);
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 0633f3238e012..12edc73abc9c6 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -85,10 +85,18 @@ static void test_meta_active_toggle(struct kunit *test)
__free_pages(page, PISA_ORDER);
}
+static void test_alloc_and_recycle(struct kunit *test)
+{
+ KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, 0, DMA_NONE));
+ KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, PISA_ORDER + 1, DMA_BIDIRECTIONAL));
+ pisa_set_name(NULL, "ignored");
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
+ KUNIT_CASE(test_alloc_and_recycle),
{}
};
@@ -98,5 +106,5 @@ static struct kunit_suite pisa_meta_test_suite = {
};
kunit_test_suite(pisa_meta_test_suite);
-MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table and PISA allocator");
MODULE_LICENSE("Dual BSD/GPL");
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index c873970f6a670..769b0ef071705 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -2,8 +2,13 @@
#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
#define _DRIVERS_IOMMU_PISA_PRIV_H
+#include <linux/dma-direction.h>
#include <linux/pisa.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/nodemask.h>
#include <linux/spinlock.h>
+#include <linux/topology.h>
#include <linux/types.h>
#ifdef CONFIG_PISA
@@ -12,20 +17,100 @@
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/*
+ * Lock hierarchy:
+ * - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
+ * - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
+ * - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ */
+
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ * @nr_clean: Zeroed available blocks in @clean_bitmap
+ * @nr_dirty: Dirty available blocks in @dirty_bitmap
+ * @on_list: Current placement in @pa (enum pisa_meta_list)
+ * @pa: Owning pisa_alloc (NULL for reservoir pages)
+ * @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
+ * @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
+ * @rcu: RCU head used to defer release to the buddy allocator
+ * @clean_bitmap: Bitmap of zeroed available block indices
+ * @dirty_bitmap: Bitmap of dirty available block indices
*/
struct pisa_meta {
u8 flags;
+ u16 nr_clean;
+ u16 nr_dirty;
+ u8 on_list;
+ struct pisa_alloc *pa;
+ struct list_head list;
+
+ union {
+ struct llist_node llnode;
+ struct rcu_head rcu;
+ };
+
+ DECLARE_BITMAP(clean_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
+ DECLARE_BITMAP(dirty_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
} __aligned(PISA_META_SIZE);
static_assert(offsetof(struct pisa_meta, flags) == 0);
static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
+{
+ return meta->nr_clean + meta->nr_dirty;
+}
+
+/**
+ * struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
+ * @lock: Spinlock protecting lists and block bitmaps
+ * @order: Block order managed by this allocator (<= PISA_ORDER)
+ * @partial: Partially used PISA pages with @nr_clean > 0
+ * @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
+ * @idle: Completely free PISA pages (reclaimable under pressure)
+ * @full: List of PISA pages with 0 free blocks
+ * @num_idle_pages: Length of @idle
+ * @max_idle_pages: High watermark of @idle before async release to the buddy allocator
+ * @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @nid: NUMA node id
+ * @node: Node in the global pisa_allocs list
+ * @name: Debug name (defaults to caller of pisa_create())
+ */
+struct pisa_alloc {
+ /* First cacheline: hot fields touched on block alloc and free. */
+ spinlock_t lock;
+ u8 order;
+ struct list_head partial;
+ struct list_head partial_dirty;
+ struct list_head idle;
+ struct list_head full;
+ unsigned int num_idle_pages;
+ unsigned int max_idle_pages;
+ unsigned long idle_jiffies;
+
+ /* Cold. */
+ int nid;
+ struct list_head node;
+ char name[48];
+} ____cacheline_aligned;
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
+__printf(2, 3)
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+
+static inline int pisa_sanitize_nid(int nid)
+{
+ if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
+ nid = numa_mem_id();
+ if (!node_state(nid, N_MEMORY))
+ nid = first_memory_node;
+ return nid;
+}
+
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index acf92ced245d8..dddcedb8c93f6 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -3,10 +3,11 @@
#ifndef _LINUX_PISA_H
#define _LINUX_PISA_H
+#include <linux/types.h>
#include <linux/bitops.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
-#include <linux/types.h>
+#include <linux/rcupdate.h>
#ifdef CONFIG_PISA
@@ -53,6 +54,24 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return READ_ONCE(*flags) & PISA_ACTIVE;
}
+static inline bool is_pisa_page(const struct page *page);
+
+bool __pisa_free_page(struct page *page);
+bool free_page_prepare_pisa(struct page *page, unsigned int order);
+
+static inline bool pisa_free_page(struct page *page)
+{
+ bool ret;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+ rcu_read_lock();
+ ret = is_pisa_page(page) && __pisa_free_page(page);
+ rcu_read_unlock();
+
+ return ret;
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -60,11 +79,18 @@ static inline int pisa_meta_init(void)
return -EOPNOTSUPP;
}
+bool free_page_prepare_pisa(struct page *page, unsigned int order);
+
static inline bool is_pisa_pfn(unsigned long pfn)
{
return false;
}
+static inline bool pisa_free_page(struct page *page)
+{
+ return false;
+}
+
#endif /* CONFIG_PISA */
static inline bool is_pisa_phys(phys_addr_t phys)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index eb4f02617f53c..48df87a19d312 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -17,6 +17,7 @@
#include <linux/stddef.h>
#include <linux/mm.h>
#include <linux/highmem.h>
+#include <linux/pisa.h>
#include <linux/interrupt.h>
#include <linux/jiffies.h>
#include <linux/compiler.h>
@@ -1298,6 +1299,23 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
#endif /* CONFIG_MEM_ALLOC_PROFILING */
+/*
+ * pisa_free_page() in __free_pages_prepare() recycles a PISA block into
+ * its allocator before the rest of that function runs. __pisa_free_page() calls
+ * this to apply the buddy allocator's checks.
+ *
+ * Return: false if the block is in a bad state (already reported): leak it.
+ */
+bool free_page_prepare_pisa(struct page *page, unsigned int order)
+{
+ if (!IS_ENABLED(CONFIG_PISA))
+ return true;
+ if (is_check_pages_enabled() && free_page_is_bad(page))
+ return false;
+ page_table_check_free(page, order);
+ return true;
+}
+
static __always_inline bool __free_pages_prepare(struct page *page,
unsigned int order, fpi_t fpi_flags)
{
@@ -1312,6 +1330,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
VM_BUG_ON_PAGE(PageTail(page), page);
+ if (unlikely(pisa_free_page(page)))
+ return false;
+
trace_mm_page_free(page, order);
kmsan_free_page(page, order);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (2 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
` (21 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Allocating a large page may involve compacting and blocking, so it cannot
be issued in GFP_ATOMIC context.
Add a per-NUMA-node reservoir of pre-allocated and set up PISA pages,
that can serve allocations without blocking. The reservoir is refilled in
process context by a per-node work item, queued with queue_work_node()
on system_dfl_wq so each node is refilled from its own CPUs, not bound
to the CPU that caused the miss.
pisa.reservoir_max, writable at runtime, sets the number of PISA pages
per node (default 16 = 32 MB per node; 0 = disabled).
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 17 ++-
drivers/iommu/pisa-kunit.c | 7 +
drivers/iommu/pisa-meta.c | 1 +
drivers/iommu/pisa-node.c | 268 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 30 +++++
6 files changed, 323 insertions(+), 2 deletions(-)
create mode 100644 drivers/iommu/pisa-node.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index bda0b102dd3ba..bc58166f9303a 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 68f5d333ae15c..cf559ee86d6f9 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -46,6 +46,14 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+struct pisa_node pisa_nodes[MAX_NUMNODES] = {
+ [0 ... MAX_NUMNODES - 1] = {
+ .next_alloc = INITIAL_JIFFIES,
+ .lock = __SPIN_LOCK_UNLOCKED(pisa_nodes.lock),
+ },
+};
+
static void pisa_schedule_reclaim(void);
/*
@@ -75,8 +83,13 @@ static void pisa_release_page_rcu(struct rcu_head *head)
}
/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
-static void pisa_release_page(struct pisa_meta *meta)
+void pisa_release_page(struct pisa_meta *meta)
{
+ if (!meta->pa) {
+ __free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
+ return;
+ }
+
/* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
@@ -359,6 +372,8 @@ bool __pisa_free_page(struct page *page)
static int __init pisa_init(void)
{
+ pisa_node_init();
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 12edc73abc9c6..079f6220e742a 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -92,11 +92,18 @@ static void test_alloc_and_recycle(struct kunit *test)
pisa_set_name(NULL, "ignored");
}
+static void test_reservoir_and_scrubber(struct kunit *test)
+{
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+ KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
KUNIT_CASE(test_alloc_and_recycle),
+ KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
index 5ceea91e4d5e8..00939c366ba6d 100644
--- a/drivers/iommu/pisa-meta.c
+++ b/drivers/iommu/pisa-meta.c
@@ -272,6 +272,7 @@ int pisa_meta_init(void)
/* Register outside pisa_meta_mutex (taken by the notifier under chain lock). */
register_memory_notifier(&pisa_meta_memory_nb);
+ pisa_schedule_reservoir();
return 0;
out_free_vm:
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
new file mode 100644
index 0000000000000..524f0d4457ad7
--- /dev/null
+++ b/drivers/iommu/pisa-node.c
@@ -0,0 +1,268 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-NUMA-node state: PISA page reservoir and background work.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/pisa.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/nodemask.h>
+#include <linux/sched.h>
+#include <linux/sched/mm.h>
+#include <linux/spinlock.h>
+#include <linux/topology.h>
+#include <linux/workqueue.h>
+
+#include "pisa-priv.h"
+
+/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/*
+ * Per-NUMA-node reservoir of pre-allocated, pre-zeroed, and pre-decrypted
+ * PISA pages (pisa.reservoir_max, default 16, 0 = disabled). Refilled and
+ * trimmed by each node's work item on system_dfl_wq, which also scrubs
+ * allocators on that node.
+ */
+unsigned int pisa_reservoir_max __read_mostly = 16;
+
+static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ /* Schedule per-node workers to refill or trim once pisa_meta_init() has run. */
+ if (!ret && static_branch_unlikely(&pisa_key))
+ pisa_schedule_reservoir();
+ return ret;
+}
+
+static const struct kernel_param_ops pisa_reservoir_max_ops = {
+ .set = pisa_reservoir_max_set,
+ .get = param_get_uint,
+};
+module_param_cb(reservoir_max, &pisa_reservoir_max_ops, &pisa_reservoir_max, 0644);
+
+/* Apply GFP_NOFS/GFP_NOIO scope around internal GFP_KERNEL allocations. */
+static unsigned int pisa_memalloc_save(gfp_t gfp)
+{
+ unsigned int flags = 0;
+
+ if (!(gfp & __GFP_FS))
+ flags |= PF_MEMALLOC_NOFS;
+ if (!(gfp & __GFP_IO))
+ flags |= PF_MEMALLOC_NOIO;
+ return memalloc_flags_save(flags);
+}
+
+/* Allocate a PISA page from the buddy allocator and initialize its metadata. */
+static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
+ bool *clean)
+{
+ struct pisa_meta *meta;
+ unsigned int memflags;
+ struct page *page;
+
+ if (unlikely(!static_branch_unlikely(&pisa_key))) {
+ if (!can_block || (gfp & (__GFP_FS | __GFP_IO)) != (__GFP_FS | __GFP_IO) ||
+ pisa_meta_init())
+ return ERR_PTR(-EAGAIN);
+ }
+
+ /*
+ * Long-lived active pages must be unmovable, lowmem, non-compound, and
+ * must not dip into emergency reserves or trigger OOM/warnings.
+ */
+ gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+ gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
+
+ if (can_block)
+ memflags = pisa_memalloc_save(gfp);
+ page = alloc_pages_node(nid, gfp, PISA_ORDER);
+ if (!page) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ /* Ensure the metadata chunk for this PFN is populated. */
+ if (unlikely(pisa_meta_ensure_pfn(page_to_pfn(page), can_block))) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ __free_pages(page, PISA_ORDER);
+ return ERR_PTR(-ERANGE);
+ }
+ if (can_block)
+ memalloc_flags_restore(memflags);
+
+ meta = pisa_meta_of_pfn(page_to_pfn(page));
+ memset(meta, 0, sizeof(*meta));
+ *clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+
+ return meta;
+}
+
+
+/* Refill node @nid's reservoir up to pisa_reservoir_max pre-zeroed PISA pages. */
+static void pisa_reservoir_refill_node(int nid)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ while (res->nr_pages < READ_ONCE(pisa_reservoir_max)) {
+ struct pisa_meta *meta;
+ unsigned long flags;
+ bool clean;
+
+ meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+ if (IS_ERR(meta))
+ break;
+
+ /* init_on_alloc/init_on_free pages are already zeroed. */
+ if (!clean)
+ memset(page_address(pfn_to_page(pisa_meta_to_pfn(meta))), 0, PISA_SIZE);
+
+ spin_lock_irqsave(&res->lock, flags);
+ list_add(&meta->list, &res->pages);
+ res->nr_pages++;
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ cond_resched();
+ }
+}
+
+/* Release PISA pages in node @nid's reservoir above @max. */
+static void pisa_reservoir_trim_node(int nid, unsigned int max)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+ struct pisa_meta *meta, *tmp;
+ LIST_HEAD(free_list);
+ unsigned long flags;
+
+ spin_lock_irqsave(&res->lock, flags);
+ while (res->nr_pages > max && !list_empty(&res->pages)) {
+ list_move(res->pages.prev, &free_list);
+ res->nr_pages--;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ list_for_each_entry_safe(meta, tmp, &free_list, list) {
+ list_del_init(&meta->list);
+ pisa_release_page(meta);
+ cond_resched();
+ }
+}
+
+static void pisa_node_work_fn(struct work_struct *work)
+{
+ struct pisa_node *res = container_of(work, struct pisa_node, work);
+ int nid = res - pisa_nodes;
+ unsigned int max = READ_ONCE(pisa_reservoir_max);
+
+ if (unlikely(pisa_meta_init()))
+ return;
+
+ if (READ_ONCE(res->nr_pages) < max)
+ pisa_reservoir_refill_node(nid);
+ else if (READ_ONCE(res->nr_pages) > max)
+ pisa_reservoir_trim_node(nid, max);
+}
+
+/* Queue @nid's reservoir worker on a CPU belonging to @nid. */
+static void pisa_schedule_reservoir_node(int nid)
+{
+ if (nid >= 0 && nid < MAX_NUMNODES && node_state(nid, N_MEMORY))
+ queue_work_node(nid, system_dfl_wq, &pisa_nodes[nid].work);
+}
+
+/* Queue reservoir refill/trim workers on all memory nodes. */
+void pisa_schedule_reservoir(void)
+{
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ pisa_schedule_reservoir_node(nid);
+}
+
+/* Pop one pre-prepared PISA page from node @nid's reservoir under @res->lock. */
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+{
+ struct pisa_node *res = &pisa_nodes[nid];
+ struct pisa_meta *meta;
+ unsigned long flags;
+ bool refill = false;
+
+ if (!READ_ONCE(res->nr_pages))
+ return NULL;
+
+ spin_lock_irqsave(&res->lock, flags);
+ meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
+ if (meta) {
+ list_del_init(&meta->list);
+ res->nr_pages--;
+ refill = res->nr_pages <= max / 2;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+
+ if (refill)
+ pisa_schedule_reservoir_node(nid);
+
+ return meta;
+}
+
+/*
+ * Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
+ * refill when <= half full or empty), or allocate one inline on miss.
+ */
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+{
+ bool can_block = gfpflags_allow_blocking(gfp);
+ unsigned int max = READ_ONCE(pisa_reservoir_max);
+ struct pisa_meta *meta;
+
+ nid = pisa_sanitize_nid(nid);
+ if (max) {
+ meta = pisa_reservoir_pop_node(nid, max);
+ if (meta) {
+ *clean = true;
+ return meta;
+ }
+ /* Empty: pop_node() only schedules after a successful pop. */
+ pisa_schedule_reservoir_node(nid);
+ }
+
+ /* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
+ if (!can_block && time_before(jiffies, READ_ONCE(pisa_nodes[nid].next_alloc)))
+ return ERR_PTR(-EAGAIN);
+
+ meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+ if (IS_ERR(meta))
+ pisa_alloc_backoff(nid, can_block);
+ return meta;
+}
+
+/* Number of PISA pages held in all the reservoirs. */
+unsigned long pisa_reservoir_count(void)
+{
+ unsigned long nr = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ nr += READ_ONCE(pisa_nodes[nid].nr_pages);
+ return nr;
+}
+
+void __init pisa_node_init(void)
+{
+ int nid;
+
+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
+ spin_lock_init(&pisa_nodes[nid].lock);
+ INIT_LIST_HEAD(&pisa_nodes[nid].pages);
+ INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
+ }
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 769b0ef071705..0d097c665c790 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -4,12 +4,14 @@
#include <linux/dma-direction.h>
#include <linux/pisa.h>
+#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/nodemask.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
#include <linux/types.h>
+#include <linux/workqueue.h>
#ifdef CONFIG_PISA
@@ -95,6 +97,19 @@ struct pisa_alloc {
char name[48];
} ____cacheline_aligned;
+/* Per-NUMA-node PISA state. */
+struct pisa_node {
+ unsigned long next_alloc; /* atomic alloc backoff jiffies */
+
+ spinlock_t lock; /* protects @pages and @nr_pages */
+ struct list_head pages; /* reservoir */
+ unsigned int nr_pages;
+ struct work_struct work; /* refill/trim, on this node */
+} ____cacheline_aligned_in_smp;
+
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
@@ -102,6 +117,7 @@ unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
__printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+void pisa_release_page(struct pisa_meta *meta);
static inline int pisa_sanitize_nid(int nid)
{
@@ -112,5 +128,19 @@ static inline int pisa_sanitize_nid(int nid)
return nid;
}
+static inline void pisa_alloc_backoff(int nid, bool can_block)
+{
+ nid = pisa_sanitize_nid(nid);
+ if (!can_block)
+ WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
+}
+
+/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
+extern unsigned int pisa_reservoir_max;
+void pisa_node_init(void);
+void pisa_schedule_reservoir(void);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+unsigned long pisa_reservoir_count(void);
+
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (3 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
` (20 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Add block allocation (pisa_alloc_bulk(), pisa_alloc())
and split_page_compound(), which splits a newly acquired 2MB PISA page
into blocks of the allocator's order, frozen at refcount 0.
Introduce __GFP_PISA_RO and __GFP_PISA_RW GFP flags and hook
__alloc_frozen_pages_noprof() and alloc_pages_bulk_noprof() so callers
can allocate from per-node PISA allocators (pisa_nodes[nid].pa[ro][order])
using standard page allocation APIs.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 345 +++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 78 +++++++-
drivers/iommu/pisa-priv.h | 24 ++-
include/linux/gfp_types.h | 17 +-
include/linux/pisa.h | 25 +++
include/trace/events/mmflags.h | 11 +-
mm/page_alloc.c | 40 +++-
7 files changed, 527 insertions(+), 13 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index cf559ee86d6f9..1f1b99023094c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -14,13 +14,19 @@
#include <linux/gfp.h>
#include <linux/list.h>
#include <linux/llist.h>
+#include <linux/memcontrol.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/mutex.h>
+#include <linux/page_owner.h>
+#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
+#include <linux/sched/mm.h>
+#include <linux/skbuff.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
+#include <net/sock.h>
#include "pisa-priv.h"
@@ -328,6 +334,318 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
*release |= pisa_place_meta(meta);
}
+/*
+ * Return the predefined per-node allocator for (@nid, @order, @ro), or NULL.
+ */
+struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro)
+{
+ if (order > PISA_ORDER)
+ return NULL;
+ nid = pisa_sanitize_nid(nid);
+
+ return pisa_nodes[nid].pa[ro][order];
+}
+
+/**
+ * pisa_alloc_pages_bulk - Allocate pages from a per-node __GFP_PISA_* allocator
+ * @gfp: GFP flags (must have __GFP_PISA_RW or __GFP_PISA_RO set)
+ * @order: Page order
+ * @preferred_nid: Preferred NUMA node (or NUMA_NO_NODE)
+ * @nodemask: Allowed NUMA nodemask (declined if non-NULL)
+ * @nr_pages: Total entries in @page_array
+ * @page_array: Output array of page pointers (populated entries are skipped)
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array, bool frozen)
+{
+ struct pisa_alloc *pa = NULL;
+
+ if (unlikely(nr_pages <= 0))
+ return 0;
+ if (likely(!nodemask) && (!order || (gfp & __GFP_COMP)))
+ pa = pisa_node_pa(preferred_nid, order, !(gfp & __GFP_WRITE));
+ return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
+}
+
+/*
+ * Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
+ * pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
+ * Fails with -EBUSY if a speculative PFN walker holds a reference to @page.
+ */
+static int split_page_compound(struct page *page, unsigned int old_order, unsigned int new_order)
+{
+ unsigned int i;
+
+ if (WARN_ON_ONCE(PageCompound(page) || !page_count(page) ||
+ new_order > old_order ||
+ folio_memcg_charged(page_folio(page))))
+ return -EINVAL;
+
+ /* Freeze @page[0] (1 -> 0) so all resulting pieces start at refcount 0. */
+ if (!page_ref_freeze(page, 1))
+ return -EBUSY;
+
+ if (new_order < old_order) {
+ split_page_owner(page, old_order, new_order);
+ pgalloc_tag_split(page_folio(page), old_order, new_order);
+ }
+
+ /* Subpages of non-compound pages are already initialized as order 0. */
+ if (new_order == 0)
+ return 0;
+
+ for (i = 0; i < (1U << old_order); i += (1U << new_order))
+ prep_compound_page(page + i, new_order);
+
+ return 0;
+}
+
+/* Obtain and split a new PISA page on @nid into blocks of @order. */
+static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, int nid)
+{
+ bool can_block = gfpflags_allow_blocking(gfp), clean;
+ unsigned int bpp = PISA_BLOCKS(order);
+ struct pisa_meta *meta;
+ struct page *page;
+
+ /* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
+ meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+ if (IS_ERR(meta))
+ return meta;
+
+ page = pfn_to_page(pisa_meta_to_pfn(meta));
+
+ /* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
+ if (split_page_compound(page, PISA_ORDER, order)) {
+ pisa_release_page(meta);
+ pisa_alloc_backoff(nid, can_block);
+ return ERR_PTR(-EBUSY);
+ }
+
+ INIT_LIST_HEAD(&meta->list);
+
+ if (clean) {
+ bitmap_set(meta->clean_bitmap, 0, bpp);
+ meta->nr_clean = bpp;
+ } else {
+ bitmap_set(meta->dirty_bitmap, 0, bpp);
+ meta->nr_dirty = bpp;
+ }
+
+ return meta;
+}
+
+/* Reset fields normally cleared by prep_new_page()/post_alloc_hook(). */
+static void pisa_block_prep(struct page *block)
+{
+ set_page_private(block, 0);
+ clear_page_pfmemalloc(block);
+}
+
+/*
+ * Collect up to @want blocks from @meta's dirty or clean bitmap. With
+ * @unfreeze they get refcount 1; otherwise they stay at refcount 0.
+ */
+static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
+ unsigned int want, struct page **out, bool unfreeze)
+{
+ unsigned long *bitmap = dirty ? meta->dirty_bitmap : meta->clean_bitmap;
+ u16 *countp = dirty ? &meta->nr_dirty : &meta->nr_clean;
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int idx, got = 0, used = 0;
+ unsigned long base_pfn;
+
+ if (!*countp || !want)
+ return 0;
+
+ base_pfn = pisa_meta_to_pfn(meta);
+ for_each_set_bit(idx, bitmap, PISA_BLOCKS(pa->order)) {
+ struct page *block = pfn_to_page(base_pfn + (idx << pa->order));
+
+ __clear_bit(idx, bitmap);
+ used++;
+ if (!pisa_block_poisoned(pa, block)) {
+ pisa_block_prep(block);
+ if (unfreeze)
+ page_ref_unfreeze(block, 1);
+ out[got++] = block;
+ }
+
+ if (used == *countp || got == want)
+ break;
+ }
+ *countp -= used;
+ return got;
+}
+
+/* @zero: prefer PISA pages with blocks in @meta->clean_bitmap. */
+static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
+{
+ struct pisa_meta *m_dirty, *m_clean, *meta;
+
+ if (zero) {
+ /* Clean blocks first: the head of @partial, then of @idle. */
+ if (!list_empty(&pa->partial))
+ return list_first_entry(&pa->partial, struct pisa_meta, list);
+ meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
+ if (meta && meta->nr_clean) {
+ pa->num_idle_pages--;
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ return meta;
+ }
+ }
+
+ m_dirty = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+ m_clean = list_first_entry_or_null(&pa->partial, struct pisa_meta, list);
+ if (m_dirty && m_clean) {
+ bool dirty_dense = m_dirty->on_list == PISA_LIST_PARTIAL_DIRTY_DENSE;
+ bool clean_dense = m_clean->on_list == PISA_LIST_PARTIAL_CLEAN_DENSE;
+
+ if (dirty_dense != clean_dense)
+ return dirty_dense ? m_dirty : m_clean;
+ return pisa_meta_avail(m_dirty) <= pisa_meta_avail(m_clean) ?
+ m_dirty : m_clean;
+ }
+ if (m_dirty)
+ return m_dirty;
+ if (m_clean)
+ return m_clean;
+ if (!list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ pa->num_idle_pages--;
+ WRITE_ONCE(pa->idle_jiffies, jiffies);
+ return meta;
+ }
+ return NULL;
+}
+
+/* Extract up to @want blocks from @meta under @pa->lock. */
+static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
+ struct page **out, bool zero, bool frozen)
+{
+ unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
+ struct pisa_alloc *pa = meta->pa;
+
+ lockdep_assert_held(&pa->lock);
+
+ if (zero) {
+ unsigned int dirty, i;
+
+ /* First choice are clean blocks, followed by dirty */
+ got += pisa_scan_bitmap(meta, false, want, out, !frozen);
+ dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
+ for (i = got; i < got + dirty; i++)
+ set_page_private(out[i], 1);
+ got += dirty;
+ } else {
+ /* First choice are dirty blocks, followed by clean */
+ got += pisa_scan_bitmap(meta, true, want, out, !frozen);
+ got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+ }
+
+ used = avail_before - pisa_meta_avail(meta);
+ /* If counts disagreed with bitmaps, zero them so @meta moves to @pa->full. */
+ if (WARN_ON_ONCE(!used)) {
+ meta->nr_clean = 0;
+ meta->nr_dirty = 0;
+ }
+ pisa_place_meta(meta);
+
+ return got;
+}
+
+/* Allocate @nr_pages blocks from @pa. */
+static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+ struct page **pages, bool zero, bool frozen)
+{
+ unsigned long n, flags, i;
+ struct pisa_meta *meta;
+ unsigned int want;
+
+ /* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
+ spin_lock_irqsave(&pa->lock, flags);
+ for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+ want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
+ meta = pisa_pick_meta(pa, zero);
+ if (meta)
+ continue;
+
+ /* Fallback to a new allocation */
+ spin_unlock_irqrestore(&pa->lock, flags);
+ /* With @zero, fresh dirty blocks are zeroed below. */
+ meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
+ if (IS_ERR(meta))
+ goto out;
+ spin_lock_irqsave(&pa->lock, flags);
+ meta->pa = pa;
+ /* Set PISA_ACTIVE after @meta is fully set up. */
+ WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+out:
+ for (i = 0; zero && i < n; i++) {
+ /* Dirty blocks for a zeroing caller were marked with page_private = 1. */
+ if (!page_private(pages[i]))
+ continue;
+ set_page_private(pages[i], 0);
+ memset(page_address(pages[i]), 0, PAGE_SIZE << pa->order);
+ if (frozen) /* Order zeroing before caller publishes the page. */
+ smp_wmb();
+ else
+ page_ref_unfreeze(pages[i], 1);
+ }
+ return n;
+}
+
+/**
+ * pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
+ * @pa: Allocator to allocate from
+ * @gfp: GFP flags (requests with __GFP_THISNODE, __GFP_DMA, __GFP_DMA32, or
+ * __GFP_ACCOUNT are declined)
+ * @nr_pages: Number of entries in @page_array (populated entries are skipped)
+ * @page_array: Output array of page pointers
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+ unsigned long nr_pages, struct page **page_array, bool frozen)
+{
+ unsigned long i = 0, n, got;
+
+ if (unlikely(!pa ||
+ (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+ goto skip;
+
+ while (i < nr_pages) {
+ if (page_array[i]) {
+ i++;
+ continue;
+ }
+ /* Fill the run of @n NULL entries at @i. */
+ for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
+ ;
+ got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+ want_init_on_alloc(gfp), frozen);
+ i += got;
+ if (got < n)
+ break;
+ }
+ return i;
+
+skip:
+ /* Declined: report the populated prefix, as alloc_pages_bulk_node() does. */
+ while (i < nr_pages && page_array[i])
+ i++;
+ return i;
+}
+
+
/**
* __pisa_free_page - Recycle a freed block back into its PISA allocator
* @page: Block being freed (called from __free_pages_prepare() under RCU)
@@ -372,8 +690,35 @@ bool __pisa_free_page(struct page *page)
static int __init pisa_init(void)
{
+ const struct {
+ u8 order;
+ bool ro;
+ } cfg[] = {
+ { 0, false },
+ { 2, true },
+ { SKB_FRAG_PAGE_ORDER, true },
+ };
+ int nid, i;
+
pisa_node_init();
+ /* Preallocate the supported per-node allocators. */
+ for_each_node_state(nid, N_MEMORY) {
+ for (i = 0; i < ARRAY_SIZE(cfg); i++) {
+ unsigned int order = cfg[i].order;
+ bool ro = cfg[i].ro;
+ struct pisa_alloc *pa;
+
+ if (order > PISA_ORDER || pisa_nodes[nid].pa[ro][order])
+ continue;
+ pa = pisa_create(nid, order, ro ? DMA_TO_DEVICE : DMA_BIDIRECTIONAL);
+ if (!pa)
+ continue;
+ pisa_set_name(pa, "pisa%s_o%u_n%d", ro ? "_ro" : "", order, nid);
+ pisa_nodes[nid].pa[ro][order] = pa;
+ }
+ }
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 079f6220e742a..bdc52e6497b05 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,8 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/mm.h>
+#include <linux/skbuff.h>
+#include <net/sock.h>
#include "pisa-priv.h"
@@ -28,6 +30,7 @@ static void test_meta_init_and_roundtrip(struct kunit *test)
KUNIT_ASSERT_EQ(test, pisa_meta_ensure_pfn(pfn, true), 0);
m_pfn = pisa_meta_of_pfn(pfn);
+ KUNIT_EXPECT_PTR_EQ(test, pisa_meta_of_pfn(base_pfn), m_pfn);
KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
}
@@ -37,7 +40,7 @@ static void test_meta_invalid_phys(struct kunit *test)
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
- meta = pisa_meta_of_pfn(PHYS_PFN(PHYS_ADDR_MAX));
+ meta = pisa_meta_of_pfn(ULONG_MAX >> PAGE_SHIFT);
KUNIT_ASSERT_NOT_NULL(test, meta);
KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(meta), ULONG_MAX);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(ULONG_MAX >> PAGE_SHIFT));
@@ -87,9 +90,78 @@ static void test_meta_active_toggle(struct kunit *test)
static void test_alloc_and_recycle(struct kunit *test)
{
- KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, 0, DMA_NONE));
- KUNIT_EXPECT_NULL(test, pisa_create(NUMA_NO_NODE, PISA_ORDER + 1, DMA_BIDIRECTIONAL));
+ const struct {
+ unsigned int order;
+ bool ro;
+ } cases[] = {
+ { 0, false },
+ { 2, true },
+ { SKB_FRAG_PAGE_ORDER, true },
+ };
+ struct page *p1, *p2, *bulk[3];
+ struct pisa_alloc *pa;
+ unsigned int i;
+
+ KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, 0, DMA_NONE));
+ KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, PISA_ORDER + 1,
+ DMA_BIDIRECTIONAL));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 1, false));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 0, true));
+ KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, PISA_ORDER + 1, false));
pisa_set_name(NULL, "ignored");
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ unsigned int order = cases[i].order;
+
+ pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
+
+ p1 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p1));
+
+ /* Freeing p1 returns it to the allocator; next allocation reuses index 0. */
+ __free_pages(p1, order);
+ p2 = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p2);
+ if (!want_init_on_alloc(GFP_KERNEL) || want_init_on_free() || order == PISA_ORDER)
+ KUNIT_EXPECT_PTR_EQ(test, p1, p2);
+
+ if (order == 0) {
+ /* Bulk alloc skips already populated entries. */
+ bulk[0] = NULL;
+ bulk[1] = p2;
+ bulk[2] = NULL;
+ KUNIT_EXPECT_EQ(test,
+ pisa_alloc_bulk(pa, GFP_KERNEL, 3, bulk, false),
+ 3UL);
+ KUNIT_EXPECT_PTR_EQ(test, bulk[1], p2);
+ KUNIT_EXPECT_NOT_NULL(test, bulk[0]);
+ KUNIT_EXPECT_NOT_NULL(test, bulk[2]);
+ /* Declined GFP reports populated prefix length. */
+ KUNIT_EXPECT_EQ(test,
+ pisa_alloc_bulk(pa,
+ GFP_KERNEL | __GFP_ACCOUNT,
+ 3, bulk, false),
+ 3UL);
+ if (bulk[0])
+ __free_pages(bulk[0], 0);
+ if (bulk[2])
+ __free_pages(bulk[2], 0);
+ }
+
+ if (p2)
+ __free_pages(p2, order);
+ }
+
+ /* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
+ p1 = alloc_page(GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
+ __free_pages(p1, 0);
}
static void test_reservoir_and_scrubber(struct kunit *test)
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 0d097c665c790..f15934ccdeea4 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -100,6 +100,7 @@ struct pisa_alloc {
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
+ struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -114,11 +115,6 @@ int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
-struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
-__printf(2, 3)
-void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
-void pisa_release_page(struct pisa_meta *meta);
-
static inline int pisa_sanitize_nid(int nid)
{
if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
@@ -135,6 +131,24 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
+/* pisa-alloc.c, also used by the per-node code in pisa-node.c */
+struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
+struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro);
+__printf(2, 3)
+void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+void pisa_release_page(struct pisa_meta *meta);
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+ unsigned long nr_pages, struct page **page_array, bool frozen);
+
+static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
+{
+ struct page *page = NULL;
+
+ if (pa)
+ pisa_alloc_bulk(pa, gfp, 1, &page, false);
+ return page;
+}
+
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h
index c1331bf0b2398..65fbfca3e08d1 100644
--- a/include/linux/gfp_types.h
+++ b/include/linux/gfp_types.h
@@ -33,7 +33,7 @@ enum {
___GFP_IO_BIT,
___GFP_FS_BIT,
___GFP_ZERO_BIT,
- ___GFP_UNUSED_BIT, /* 0x200u unused */
+ ___GFP_PISA_RO_BIT,
___GFP_DIRECT_RECLAIM_BIT,
___GFP_KSWAPD_RECLAIM_BIT,
___GFP_WRITE_BIT,
@@ -68,7 +68,11 @@ enum {
#define ___GFP_IO BIT(___GFP_IO_BIT)
#define ___GFP_FS BIT(___GFP_FS_BIT)
#define ___GFP_ZERO BIT(___GFP_ZERO_BIT)
-/* 0x200u unused */
+#ifdef CONFIG_PISA
+#define ___GFP_PISA_RO BIT(___GFP_PISA_RO_BIT)
+#else
+#define ___GFP_PISA_RO 0
+#endif
#define ___GFP_DIRECT_RECLAIM BIT(___GFP_DIRECT_RECLAIM_BIT)
#define ___GFP_KSWAPD_RECLAIM BIT(___GFP_KSWAPD_RECLAIM_BIT)
#define ___GFP_WRITE BIT(___GFP_WRITE_BIT)
@@ -95,6 +99,11 @@ enum {
#else
#define ___GFP_NOLOCKDEP 0
#endif
+#ifdef CONFIG_PISA
+#define ___GFP_PISA_RW (___GFP_PISA_RO | ___GFP_WRITE)
+#else
+#define ___GFP_PISA_RW 0
+#endif
/*
* Physical address zone modifiers (see linux/mmzone.h - low four bits)
@@ -289,6 +298,10 @@ enum {
/* Disable lockdep for GFP context tracking */
#define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
+/* Allocate from per-node PISA allocators (fall back to buddy on failure) */
+#define __GFP_PISA_RO ((__force gfp_t)___GFP_PISA_RO)
+#define __GFP_PISA_RW ((__force gfp_t)___GFP_PISA_RW)
+
/* Room for N __GFP_FOO bits */
#define __GFP_BITS_SHIFT ___GFP_LAST_BIT
#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index dddcedb8c93f6..18ba1c83bd6d3 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -7,6 +7,7 @@
#include <linux/bitops.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
+#include <linux/numa.h>
#include <linux/rcupdate.h>
#ifdef CONFIG_PISA
@@ -72,6 +73,11 @@ static inline bool pisa_free_page(struct page *page)
return ret;
}
+unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
+ int preferred_nid, nodemask_t *nodemask,
+ int nr_pages, struct page **page_array,
+ bool frozen);
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -91,8 +97,26 @@ static inline bool pisa_free_page(struct page *page)
return false;
}
+static inline unsigned long
+pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array, bool frozen)
+{
+ return 0;
+}
+
#endif /* CONFIG_PISA */
+static inline struct page *
+pisa_alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask)
+{
+ struct page *page = NULL;
+
+ pisa_alloc_pages_bulk(gfp, order, preferred_nid, nodemask, 1, &page, true);
+ return page;
+}
+
static inline bool is_pisa_phys(phys_addr_t phys)
{
return is_pisa_pfn(phys >> PAGE_SHIFT);
@@ -102,4 +126,5 @@ static inline bool is_pisa_page(const struct page *page)
{
return is_pisa_pfn(page_to_pfn(page));
}
+
#endif /* _LINUX_PISA_H */
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 3c153b3ad8450..65331bd8f0315 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -54,10 +54,18 @@
# define TRACE_GFP_FLAGS_LOCKDEP
#endif
+#ifdef CONFIG_PISA
+# define TRACE_GFP_FLAGS_PISA \
+ TRACE_GFP_EM(PISA_RO)
+#else
+# define TRACE_GFP_FLAGS_PISA
+#endif
+
#define TRACE_GFP_FLAGS \
TRACE_GFP_FLAGS_GENERAL \
TRACE_GFP_FLAGS_KASAN \
- TRACE_GFP_FLAGS_LOCKDEP
+ TRACE_GFP_FLAGS_LOCKDEP \
+ TRACE_GFP_FLAGS_PISA
#undef TRACE_GFP_EM
#define TRACE_GFP_EM(a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
@@ -65,7 +73,6 @@
TRACE_GFP_FLAGS
/* Just in case these are ever used */
-TRACE_DEFINE_ENUM(___GFP_UNUSED_BIT);
TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
#define gfpflag_string(flag) {(__force unsigned long)flag, #flag}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 48df87a19d312..b8e1f0fcb3f41 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1302,17 +1302,40 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr)
/*
* pisa_free_page() in __free_pages_prepare() recycles a PISA block into
* its allocator before the rest of that function runs. __pisa_free_page() calls
- * this to apply the buddy allocator's checks.
+ * this to apply the buddy allocator's checks, and to clear the order-0 state of
+ * blocks that were user pages (see drivers/iommu/pisa-user.c).
*
* Return: false if the block is in a bad state (already reported): leak it.
*/
bool free_page_prepare_pisa(struct page *page, unsigned int order)
{
+ struct folio *folio;
+ unsigned int i;
+
if (!IS_ENABLED(CONFIG_PISA))
return true;
+
+ folio = page_folio(page);
+
+ if (!order) {
+ if (unlikely(folio_test_mlocked(folio))) {
+ __folio_clear_mlocked(folio);
+ zone_stat_mod_folio(folio, NR_MLOCK, -1);
+ count_vm_events(UNEVICTABLE_PGCLEARED, 1);
+ }
+ folio->mapping = NULL;
+ }
if (is_check_pages_enabled() && free_page_is_bad(page))
return false;
page_table_check_free(page, order);
+ for (i = 0; i < (1U << order); i++)
+ page_kasan_tag_reset(page + i);
+ kasan_unpoison_range(page_address(page), PAGE_SIZE << order);
+ if (!order) {
+ page_cpupid_reset_last(page);
+ page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+ page->private = 0;
+ }
return true;
}
@@ -5223,6 +5246,14 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
while (nr_populated < nr_pages && page_array[nr_populated])
nr_populated++;
+ if (unlikely(gfp & __GFP_PISA_RO)) {
+ nr_populated = pisa_alloc_pages_bulk(gfp, 0, preferred_nid, nodemask,
+ nr_pages, page_array, false);
+ gfp &= ~__GFP_PISA_RW;
+ if (nr_populated == nr_pages)
+ goto out;
+ }
+
/* No pages requested? */
if (unlikely(nr_pages <= 0))
goto out;
@@ -5457,6 +5488,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
fastpath_alloc_flags |= ALLOC_WMARK_LOW;
}
+ if (unlikely(gfp & __GFP_PISA_RO)) {
+ page = pisa_alloc_pages(gfp, order, preferred_nid, nodemask);
+ if (page)
+ return page;
+ gfp &= ~__GFP_PISA_RW;
+ }
+
gfp &= gfp_allowed_mask;
/*
* Apply scoped allocation constraints. This is mainly about GFP_NOFS
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (4 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
` (19 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Add two mechanisms to bound the number of unused PISA pages:
- a global ceiling, pisa.max_pages, on the number of PISA pages pinned
across all allocators and reservoirs, defaulting to one eighth of RAM,
with a floor for small machines.
- a shrinker that, under reclaim, drains the per-node reservoirs first
and then completely idle pages across all allocators.
Idle pages are detached under the allocator lock but retired outside
it, to avoid holding the lock during the expensive unmaps/iotlb flush.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 91 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 15 +++++++
drivers/iommu/pisa-node.c | 25 +++++++++++
drivers/iommu/pisa-priv.h | 15 +++++++
4 files changed, 146 insertions(+)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 1f1b99023094c..de8009fd6fff3 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -4,6 +4,7 @@
* See Documentation/core-api/pisa.rst for the architecture overview.
*/
+#include <kunit/visibility.h>
#include <linux/atomic.h>
#include <linux/bitmap.h>
#include <linux/bitops.h>
@@ -22,6 +23,7 @@
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
+#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
@@ -40,6 +42,11 @@ static LLIST_HEAD(pisa_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *pisa_wq __ro_after_init;
+/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */
+atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp;
+unsigned long pisa_max_pages __read_mostly;
+module_param_named(max_pages, pisa_max_pages, ulong, 0644);
+
/* Max completely idle PISA pages kept per allocator before async release. */
static unsigned int pisa_max_idle __read_mostly = 1;
module_param_named(max_idle, pisa_max_idle, uint, 0644);
@@ -81,6 +88,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
WRITE_ONCE(meta->pa, NULL);
+ atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
page_ref_unfreeze(block, 1);
@@ -93,6 +101,7 @@ void pisa_release_page(struct pisa_meta *meta)
{
if (!meta->pa) {
__free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
+ atomic_long_dec(&pisa_nr_pages);
return;
}
@@ -688,6 +697,72 @@ bool __pisa_free_page(struct page *page)
return true;
}
+/* Shrinker: releases reservoir pages first, then idle allocator pages. */
+static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+ unsigned long nr = pisa_reservoir_count();
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node)
+ nr += READ_ONCE(pa->num_idle_pages);
+ read_unlock(&pisa_allocs_lock);
+
+ return nr << PISA_ORDER;
+}
+
+/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+ unsigned int *nr, struct list_head *release_list,
+ unsigned long nr_to_scan)
+{
+ unsigned long freed = 0, flags;
+
+ spin_lock_irqsave(lock, flags);
+ while (freed < nr_to_scan && !list_empty(list)) {
+ list_move(list->next, release_list);
+ (*nr)--;
+ freed += 1UL << PISA_ORDER;
+ }
+ spin_unlock_irqrestore(lock, flags);
+
+ return freed;
+}
+
+VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+ struct pisa_meta *meta, *tmp;
+ struct pisa_alloc *pa;
+ unsigned long freed;
+ LIST_HEAD(release_list);
+
+ freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+
+ if (freed < sc->nr_to_scan) {
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ freed += pisa_shrink_list(&pa->lock, &pa->idle,
+ &pa->num_idle_pages, &release_list,
+ sc->nr_to_scan - freed);
+
+ if (freed >= sc->nr_to_scan)
+ break;
+ }
+ read_unlock(&pisa_allocs_lock);
+ }
+
+ /* Retiring must not run under @pa->lock. */
+ list_for_each_entry_safe(meta, tmp, &release_list, list) {
+ list_del_init(&meta->list);
+ pisa_release_page(meta);
+ }
+
+ /* Charge do_shrink_slab() for the full PISA pages reclaimed. */
+ sc->nr_scanned = freed;
+
+ return freed ?: SHRINK_STOP;
+}
+
static int __init pisa_init(void)
{
const struct {
@@ -698,10 +773,15 @@ static int __init pisa_init(void)
{ 2, true },
{ SKB_FRAG_PAGE_ORDER, true },
};
+ struct shrinker *shrink;
int nid, i;
pisa_node_init();
+ /* Default ceiling is 1/8th of RAM (floor of 16 PISA pages). */
+ if (!pisa_max_pages)
+ pisa_max_pages = max(totalram_pages() >> (PISA_ORDER + 3), 16UL);
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
@@ -724,6 +804,17 @@ static int __init pisa_init(void)
if (!pisa_wq)
pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+ shrink = shrinker_alloc(0, "pisa");
+ if (!shrink) {
+ pr_warn("pisa: shrinker registration failed\n");
+ return 0;
+ }
+
+ shrink->count_objects = pisa_shrink_count;
+ shrink->scan_objects = pisa_shrink_scan;
+ shrink->seeks = DEFAULT_SEEKS;
+ shrinker_register(shrink);
+
return 0;
}
subsys_initcall(pisa_init);
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index bdc52e6497b05..96bc8dbc38fb4 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,7 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/mm.h>
+#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <net/sock.h>
@@ -98,8 +99,10 @@ static void test_alloc_and_recycle(struct kunit *test)
{ 2, true },
{ SKB_FRAG_PAGE_ORDER, true },
};
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct page *p1, *p2, *bulk[3];
struct pisa_alloc *pa;
+ unsigned long old_max;
unsigned int i;
KUNIT_EXPECT_NULL(test, pisa_create(first_memory_node, 0, DMA_NONE));
@@ -115,6 +118,7 @@ static void test_alloc_and_recycle(struct kunit *test)
pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
KUNIT_ASSERT_NOT_NULL(test, pa);
+ pisa_shrink_scan(NULL, &sc);
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
@@ -155,6 +159,7 @@ static void test_alloc_and_recycle(struct kunit *test)
if (p2)
__free_pages(p2, order);
+ pisa_shrink_scan(NULL, &sc);
}
/* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
@@ -162,6 +167,16 @@ static void test_alloc_and_recycle(struct kunit *test)
KUNIT_ASSERT_NOT_NULL(test, p1);
KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
__free_pages(p1, 0);
+
+ /* Global budget cap (pisa_max_pages = 0) refuses new PISA pages. */
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ old_max = READ_ONCE(pisa_max_pages);
+ WRITE_ONCE(pisa_max_pages, 0);
+ WRITE_ONCE(pisa_reservoir_max, 0);
+ KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL));
+ WRITE_ONCE(pisa_reservoir_max, 16);
+ WRITE_ONCE(pisa_max_pages, old_max);
}
static void test_reservoir_and_scrubber(struct kunit *test)
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 524f0d4457ad7..0f23b94a7a6f8 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -74,6 +74,10 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
return ERR_PTR(-EAGAIN);
}
+ /* Racy check against global ceiling across allocators and reservoirs. */
+ if (atomic_long_read(&pisa_nr_pages) >= READ_ONCE(pisa_max_pages))
+ return ERR_PTR(-ENOSPC);
+
/*
* Long-lived active pages must be unmovable, lowmem, non-compound, and
* must not dip into emergency reserves or trigger OOM/warnings.
@@ -102,6 +106,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
meta = pisa_meta_of_pfn(page_to_pfn(page));
memset(meta, 0, sizeof(*meta));
+
+ atomic_long_inc(&pisa_nr_pages);
*clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
return meta;
@@ -256,6 +262,25 @@ unsigned long pisa_reservoir_count(void)
return nr;
}
+/* Detach up to @nr_to_scan base pages from reservoirs onto @release_list. */
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan)
+{
+ unsigned long freed = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ if (!READ_ONCE(res->nr_pages))
+ continue;
+ freed += pisa_shrink_list(&res->lock, &res->pages, &res->nr_pages,
+ release_list, nr_to_scan - freed);
+ if (freed >= nr_to_scan)
+ break;
+ }
+ return freed;
+}
+
void __init pisa_node_init(void)
{
int nid;
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f15934ccdeea4..d5ea83745ba47 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -115,6 +115,16 @@ int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
+struct shrink_control;
+struct shrinker;
+#if IS_ENABLED(CONFIG_KUNIT)
+unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+#endif
+
+/* Global PISA page budget across all allocators and per-node reservoirs. */
+extern atomic_long_t pisa_nr_pages;
+extern unsigned long pisa_max_pages;
+
static inline int pisa_sanitize_nid(int nid)
{
if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
@@ -149,12 +159,17 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
return page;
}
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+ unsigned int *nr, struct list_head *release_list,
+ unsigned long nr_to_scan);
+
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
unsigned long pisa_reservoir_count(void);
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
#endif /* CONFIG_PISA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window for PISA pages
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (5 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
` (18 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA buffers need to be quickly identified from their iova, and may need
to be mapped in multiple domains. We can achieve both by reserving an IOVA
range for PISA pages in each domain.
Add per-domain PISA IOVA window descriptors in struct iommu_dma_cookie
and a global slot allocator (pisa_slot_alloc() / pisa_slot_free()).
The window is sized to twice the maximum allowed PISA buffer space,
defaulting to max(64MB, 1/4 of physical RAM).
A buffer is mapped at the same slot offset in all windows for the domains
that need to access it. IOVA allocation is done with a bitmap (1 bit
per 2MB buffer, hence a 4KB bitmap supports 64GB of buffer space) at
negligible cost.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 3 ++
drivers/iommu/dma-iommu.c | 18 ++++++++
drivers/iommu/dma-iommu.h | 16 +++++++
drivers/iommu/pisa-alloc.c | 12 ++++-
drivers/iommu/pisa-kunit.c | 24 ++++++++++
drivers/iommu/pisa-map.c | 90 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 58 ++++++++++++++++++++++++
7 files changed, 220 insertions(+), 1 deletion(-)
create mode 100644 drivers/iommu/pisa-map.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index bc58166f9303a..125cf7e813cdc 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -12,6 +12,9 @@ obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
+ifeq ($(CONFIG_IOMMU_DMA),y)
+obj-$(CONFIG_PISA) += pisa-map.o
+endif
obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 78742aeba932d..fba6084feac81 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -36,6 +36,7 @@
#include <trace/events/swiotlb.h>
#include "dma-iommu.h"
+#include "pisa-priv.h"
#include "iommu-pages.h"
#include "iommu-priv.h"
@@ -76,6 +77,8 @@ struct iommu_dma_cookie {
struct iommu_domain *fq_domain;
/* Options for dma-iommu use */
struct iommu_dma_options options;
+ /* IOVA window reserved for PISA pages. Nothing else can go here. */
+ struct pisa_window win_pisa;
};
struct iommu_dma_msi_cookie {
@@ -1215,6 +1218,21 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,
return iova_offset(iovad, phys | size);
}
+/* Return @domain's PISA window, or NULL if @domain cannot use PISA. */
+struct pisa_window *pisa_dma_window(struct iommu_domain *domain)
+{
+ if (static_branch_unlikely(&iommu_deferred_attach_enabled) ||
+ !domain || domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
+ return NULL;
+
+ return &domain->iova_cookie->win_pisa;
+}
+
+struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain)
+{
+ return pisa_dma_window(domain) ? &domain->iova_cookie->iovad : NULL;
+}
+
dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
enum dma_data_direction dir, unsigned long attrs)
{
diff --git a/drivers/iommu/dma-iommu.h b/drivers/iommu/dma-iommu.h
index 040d002525632..f112b47396984 100644
--- a/drivers/iommu/dma-iommu.h
+++ b/drivers/iommu/dma-iommu.h
@@ -7,6 +7,9 @@
#include <linux/iommu.h>
+struct pisa_window;
+struct iova_domain;
+
#ifdef CONFIG_IOMMU_DMA
void iommu_setup_dma_ops(struct device *dev, struct iommu_domain *domain);
@@ -22,6 +25,9 @@ void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list);
int iommu_dma_sw_msi(struct iommu_domain *domain, struct msi_desc *desc,
phys_addr_t msi_addr);
+struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain);
+struct pisa_window *pisa_dma_window(struct iommu_domain *domain);
+
extern bool iommu_dma_forcedac;
#else /* CONFIG_IOMMU_DMA */
@@ -59,5 +65,15 @@ static inline int iommu_dma_sw_msi(struct iommu_domain *domain,
return -ENODEV;
}
+static inline struct iova_domain *pisa_dma_iovad(struct iommu_domain *domain)
+{
+ return NULL;
+}
+
+static inline struct pisa_window *pisa_dma_window(struct iommu_domain *domain)
+{
+ return NULL;
+}
+
#endif /* CONFIG_IOMMU_DMA */
#endif /* __DMA_IOMMU_H */
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index de8009fd6fff3..d7b73c3a6c0b6 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -84,10 +84,12 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Read @meta before freeing blocks; the frame can be reused immediately. */
struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
- unsigned int i, order = meta->pa->order;
+ unsigned int i, order = meta->pa->order, slot = meta->slot;
WRITE_ONCE(meta->pa, NULL);
+ /* The page has been unmapped from every domain by pisa_release_page() */
+ pisa_slot_free(slot);
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -426,10 +428,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
if (IS_ERR(meta))
return meta;
+ if (unlikely(pisa_slot_alloc(meta))) {
+ pisa_release_page(meta);
+ return ERR_PTR(-ENOSPC);
+ }
+
page = pfn_to_page(pisa_meta_to_pfn(meta));
/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
if (split_page_compound(page, PISA_ORDER, order)) {
+ pisa_slot_free(meta->slot);
pisa_release_page(meta);
pisa_alloc_backoff(nid, can_block);
return ERR_PTR(-EBUSY);
@@ -782,6 +790,8 @@ static int __init pisa_init(void)
if (!pisa_max_pages)
pisa_max_pages = max(totalram_pages() >> (PISA_ORDER + 3), 16UL);
+ pisa_slots_init();
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 96bc8dbc38fb4..5a2e806e74654 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -10,6 +10,7 @@
#include <linux/skbuff.h>
#include <net/sock.h>
+#include "dma-iommu.h"
#include "pisa-priv.h"
static void test_meta_init_and_roundtrip(struct kunit *test)
@@ -179,6 +180,26 @@ static void test_alloc_and_recycle(struct kunit *test)
WRITE_ONCE(pisa_max_pages, old_max);
}
+#ifdef CONFIG_IOMMU_DMA
+static void test_window_helpers(struct kunit *test)
+{
+ struct pisa_window win = {};
+
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+ /* Empty window (base == 0) never owns any IOVA. */
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, 0));
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G));
+
+ win.base = SZ_4G;
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G - 1));
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G));
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size - 1));
+ KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size));
+}
+
+#endif /* CONFIG_IOMMU_DMA */
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -190,6 +211,9 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_invalid_phys),
KUNIT_CASE(test_meta_active_toggle),
KUNIT_CASE(test_alloc_and_recycle),
+#ifdef CONFIG_IOMMU_DMA
+ KUNIT_CASE(test_window_helpers),
+#endif
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
new file mode 100644
index 0000000000000..2e8d635dec34c
--- /dev/null
+++ b/drivers/iommu/pisa-map.c
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-domain IOVA window management and IOMMU mapping.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/pisa.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+
+#include "dma-iommu.h"
+#include "pisa-priv.h"
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Optional module param override for pisa_nr_slots (0 = auto: 2 * max_pages). */
+static unsigned int pisa_cfg_slots __read_mostly;
+module_param_named(slots, pisa_cfg_slots, uint, 0444);
+
+/*
+ * Number of 2MB IOVA slots in each domain's window (computed at init from
+ * pisa_cfg_slots or 2 * max_pages). Each active PISA page holds one slot
+ * in [0, pisa_nr_slots) and maps at slot * PISA_SIZE in every domain's window.
+ */
+unsigned int pisa_nr_slots __read_mostly;
+u64 pisa_window_size __read_mostly;
+/* One bit per IOVA slot, set while an active PISA page holds the slot. */
+static unsigned long *pisa_slot_bitmap __read_mostly;
+/* Metadata of the PISA page backing each active slot, or NULL. */
+static struct pisa_meta **pisa_slot2meta __read_mostly;
+static unsigned int pisa_slot_hint; /* no free slot below this */
+/* Protects pisa_slot_bitmap and pisa_slot_hint. */
+static DEFINE_SPINLOCK(pisa_slot_lock);
+
+/* Cost is typically low though the bitmap can be up to O(100K) bits on large systems. */
+int pisa_slot_alloc(struct pisa_meta *meta)
+{
+ unsigned long flags;
+ unsigned int slot;
+
+ spin_lock_irqsave(&pisa_slot_lock, flags);
+ slot = find_next_zero_bit(pisa_slot_bitmap, pisa_nr_slots, pisa_slot_hint);
+ if (slot < pisa_nr_slots) {
+ __set_bit(slot, pisa_slot_bitmap);
+ pisa_slot_hint = slot + 1;
+ WRITE_ONCE(meta->slot, slot);
+ WRITE_ONCE(pisa_slot2meta[slot], meta);
+ }
+ spin_unlock_irqrestore(&pisa_slot_lock, flags);
+
+ if (slot >= pisa_nr_slots)
+ return -ENOSPC;
+ return 0;
+}
+
+/* Return @slot once no domain can translate it any more. */
+void pisa_slot_free(unsigned int slot)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&pisa_slot_lock, flags);
+ WRITE_ONCE(pisa_slot2meta[slot], NULL);
+ __clear_bit(slot, pisa_slot_bitmap);
+ if (slot < pisa_slot_hint)
+ pisa_slot_hint = slot;
+ spin_unlock_irqrestore(&pisa_slot_lock, flags);
+}
+
+void __init pisa_slots_init(void)
+{
+ /* Fixed at boot: sizes every domain's window (default 2 * max_pages). */
+ pisa_nr_slots = pisa_cfg_slots ?: min_t(unsigned long, 2 * pisa_max_pages, UINT_MAX >> 1);
+ pisa_slot_bitmap = kvcalloc(BITS_TO_LONGS(pisa_nr_slots),
+ sizeof(unsigned long), GFP_KERNEL);
+ pisa_slot2meta = kvcalloc(pisa_nr_slots, sizeof(*pisa_slot2meta), GFP_KERNEL);
+ if (!pisa_slot_bitmap || !pisa_slot2meta) {
+ kvfree(pisa_slot_bitmap);
+ kvfree(pisa_slot2meta);
+ pisa_slot_bitmap = NULL;
+ pisa_slot2meta = NULL;
+ pisa_nr_slots = 0;
+ }
+ /* Size of every domain's window: one 2MB page per slot. */
+ pisa_window_size = (u64)pisa_nr_slots << PISA_SHIFT;
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index d5ea83745ba47..3ef6e51402ceb 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -13,12 +13,35 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+/**
+ * struct pisa_window - A domain's IOVA window reserved for PISA pages
+ * @base: Window base IOVA (PISA_SIZE aligned), or 0 if unassigned
+ * @domain_idx: Dense index + PISA_IDX_FIRST, or PISA_IDX_* sentinel while @base == 0
+ * @idx_gen: Value of pisa_idx_gen when @domain_idx was set to NOIDX
+ *
+ * Lives in iommu_dma_cookie. Active PISA pages map at @base + slot * PISA_SIZE.
+ */
+struct pisa_window {
+ dma_addr_t base;
+ u16 domain_idx;
+ u32 idx_gen;
+};
+
#ifdef CONFIG_PISA
#define PISA_MAX_BLOCKS 512U
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/* Sentinel values of pisa_window.domain_idx below PISA_IDX_FIRST. */
+enum {
+ PISA_IDX_NONE, /* not attempted yet */
+ PISA_IDX_NOHUGE, /* @domain can never use PISA */
+ PISA_IDX_NOWINDOW, /* no IOVA range for the window */
+ PISA_IDX_NOIDX, /* all domain indices were in use */
+ PISA_IDX_FIRST, /* first valid domain index */
+};
+
/*
* Lock hierarchy:
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
@@ -32,6 +55,7 @@
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
+ * @slot: Dense IOVA slot index in every domain's window
* @pa: Owning pisa_alloc (NULL for reservoir pages)
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
@@ -44,6 +68,7 @@ struct pisa_meta {
u16 nr_clean;
u16 nr_dirty;
u8 on_list;
+ u32 slot;
struct pisa_alloc *pa;
struct list_head list;
@@ -172,4 +197,37 @@ unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
#endif /* CONFIG_PISA */
+
+#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
+
+/* Size in bytes of every domain's PISA IOVA window, fixed at boot. */
+extern u64 pisa_window_size;
+extern unsigned int pisa_nr_slots;
+
+void pisa_slots_init(void);
+int pisa_slot_alloc(struct pisa_meta *meta);
+void pisa_slot_free(unsigned int slot);
+
+/* Return true if @dma falls within @win's reserved PISA IOVA window. */
+static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
+{
+ /* Acquire pairs with the release of @base in pisa_window_assign(). */
+ dma_addr_t base = smp_load_acquire(&win->base);
+
+ return base && dma - base < pisa_window_size;
+}
+
+#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
+
+#define pisa_nr_slots 0U
+static inline void pisa_slots_init(void) {}
+static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
+static inline void pisa_slot_free(unsigned int slot) {}
+
+static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
+{
+ return false;
+}
+
+#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (6 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
` (17 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Each PISA page can be mapped by multiple IOMMU domains.
Add pisa_domain_release() (called from iommu_put_dma_cookie()) to drop
a dying domain from the registry, clear its bit across all active PISA
pages, and wait on SRCU before reusing the domain index. If iommu_unmap()
fails on page release, mark the 2MB page PISA_QUARANTINED and leak it
so a device can never access reallocated memory.
When a group's last DMA API user detaches
(iommu_device_unuse_default_domain() with owner_cnt reaching 0), call
pisa_domain_quiesce() to unmap all active PISA pages from the default
domain and flush the IOTLB while keeping the domain's window and index
for future demand mappings.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-iommu.c | 3 +
drivers/iommu/iommu.c | 5 +
drivers/iommu/pisa-alloc.c | 14 ++-
drivers/iommu/pisa-kunit.c | 18 ++++
drivers/iommu/pisa-map.c | 214 ++++++++++++++++++++++++++++++++++++-
drivers/iommu/pisa-priv.h | 56 +++++++++-
include/linux/pisa.h | 1 +
7 files changed, 306 insertions(+), 5 deletions(-)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index fba6084feac81..025bceebb2262 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -430,6 +430,9 @@ void iommu_put_dma_cookie(struct iommu_domain *domain)
struct iommu_dma_cookie *cookie = domain->iova_cookie;
struct iommu_dma_msi_page *msi, *tmp;
+ /* Drop cached PISA mappings before the IOVA domain goes away. */
+ pisa_domain_release(domain);
+
if (cookie->iovad.granule) {
iommu_dma_free_fq(cookie);
put_iova_domain(&cookie->iovad);
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b486b8bbd1fcd..dfba3d5c277c2 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -37,6 +37,7 @@
#include <linux/generic_pt/iommu.h>
#include "dma-iommu.h"
+#include "pisa-priv.h"
#include "iommu-priv.h"
static struct kset *iommu_group_kset;
@@ -3363,6 +3364,10 @@ void iommu_device_unuse_default_domain(struct device *dev)
if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
group->owner_cnt--;
+ /* Unmap persistent PISA windows when the last driver detaches. */
+ if (!group->owner_cnt)
+ pisa_domain_quiesce(group->default_domain);
+
mutex_unlock(&group->mutex);
}
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index d7b73c3a6c0b6..dd90aad296edf 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -13,6 +13,7 @@
#include <linux/pisa.h>
#include <linux/export.h>
#include <linux/gfp.h>
+#include <linux/iommu.h>
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/memcontrol.h>
@@ -30,6 +31,7 @@
#include <linux/workqueue.h>
#include <net/sock.h>
+#include "dma-iommu.h"
#include "pisa-priv.h"
/* Parameters are pisa.<name> at boot and in /sys/module/pisa/parameters/. */
@@ -88,6 +90,10 @@ static void pisa_release_page_rcu(struct rcu_head *head)
WRITE_ONCE(meta->pa, NULL);
+ /* Possibly still mapped by a device: leak it, still charged. */
+ if (meta->flags & PISA_QUARANTINED)
+ return;
+
/* The page has been unmapped from every domain by pisa_release_page() */
pisa_slot_free(slot);
atomic_long_dec(&pisa_nr_pages);
@@ -107,7 +113,8 @@ void pisa_release_page(struct pisa_meta *meta)
return;
}
- /* pisa_unmap_all(meta) will be called here once IOMMU mappings are added. */
+ /* Quarantined on unmap failure and leaked in pisa_release_page_rcu(). */
+ pisa_unmap_all(meta);
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
@@ -194,7 +201,8 @@ static bool pisa_place_meta(struct pisa_meta *meta)
lockdep_assert_held(&pa->lock);
if (avail == bpp) {
- if (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms)) {
+ if ((meta->flags & PISA_QUARANTINED) ||
+ (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms))) {
list_del_init(&meta->list);
WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
llist_add(&meta->llnode, &pisa_free_list);
@@ -204,7 +212,7 @@ static bool pisa_place_meta(struct pisa_meta *meta)
WRITE_ONCE(pa->idle_jiffies, jiffies);
pa->num_idle_pages++;
on_list = meta->nr_dirty ? PISA_LIST_IDLE_DIRTY : PISA_LIST_IDLE_CLEAN;
- } else if (!avail) {
+ } else if (!avail || unlikely(meta->flags & PISA_QUARANTINED)) {
on_list = PISA_LIST_FULL;
} else {
bool sparse;
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5a2e806e74654..f0775aa45a058 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -5,6 +5,7 @@
#include <kunit/test.h>
#include <linux/gfp.h>
#include <linux/pisa.h>
+#include <linux/iommu.h>
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -183,6 +184,7 @@ static void test_alloc_and_recycle(struct kunit *test)
#ifdef CONFIG_IOMMU_DMA
static void test_window_helpers(struct kunit *test)
{
+ struct iommu_domain dummy_domain = {};
struct pisa_window win = {};
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -196,6 +198,22 @@ static void test_window_helpers(struct kunit *test)
KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G));
KUNIT_EXPECT_TRUE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size - 1));
KUNIT_EXPECT_FALSE(test, is_pisa_iova(&win, SZ_4G + pisa_window_size));
+
+ /* Sentinel values in pisa_window_declined(). */
+ win.domain_idx = PISA_IDX_NONE;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+ win.domain_idx = PISA_IDX_NOHUGE;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -EOPNOTSUPP);
+ win.domain_idx = PISA_IDX_NOWINDOW;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -ENOSPC);
+ win.domain_idx = PISA_IDX_NOIDX;
+ win.idx_gen = READ_ONCE(pisa_idx_gen);
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), -EBUSY);
+ win.idx_gen--;
+ KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+
+ /* Releasing an unregistered domain with an active allocator is a safe no-op. */
+ pisa_domain_release(&dummy_domain);
}
#endif /* CONFIG_IOMMU_DMA */
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2e8d635dec34c..2ba977bb68734 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -4,13 +4,19 @@
* See Documentation/core-api/pisa.rst for the architecture overview.
*/
+#include <linux/atomic.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/dma-map-ops.h>
+#include <linux/dma-mapping.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/iommu.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
-#include <linux/pisa.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <linux/srcu.h>
#include "dma-iommu.h"
#include "pisa-priv.h"
@@ -88,3 +94,209 @@ void __init pisa_slots_init(void)
/* Size of every domain's window: one 2MB page per slot. */
pisa_window_size = (u64)pisa_nr_slots << PISA_SHIFT;
}
+
+/*
+ * Registry of PISA IOMMU domains (0 .. PISA_MAX_DOMAINS - 1). Each slot is
+ * FREE (!domain && !base), ACTIVE (domain), or DRAINING (!domain && base) while
+ * pisa_domain_release() clears @meta->domains_mapped bits.
+ */
+static struct {
+ struct iommu_domain *domain;
+ dma_addr_t base;
+} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
+static DEFINE_SPINLOCK(pisa_domains_lock);
+
+/* Protects pisa_unmap_all() readers against concurrent pisa_domain_release(). */
+DEFINE_STATIC_SRCU(pisa_srcu);
+
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+unsigned int pisa_idx_gen __read_mostly;
+
+/* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
+static void pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
+ int idx, dma_addr_t iova, size_t unmapped)
+{
+ pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
+ &iova, domain, idx, unmapped);
+ WRITE_ONCE(meta->flags, meta->flags | PISA_QUARANTINED);
+}
+
+/*
+ * Unmap @meta's leaf PTE from every active domain holding it. Returns 0 on
+ * success, or -EIO if any unmap failed (@meta is then quarantined).
+ */
+int pisa_unmap_all(struct pisa_meta *meta)
+{
+ dma_addr_t offset = pisa_meta_win_offset(meta);
+ int idx, srcu_idx;
+
+ srcu_idx = srcu_read_lock(&pisa_srcu);
+ for_each_set_bit(idx, meta->domains_mapped, PISA_MAX_DOMAINS) {
+ struct iommu_domain *domain;
+ unsigned long flags;
+ size_t unmapped;
+ dma_addr_t base;
+
+ if (!test_and_clear_bit(idx, meta->domains_mapped))
+ continue;
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ domain = pisa_domains[idx].domain;
+ base = pisa_domains[idx].base;
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ /* Free, or draining: pisa_domain_release() handles the bit. */
+ if (!domain)
+ continue;
+
+ unmapped = iommu_unmap(domain, base + offset, PISA_SIZE);
+ if (unlikely(unmapped != PISA_SIZE))
+ pisa_quarantine_meta(meta, domain, idx, base + offset, unmapped);
+ }
+ srcu_read_unlock(&pisa_srcu, srcu_idx);
+
+ return (meta->flags & PISA_QUARANTINED) ? -EIO : 0;
+}
+
+/* Clear domain @idx from all populated slots. */
+static unsigned int pisa_forget_domain(int idx)
+{
+ unsigned int slot, dropped = 0;
+
+ rcu_read_lock();
+ for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+ struct pisa_meta *meta = READ_ONCE(pisa_slot2meta[slot]);
+
+ if (meta && test_bit(idx, meta->domains_mapped) &&
+ test_and_clear_bit(idx, meta->domains_mapped))
+ dropped++;
+ }
+ rcu_read_unlock();
+ return dropped;
+}
+
+/*
+ * Release @domain's registry index and clear its bit in all active
+ * PISA pages, waiting out concurrent unmappers on @pisa_srcu before freeing
+ * the slot.
+ */
+void pisa_domain_release(struct iommu_domain *domain)
+{
+ struct pisa_window *win;
+ unsigned int dropped;
+ unsigned long flags;
+ int idx;
+
+ /* Nothing has ever been active, so nothing can reference @domain. */
+ if (!static_branch_unlikely(&pisa_key))
+ return;
+
+ win = pisa_dma_window(domain);
+ /* @domain never active, so no PISA page can be holding a bit for it. */
+ if (!win || !smp_load_acquire(&win->base))
+ return;
+
+ idx = win->domain_idx - PISA_IDX_FIRST;
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ if (pisa_domains[idx].domain != domain) {
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+ return;
+ }
+ /* Leaving @base set moves the slot to DRAINING state */
+ pisa_domains[idx].domain = NULL;
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ dropped = pisa_forget_domain(idx);
+
+ /* Wait out in-flight pisa_unmap_all() readers before reusing @idx. */
+ synchronize_srcu(&pisa_srcu);
+
+ /* Mark slot FREE and bump @pisa_idx_gen so NOIDX domains retry. */
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ pisa_domains[idx].base = 0;
+ WRITE_ONCE(pisa_idx_gen, pisa_idx_gen + 1);
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ if (dropped)
+ pr_debug("pisa: released %u PISA page mappings for domain %p (index %d)\n",
+ dropped, domain, idx);
+}
+
+/*
+ * Remove @meta's leaf PTE from live @domain (registered at @idx with @base),
+ * batching the IOTLB flush into @gather. Returns 1 if unmapped, 0 otherwise.
+ */
+static unsigned int pisa_unmap_domain_locked(struct pisa_meta *meta,
+ struct iommu_domain *domain,
+ int idx, dma_addr_t base,
+ struct iommu_iotlb_gather *gather)
+{
+ dma_addr_t iova = base + pisa_meta_win_offset(meta);
+ size_t unmapped;
+
+ if (!test_and_clear_bit(idx, meta->domains_mapped))
+ return 0;
+
+ unmapped = iommu_unmap_fast(domain, iova, PISA_SIZE, gather);
+ if (unlikely(unmapped != PISA_SIZE))
+ pisa_quarantine_meta(meta, domain, idx, iova, unmapped);
+ return 1;
+}
+
+/*
+ * Unmap all PISA leaf PTEs from a domain whose last DMA API user detached,
+ * keeping its window and domain index so mappings are re-installed on demand.
+ */
+void pisa_domain_quiesce(struct iommu_domain *domain)
+{
+ struct iommu_iotlb_gather gather;
+ struct pisa_window *win;
+ unsigned int slot, done = 0;
+ bool release = false;
+ dma_addr_t base;
+ int idx;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return;
+ win = pisa_dma_window(domain);
+ /* Pairs with smp_store_release() in pisa_window_assign(). */
+ base = win ? smp_load_acquire(&win->base) : 0;
+ if (!base)
+ return;
+ idx = win->domain_idx - PISA_IDX_FIRST;
+
+ iommu_iotlb_gather_init(&gather);
+ /* Keep call_rcu()-freed pages from reaching the buddy allocator before IOTLB sync. */
+ rcu_read_lock();
+ for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+ struct pisa_meta *meta = READ_ONCE(pisa_slot2meta[slot]);
+ struct pisa_alloc *pa;
+ unsigned long flags;
+
+ if (!meta || !test_bit(idx, meta->domains_mapped))
+ continue;
+ done += pisa_unmap_domain_locked(meta, domain, idx, base, &gather);
+ if (likely(!(meta->flags & PISA_QUARANTINED)))
+ continue;
+ pa = READ_ONCE(meta->pa);
+ if (!pa)
+ continue;
+ spin_lock_irqsave(&pa->lock, flags);
+ if (meta->pa == pa) {
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+ release |= pisa_place_meta(meta);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ }
+
+ if (done)
+ iommu_iotlb_sync(domain, &gather);
+ rcu_read_unlock();
+ if (release)
+ pisa_schedule_reclaim();
+ if (done)
+ pr_debug("pisa: unmapped %u PISA pages from idle domain %p (index %d)\n",
+ done, domain, idx);
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 3ef6e51402ceb..f97f69a13d1a9 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -13,6 +13,9 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+struct iommu_domain;
+struct iommu_iotlb_gather;
+
/**
* struct pisa_window - A domain's IOVA window reserved for PISA pages
* @base: Window base IOVA (PISA_SIZE aligned), or 0 if unassigned
@@ -33,6 +36,9 @@ struct pisa_window {
#define PISA_BLOCKS(order) (1U << (PISA_ORDER - (order)))
+/* Max simultaneous PISA IOMMU domains (bit index in pisa_meta.domains_mapped). */
+#define PISA_MAX_DOMAINS 128
+
/* Sentinel values of pisa_window.domain_idx below PISA_IDX_FIRST. */
enum {
PISA_IDX_NONE, /* not attempted yet */
@@ -47,15 +53,18 @@ enum {
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ * - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
+ * - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
- * @flags: PISA_ACTIVE while this page is owned by a pisa_alloc
+ * @flags: PISA_ACTIVE / PISA_QUARANTINED
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
* @slot: Dense IOVA slot index in every domain's window
+ * @domains_mapped: Bitmap of domain indices holding this page's leaf PTE
* @pa: Owning pisa_alloc (NULL for reservoir pages)
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
@@ -69,6 +78,9 @@ struct pisa_meta {
u16 nr_dirty;
u8 on_list;
u32 slot;
+
+ DECLARE_BITMAP(domains_mapped, PISA_MAX_DOMAINS);
+
struct pisa_alloc *pa;
struct list_head list;
@@ -136,6 +148,12 @@ struct pisa_node {
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+/* Offset of @meta from the start of a domain's window. */
+static inline dma_addr_t pisa_meta_win_offset(const struct pisa_meta *meta)
+{
+ return (dma_addr_t)meta->slot << PISA_SHIFT;
+}
+
int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta);
@@ -202,11 +220,30 @@ unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned lon
/* Size in bytes of every domain's PISA IOVA window, fixed at boot. */
extern u64 pisa_window_size;
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+extern unsigned int pisa_idx_gen;
extern unsigned int pisa_nr_slots;
void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
+int pisa_unmap_all(struct pisa_meta *meta);
+
+/* Return negative errno if @win's domain was permanently/currently declined, or 0. */
+static inline int pisa_window_declined(const struct pisa_window *win)
+{
+ switch (READ_ONCE(win->domain_idx)) {
+ case PISA_IDX_NOHUGE:
+ return -EOPNOTSUPP;
+ case PISA_IDX_NOWINDOW:
+ return -ENOSPC;
+ case PISA_IDX_NOIDX:
+ return READ_ONCE(win->idx_gen) == READ_ONCE(pisa_idx_gen) ?
+ -EBUSY : 0;
+ default:
+ return 0;
+ }
+}
/* Return true if @dma falls within @win's reserved PISA IOVA window. */
static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
@@ -217,17 +254,34 @@ static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
return base && dma - base < pisa_window_size;
}
+void pisa_domain_release(struct iommu_domain *domain);
+void pisa_domain_quiesce(struct iommu_domain *domain);
+
#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
#define pisa_nr_slots 0U
static inline void pisa_slots_init(void) {}
static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
+static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+
+static inline int pisa_window_declined(const struct pisa_window *win)
+{
+ return -EOPNOTSUPP;
+}
static inline bool is_pisa_iova(const struct pisa_window *win, dma_addr_t dma)
{
return false;
}
+static inline void pisa_domain_release(struct iommu_domain *domain)
+{
+}
+
+static inline void pisa_domain_quiesce(struct iommu_domain *domain)
+{
+}
+
#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
#endif /* _DRIVERS_IOMMU_PISA_PRIV_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 18ba1c83bd6d3..ab1c32e065bbb 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -27,6 +27,7 @@
enum {
PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
+ PISA_QUARANTINED = BIT(1), /* IOMMU unmap failed: never reuse */
};
/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (7 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
` (16 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Intercept the DMA mapping and unmapping functions so that PISA
buffers are mapped within the reserved IOVA range.
This is implemented as follows:
- each domain that enables PISA (via per-device optin) is dynamically
assigned an IOVA window and a compact index the first time it maps a
PISA page.
This way, one bit in pisa_meta suffices to record whether the domain
has already mapped the page.
- The page is mapped in the domain at win->base + slot * 2MB on first
use, and subsequent mappings reuse that translation.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Kconfig | 1 +
drivers/iommu/dma-iommu.c | 15 +++
drivers/iommu/iommu.c | 5 +-
drivers/iommu/pisa-alloc.c | 5 +-
drivers/iommu/pisa-kunit.c | 133 +++++++++++++++++++++++++++
drivers/iommu/pisa-map.c | 184 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 35 ++++++-
include/linux/pisa.h | 3 +
8 files changed, 377 insertions(+), 4 deletions(-)
diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 15e06eaed0924..52cce0963525b 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -428,6 +428,7 @@ config PISA
bool "Pre-mapped IOVA Superpage Allocator"
depends on X86_64 || ARM64
depends on !PREEMPT_RT
+ select NEED_SG_DMA_FLAGS
default y
help
PISA is a specialized memory allocator with an API similar to
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 025bceebb2262..eb920ad77c783 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -837,6 +837,8 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
struct iommu_iotlb_gather iotlb_gather;
size_t unmapped;
+ if (is_pisa_iova(&cookie->win_pisa, dma_addr))
+ return;
dma_addr -= iova_off;
size = iova_align(iovad, size + iova_off);
iommu_iotlb_gather_init(&iotlb_gather);
@@ -863,6 +865,8 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
iommu_deferred_attach(dev, domain))
return DMA_MAPPING_ERROR;
+ if (!pisa_map_phys(&iova, dev, domain, phys, size, prot))
+ return iova;
/* If anyone ever wants this we'd need support in the IOVA allocator */
if (dev_WARN_ONCE(dev, dma_get_min_align_mask(dev) > iova_mask(iovad),
"Unsupported alignment constraint\n"))
@@ -1277,6 +1281,10 @@ void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
{
phys_addr_t phys;
+ /* PISA mapped buffer: nothing to unmap or release. */
+ if (is_pisa_iova(&iommu_get_dma_domain(dev)->iova_cookie->win_pisa, dma_handle))
+ return;
+
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) {
__iommu_dma_unmap(dev, dma_handle, size);
return;
@@ -1478,6 +1486,13 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
*/
break;
case PCI_P2PDMA_MAP_NONE:
+ /* Map PISA pages and mark as bus addresses to skip IOVA alloc and unmap. */
+ if (!pisa_map_phys(&s->dma_address, dev, domain,
+ sg_phys(s), s_length, prot)) {
+ sg_dma_len(s) = s_length;
+ sg_dma_mark_bus_address(s);
+ continue;
+ }
break;
case PCI_P2PDMA_MAP_BUS_ADDR:
/*
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index dfba3d5c277c2..068eaf262c7f6 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -2867,8 +2867,9 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
while (i <= nents) {
phys_addr_t s_phys = sg_phys(sg);
+ bool is_bus = i < nents && sg_dma_is_bus_address(sg);
- if (len && s_phys != start + len) {
+ if (len && (is_bus || s_phys != start + len)) {
ret = iommu_map_nosync(domain, iova + mapped, start,
len, prot, gfp);
if (ret)
@@ -2878,7 +2879,7 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
len = 0;
}
- if (sg_dma_is_bus_address(sg))
+ if (is_bus)
goto next;
if (len) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index dd90aad296edf..f98142882519c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -299,9 +299,11 @@ struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direct
spin_lock_init(&pa->lock);
pa->order = order;
- pa->nid = pisa_sanitize_nid(nid);
+ pa->prot = (dir != DMA_FROM_DEVICE ? IOMMU_READ : 0) |
+ (dir != DMA_TO_DEVICE ? IOMMU_WRITE : 0);
pa->max_idle_pages = READ_ONCE(pisa_max_idle) ? : 1;
pisa_set_name(pa, "%ps", (void *)_RET_IP_);
+ pa->nid = pisa_sanitize_nid(nid);
INIT_LIST_HEAD(&pa->partial);
INIT_LIST_HEAD(&pa->partial_dirty);
INIT_LIST_HEAD(&pa->idle);
@@ -607,6 +609,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
goto out;
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
+ meta->prot = pa->prot;
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
}
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index f0775aa45a058..e16268d55f3de 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -6,6 +6,7 @@
#include <linux/gfp.h>
#include <linux/pisa.h>
#include <linux/iommu.h>
+#include <linux/iova.h>
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -184,8 +185,10 @@ static void test_alloc_and_recycle(struct kunit *test)
#ifdef CONFIG_IOMMU_DMA
static void test_window_helpers(struct kunit *test)
{
+ struct device dev = { .numa_node = NUMA_NO_NODE };
struct iommu_domain dummy_domain = {};
struct pisa_window win = {};
+ dma_addr_t d;
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -212,10 +215,139 @@ static void test_window_helpers(struct kunit *test)
win.idx_gen--;
KUNIT_EXPECT_EQ(test, pisa_window_declined(&win), 0);
+ /* A device without an IOMMU group has no PISA window. */
+ KUNIT_EXPECT_NE(test, pisa_map_phys(&d, NULL, NULL, 0, PAGE_SIZE, 0), 0);
+ KUNIT_EXPECT_NE(test, pisa_map_phys(&d, &dev, NULL, 0, PAGE_SIZE, 0), 0);
/* Releasing an unregistered domain with an active allocator is a safe no-op. */
pisa_domain_release(&dummy_domain);
}
+/*
+ * Fake DMA domain for the mapping and coherent allocation tests, backed by a
+ * real iommu_dma_cookie and no-op page table ops.
+ */
+static int fake_map_pages(struct iommu_domain *domain, unsigned long iova,
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
+{
+ *mapped = pgsize * pgcount;
+ return 0;
+}
+
+static size_t fake_unmap_pages(struct iommu_domain *domain, unsigned long iova,
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *iotlb_gather)
+{
+ return pgsize * pgcount;
+}
+
+static const struct iommu_domain_ops fake_domain_ops = {
+ .map_pages = fake_map_pages,
+ .unmap_pages = fake_unmap_pages,
+};
+
+static struct iommu_domain fake_domain = {
+ .type = IOMMU_DOMAIN_DMA,
+ .pgsize_bitmap = SZ_4K | PISA_SIZE,
+ .ops = &fake_domain_ops,
+};
+
+static void fake_domain_teardown(void *unused)
+{
+ iommu_put_dma_cookie(&fake_domain);
+ fake_domain.cookie_type = IOMMU_COOKIE_NONE;
+ fake_domain.iova_cookie = NULL;
+ iova_cache_put();
+}
+
+static int fake_domain_setup(struct kunit *test)
+{
+ int ret = iova_cache_get();
+
+ if (ret)
+ return ret;
+ ret = iommu_get_dma_cookie(&fake_domain);
+ if (ret) {
+ iova_cache_put();
+ return ret;
+ }
+ init_iova_domain(pisa_dma_iovad(&fake_domain), SZ_4K, 1);
+ return kunit_add_action_or_reset(test, fake_domain_teardown, NULL);
+}
+
+static void test_map_and_domain_lifecycle(struct kunit *test)
+{
+ struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
+ .coherent_dma_mask = DMA_BIT_MASK(64) };
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct pisa_window *win;
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ unsigned int domain_idx;
+ dma_addr_t d1, d2;
+ struct page *p;
+ phys_addr_t phys;
+
+ dev.dma_mask = &dev.coherent_dma_mask;
+ set_dev_node(&dev, first_memory_node);
+#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
+ dev.dma_coherent = true;
+#endif
+ KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+ win = pisa_dma_window(&fake_domain);
+
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ p = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p);
+
+ phys = page_to_phys(p);
+ meta = pisa_meta_of_pfn(page_to_pfn(p));
+ KUNIT_EXPECT_TRUE(test, is_pisa_phys(phys));
+
+ /* TX-only allocator declines IOMMU_WRITE mappings. */
+ KUNIT_EXPECT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), -EACCES);
+
+ /* First map assigns the window and installs the 2MB PTE. */
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(win, d1));
+ domain_idx = win->domain_idx - PISA_IDX_FIRST;
+ KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Second map at an offset hits the lockless fast path. */
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, phys + 128, 256,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1 + 128);
+
+ /* Quiescing the domain unmaps the PISA page but keeps the window. */
+ pisa_domain_quiesce(&fake_domain);
+ KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1);
+ KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Releasing the domain forgets the mapping bit across allocators. */
+ pisa_domain_release(&fake_domain);
+ KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+
+ /* Re-map into a fresh window so shrinking exercises pisa_unmap_all(). */
+ *win = (struct pisa_window){};
+ KUNIT_EXPECT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+ IOMMU_READ | IOMMU_CACHE), 0);
+
+ __free_pages(p, 2);
+ pisa_shrink_scan(NULL, &sc);
+}
#endif /* CONFIG_IOMMU_DMA */
static void test_reservoir_and_scrubber(struct kunit *test)
@@ -231,6 +363,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_alloc_and_recycle),
#ifdef CONFIG_IOMMU_DMA
KUNIT_CASE(test_window_helpers),
+ KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
KUNIT_CASE(test_reservoir_and_scrubber),
{}
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2ba977bb68734..2c481e514575c 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -12,6 +12,8 @@
#include <linux/pisa.h>
#include <linux/export.h>
#include <linux/iommu.h>
+#include <linux/iova.h>
+#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/slab.h>
@@ -103,6 +105,7 @@ void __init pisa_slots_init(void)
static struct {
struct iommu_domain *domain;
dma_addr_t base;
+ spinlock_t map_lock; /* serialises iommu_map() in __pisa_map_phys() */
} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
static DEFINE_SPINLOCK(pisa_domains_lock);
@@ -300,3 +303,184 @@ void pisa_domain_quiesce(struct iommu_domain *domain)
pr_debug("pisa: unmapped %u PISA pages from idle domain %p (index %d)\n",
done, domain, idx);
}
+
+/* Allocate a PISA_SIZE-aligned IOVA window of @size bytes from @domain. */
+static dma_addr_t pisa_dma_window_alloc(struct device *dev, struct iommu_domain *domain, u64 size)
+{
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ struct iova_domain *iovad = pisa_dma_iovad(domain);
+ unsigned long shift, iova_len;
+ struct iova *new_iova;
+
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+
+ if (!iovad)
+ return 0;
+
+ if (domain->geometry.force_aperture)
+ limit = min_t(u64, limit, domain->geometry.aperture_end);
+
+ /* Ignore devices with too many constraints. */
+ if (limit <= DMA_BIT_MASK(32) || size > limit - PISA_SIZE)
+ return 0;
+
+ shift = iova_shift(iovad);
+ iova_len = (size + PISA_SIZE) >> shift;
+
+ /* Ignore advisory dev->iommu->pci_32bit_workaround preference. */
+ new_iova = alloc_iova(iovad, iova_len, limit >> shift, false);
+ if (!new_iova)
+ return 0;
+
+ return ALIGN((dma_addr_t)new_iova->pfn_lo << shift, PISA_SIZE);
+}
+
+/*
+ * Assign @domain an IOVA window and dense domain index on first use, recording
+ * permanent or retryable failures in @win->domain_idx.
+ */
+int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win)
+{
+ u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ u64 size = pisa_window_size;
+ unsigned long flags;
+ dma_addr_t base = 0;
+ int idx, ret;
+
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+
+ if (!dev_is_dma_coherent(dev))
+ return -EOPNOTSUPP;
+
+ if (limit <= DMA_BIT_MASK(32) || size > limit - PISA_SIZE)
+ return -ENOSPC;
+
+ if (!(domain->pgsize_bitmap & PISA_SIZE)) {
+ if (cmpxchg(&win->domain_idx, PISA_IDX_NONE, PISA_IDX_NOHUGE) == PISA_IDX_NONE)
+ dev_warn_ratelimited(dev, "pisa: domain lacks a %luKB page size, not PISA\n",
+ (unsigned long)(PISA_SIZE >> 10));
+ return -EOPNOTSUPP;
+ }
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+
+ /* Bail if another CPU got here first, or if declined earlier. */
+ ret = win->base ? 0 : pisa_window_declined(win);
+ if (win->base || ret) {
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+ return ret;
+ }
+
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (!pisa_domains[idx].domain && !pisa_domains[idx].base)
+ break;
+ }
+ if (idx == PISA_MAX_DOMAINS) {
+ win->idx_gen = pisa_idx_gen;
+ WRITE_ONCE(win->domain_idx, PISA_IDX_NOIDX);
+ ret = -EBUSY;
+ } else {
+ base = pisa_dma_window_alloc(dev, domain, size);
+ if (!base) {
+ WRITE_ONCE(win->domain_idx, PISA_IDX_NOWINDOW);
+ ret = -ENOSPC;
+ } else {
+ pisa_domains[idx].domain = domain;
+ pisa_domains[idx].base = base;
+ spin_lock_init(&pisa_domains[idx].map_lock);
+ WRITE_ONCE(win->domain_idx, idx + PISA_IDX_FIRST);
+ /* Publish .domain_idx before .base (paired with is_pisa_iova()). */
+ smp_store_release(&win->base, base);
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ if (ret == -EBUSY)
+ pr_warn_ratelimited("pisa: all %d domain indices in use, not PISA for domain %p until one is released\n",
+ PISA_MAX_DOMAINS, domain);
+ else if (ret == -ENOSPC)
+ dev_warn_ratelimited(dev,
+ "pisa: no free %llu MB IOVA range for a PISA window\n",
+ size >> 20);
+ else
+ dev_info(dev, "pisa: domain %p index %d window %pad + %llu MB\n",
+ domain, idx, &base, size >> 20);
+ return ret;
+}
+
+/*
+ * Compute the window IOVA of active address @phys for @dev and store it in
+ * *@iova. On first use in a domain, reserves @win via alloc_iova(), and on
+ * first use of a PISA page in that domain, installs its 2MB leaf PTE via
+ * iommu_map(..., GFP_ATOMIC) under spinlocks.
+ * Returns 0 on success, or negative errno if @phys/@prot/@dev cannot use the window.
+ */
+int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ struct pisa_window *win = pisa_dma_window(domain);
+ unsigned long pfn = phys >> PAGE_SHIFT, flags;
+ struct pisa_meta *meta;
+ unsigned int domain_idx;
+ int block_prot, ret;
+ dma_addr_t addr;
+ u64 limit;
+
+ if (unlikely(!iova || !dev || !win || !dev_is_dma_coherent(dev) ||
+ (phys & (PISA_SIZE - 1)) + size > PISA_SIZE))
+ return -EINVAL;
+
+ meta = pisa_meta_of_pfn(pfn);
+
+ /* Every leaf PTE of the page carries IOMMU_CACHE | @meta->prot. */
+ block_prot = IOMMU_CACHE | meta->prot;
+ if (unlikely(prot & ~block_prot))
+ return -EACCES;
+
+ /* Acquire pairs with the .base release in pisa_window_assign(). */
+ if (unlikely(!smp_load_acquire(&win->base))) {
+ /* Test cached decline in @domain_idx before taking @pisa_domains_lock. */
+ ret = pisa_window_declined(win);
+ if (likely(!ret))
+ ret = pisa_window_assign(dev, domain, win);
+
+ if (ret)
+ return ret;
+ }
+
+ addr = pisa_phys_to_iova(win, meta, phys);
+
+ /* Check @dev's own DMA mask and bus limit against the window IOVA. */
+ limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+ limit = min_not_zero(limit, dev->coherent_dma_mask);
+ if (unlikely(addr + size - 1 > limit))
+ return -ENOSPC;
+
+ domain_idx = win->domain_idx - PISA_IDX_FIRST;
+ if (likely(test_bit(domain_idx, meta->domains_mapped))) {
+ *iova = addr;
+ return 0;
+ }
+
+ /* First use of this PISA page in this domain: install the leaf PTE. */
+ ret = 0;
+ spin_lock_irqsave(&pisa_domains[domain_idx].map_lock, flags);
+ if (!test_bit(domain_idx, meta->domains_mapped)) {
+ phys_addr_t pisa_phys = phys & PISA_MASK;
+
+ ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
+ pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+ if (likely(!ret)) {
+ /* Ensure leaf PTE is visible before setting the domain bit. */
+ smp_mb__before_atomic();
+ set_bit(domain_idx, meta->domains_mapped);
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains[domain_idx].map_lock, flags);
+ if (unlikely(ret))
+ return ret;
+
+ *iova = addr;
+ return 0;
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f97f69a13d1a9..8d83ec09c19a9 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -2,7 +2,7 @@
#ifndef _DRIVERS_IOMMU_PISA_PRIV_H
#define _DRIVERS_IOMMU_PISA_PRIV_H
-#include <linux/dma-direction.h>
+#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/jiffies.h>
#include <linux/list.h>
@@ -13,6 +13,7 @@
#include <linux/types.h>
#include <linux/workqueue.h>
+struct device;
struct iommu_domain;
struct iommu_iotlb_gather;
@@ -53,6 +54,7 @@ enum {
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
+ * - pisa_domains[].map_lock (spinlock, irqsave): first-time iommu_map() in __pisa_map_phys().
* - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
* - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
@@ -60,6 +62,7 @@ enum {
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE / PISA_QUARANTINED
+ * @prot: Owning allocator's IOMMU_READ/WRITE protection bits
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
* @on_list: Current placement in @pa (enum pisa_meta_list)
@@ -74,6 +77,7 @@ enum {
*/
struct pisa_meta {
u8 flags;
+ u8 prot;
u16 nr_clean;
u16 nr_dirty;
u8 on_list;
@@ -105,6 +109,7 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
* struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
* @lock: Spinlock protecting lists and block bitmaps
* @order: Block order managed by this allocator (<= PISA_ORDER)
+ * @prot: IOMMU_READ and/or IOMMU_WRITE from @dir passed to pisa_create()
* @partial: Partially used PISA pages with @nr_clean > 0
* @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
* @idle: Completely free PISA pages (reclaimable under pressure)
@@ -120,6 +125,7 @@ struct pisa_alloc {
/* First cacheline: hot fields touched on block alloc and free. */
spinlock_t lock;
u8 order;
+ u8 prot;
struct list_head partial;
struct list_head partial_dirty;
struct list_head idle;
@@ -148,6 +154,14 @@ struct pisa_node {
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+/* IOVA of @phys inside active RAM page @meta in @win. */
+static inline dma_addr_t pisa_phys_to_iova(const struct pisa_window *win,
+ const struct pisa_meta *meta, phys_addr_t phys)
+{
+ return win->base + ((dma_addr_t)meta->slot << PISA_SHIFT) +
+ (phys & (PISA_SIZE - 1));
+}
+
/* Offset of @meta from the start of a domain's window. */
static inline dma_addr_t pisa_meta_win_offset(const struct pisa_meta *meta)
{
@@ -228,6 +242,18 @@ void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
int pisa_unmap_all(struct pisa_meta *meta);
+int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win);
+int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot);
+
+static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ return is_pisa_phys(phys) ?
+ __pisa_map_phys(iova, dev, domain, phys, size, prot) : -EINVAL;
+}
/* Return negative errno if @win's domain was permanently/currently declined, or 0. */
static inline int pisa_window_declined(const struct pisa_window *win)
@@ -265,6 +291,13 @@ static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
+ struct iommu_domain *domain, phys_addr_t phys,
+ size_t size, int prot)
+{
+ return -EOPNOTSUPP;
+}
+
static inline int pisa_window_declined(const struct pisa_window *win)
{
return -EOPNOTSUPP;
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index ab1c32e065bbb..c64689b81befe 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -5,11 +5,14 @@
#include <linux/types.h>
#include <linux/bitops.h>
+#include <linux/dma-mapping.h>
#include <linux/jump_label.h>
#include <linux/mm.h>
#include <linux/numa.h>
#include <linux/rcupdate.h>
+struct device;
+
#ifdef CONFIG_PISA
#define PISA_SHIFT 21U
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (8 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
` (15 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Because of the performance and security implications, the use of PISA
features is opt-in and configurable per device with a fine granularity.
Add boolean attributes to struct device exposed under
/sys/devices/*/*/pisa_*:
- pisa_coherent: enable PISA for coherent allocations
- pisa_rx_bufs: enable PISA for RX buffers / page_pool
- pisa_tx_bufs: enable PISA for TX socket page frags
- pisa_tx_skb_heads: enable PISA slab cache for small TX skb heads
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 64 ++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-alloc.c | 24 ++++++++++++++
drivers/iommu/pisa-kunit.c | 67 ++++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-priv.h | 1 -
include/linux/device.h | 12 +++++++
include/linux/pisa.h | 19 +++++++++++
6 files changed, 186 insertions(+), 1 deletion(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index cbe05197d4a0b..97cd2d03ff77c 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -25,6 +25,7 @@
#include <linux/notifier.h>
#include <linux/of.h>
#include <linux/of_device.h>
+#include <linux/pisa.h>
#include <linux/pm_runtime.h>
#include <linux/sched/mm.h>
#include <linux/sched/signal.h>
@@ -2907,6 +2908,54 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
}
static const DEVICE_ATTR_RO(removable);
+/* Per-device configuration to selectively enable PISA */
+static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
+{
+ bool val;
+ int ret = kstrtobool(buf, &val);
+
+ if (ret < 0)
+ return ret;
+ if (val) {
+ ret = pisa_meta_init();
+ if (ret < 0)
+ return ret;
+ }
+ WRITE_ONCE(*field, val);
+ return count;
+}
+
+#define PISA_BOOL_ATTR(name) \
+static ssize_t name##_show(struct device *dev, \
+ struct device_attribute *attr, char *buf) \
+{ \
+ return sysfs_emit(buf, "%u\n", READ_ONCE(dev->name)); \
+} \
+static ssize_t name##_store(struct device *dev, \
+ struct device_attribute *attr, \
+ const char *buf, size_t count) \
+{ \
+ return pisa_attr_store(buf, count, &dev->name); \
+} \
+static DEVICE_ATTR_RW(name)
+
+PISA_BOOL_ATTR(pisa_coherent);
+PISA_BOOL_ATTR(pisa_rx_bufs);
+PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_BOOL_ATTR(pisa_tx_skb_heads);
+
+static struct attribute *dev_attr_pisa[] = {
+ &dev_attr_pisa_coherent.attr,
+ &dev_attr_pisa_rx_bufs.attr,
+ &dev_attr_pisa_tx_bufs.attr,
+ &dev_attr_pisa_tx_skb_heads.attr,
+ NULL,
+};
+
+static const struct attribute_group dev_attr_pisa_group = {
+ .attrs = dev_attr_pisa,
+};
+
int device_add_groups(struct device *dev,
const struct attribute_group *const *groups)
{
@@ -3014,8 +3063,20 @@ static int device_add_attrs(struct device *dev)
goto err_remove_dev_removable;
}
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
+ error = device_add_group(dev, &dev_attr_pisa_group);
+ if (error)
+ goto err_remove_dev_physical_location;
+ }
+
return 0;
+ err_remove_dev_physical_location:
+ if (dev->physical_location) {
+ device_remove_group(dev, &dev_attr_physical_location_group);
+ kfree(dev->physical_location);
+ dev->physical_location = NULL;
+ }
err_remove_dev_removable:
device_remove_file(dev, &dev_attr_removable);
err_remove_dev_waiting_for_supplier:
@@ -3039,6 +3100,9 @@ static void device_remove_attrs(struct device *dev)
const struct class *class = dev->class;
const struct device_type *type = dev->type;
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+ device_remove_group(dev, &dev_attr_pisa_group);
+
if (dev->physical_location) {
device_remove_group(dev, &dev_attr_physical_location_group);
kfree(dev->physical_location);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index f98142882519c..b0d968aa9181c 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/device.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/export.h>
@@ -392,6 +393,29 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred
return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
}
+/**
+ * pisa_has_free_blocks_rx - Check if @dev has free PISA RX blocks of @order on @nid
+ * @dev: Device to check
+ * @order: Block order
+ * @nid: Target NUMA node (or NUMA_NO_NODE)
+ *
+ * Return: true if @dev->pisa_rx_bufs is enabled and the per-node allocator
+ * has a free block available.
+ */
+bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid)
+{
+ struct pisa_alloc *pa;
+
+ if (!dev || !READ_ONCE(dev->pisa_rx_bufs))
+ return false;
+ pa = pisa_node_pa(nid, order, false);
+ return pa &&
+ (!list_empty_careful(&pa->partial) ||
+ !list_empty_careful(&pa->partial_dirty) ||
+ !list_empty_careful(&pa->idle));
+}
+EXPORT_SYMBOL(pisa_has_free_blocks_rx);
+
/*
* Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
* pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index e16268d55f3de..ab64657ad5c58 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -5,6 +5,8 @@
#include <kunit/test.h>
#include <linux/gfp.h>
#include <linux/pisa.h>
+#include <linux/kernfs.h>
+#include <linux/sysfs.h>
#include <linux/iommu.h>
#include <linux/iova.h>
#include <linux/mm.h>
@@ -350,6 +352,70 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
}
#endif /* CONFIG_IOMMU_DMA */
+static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
+ const char *name, bool *field)
+{
+ const struct sysfs_ops *ops = dev->kobj.ktype->sysfs_ops;
+ struct kernfs_node *kn;
+ char *buf;
+
+ buf = (char *)__get_free_page(GFP_KERNEL);
+ KUNIT_EXPECT_NOT_NULL(test, buf);
+ if (!buf)
+ return;
+ kn = kernfs_find_and_get(dev->kobj.sd, name);
+ KUNIT_EXPECT_NOT_NULL(test, kn);
+ if (!kn)
+ goto out_free;
+
+ KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+ KUNIT_EXPECT_STREQ(test, buf, "0\n");
+ KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+ KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "1\n", 2), 2);
+ KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+ KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+ KUNIT_EXPECT_STREQ(test, buf, "1\n");
+
+ KUNIT_EXPECT_LT(test, ops->store(&dev->kobj, kn->priv, "bad", 3), 0);
+ KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+
+ KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "0\n", 2), 2);
+ KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+ kernfs_put(kn);
+out_free:
+ free_page((unsigned long)buf);
+}
+
+static void kunit_attr_dev_release(struct device *dev)
+{
+ kfree(dev);
+}
+
+static void test_dev_pisa_attrs(struct kunit *test)
+{
+ struct device *dev;
+
+ dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, dev);
+ dev->init_name = "pisa_attr_kunit";
+ dev->release = kunit_attr_dev_release;
+ dev->coherent_dma_mask = DMA_BIT_MASK(64);
+ dev->dma_mask = &dev->coherent_dma_mask;
+
+ if (device_register(dev)) {
+ put_device(dev);
+ KUNIT_FAIL(test, "device_register failed");
+ return;
+ }
+ kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
+ kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ device_unregister(dev);
+}
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -365,6 +431,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_window_helpers),
KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
+ KUNIT_CASE(test_dev_pisa_attrs),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 8d83ec09c19a9..e5ca95d2654cd 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -133,7 +133,6 @@ struct pisa_alloc {
unsigned int num_idle_pages;
unsigned int max_idle_pages;
unsigned long idle_jiffies;
-
/* Cold. */
int nid;
struct list_head node;
diff --git a/include/linux/device.h b/include/linux/device.h
index 429e39f647cf9..b3c11e32116ca 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -695,6 +695,13 @@ enum struct_device_flags {
* @removable: Whether the device can be removed from the system. This
* should be set by the subsystem / bus driver that discovered
* the device.
+ * @pisa_coherent: Back coherent dma_alloc_attrs() with PISA
+ * (sysfs pisa_coherent).
+ * @pisa_rx_bufs: Use PISA allocators for RX buffers (sysfs pisa_rx_bufs).
+ * @pisa_tx_bufs: Use PISA allocators for socket TX page fragments
+ * (sysfs pisa_tx_bufs).
+ * @pisa_tx_skb_heads: Use PISA slab cache for small TX skb heads
+ * (sysfs pisa_tx_skb_heads).
* @flags: DEV_FLAG_XXX flags. Use atomic bitfield operations to modify.
*
* At the lowest level, every device in a Linux system is represented by an
@@ -799,6 +806,11 @@ struct device {
enum device_removable removable;
+ bool pisa_coherent;
+ bool pisa_rx_bufs;
+ bool pisa_tx_bufs;
+ bool pisa_tx_skb_heads;
+
DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
};
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index c64689b81befe..92482fce943e9 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -7,6 +7,7 @@
#include <linux/bitops.h>
#include <linux/dma-mapping.h>
#include <linux/jump_label.h>
+#include <linux/device.h>
#include <linux/mm.h>
#include <linux/numa.h>
#include <linux/rcupdate.h>
@@ -82,6 +83,13 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int nr_pages, struct page **page_array,
bool frozen);
+bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
+
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+ return dev && READ_ONCE(dev->pisa_rx_bufs) ? __GFP_PISA_RW : 0;
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -109,6 +117,17 @@ pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
return 0;
}
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+ return 0;
+}
+
+static inline bool pisa_has_free_blocks_rx(const struct device *dev,
+ unsigned int order, int nid)
+{
+ return false;
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (9 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
` (14 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Coherent DMA allocations (e.g. TX/RX descriptor rings allocated via
dma_alloc_attrs() / dma_alloc_coherent()) are long-lived and accessed
on every packet or I/O operation, so they benefit from sharing 2MB IOMMU
leaf mappings.
When dev->pisa_coherent is enabled, serve coherent DMA allocations up
to 2MB from the device's per-NUMA-node order-0 read-write PISA
allocator.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/dma-iommu.c | 10 +++
drivers/iommu/pisa-alloc.c | 146 +++++++++++++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 67 +++++++++++++++++
include/linux/pisa.h | 30 ++++++++
kernel/dma/direct.c | 9 +++
5 files changed, 262 insertions(+)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index eb920ad77c783..a0c5ddab7a8b6 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1658,6 +1658,8 @@ static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr)
} else {
/* Lowmem means a coherent atomic or CMA allocation */
page = virt_to_page(cpu_addr);
+ if (pisa_free_coherent(page, alloc_size))
+ return;
}
if (pages)
@@ -1716,6 +1718,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
gfp_t gfp, unsigned long attrs)
{
bool coherent = dev_is_dma_coherent(dev);
+ struct iommu_domain *domain = iommu_get_dma_domain(dev);
int ioprot = dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs);
bool is_alloc_cc_shared = attrs & __DMA_ATTR_ALLOC_CC_SHARED;
struct page *page = NULL;
@@ -1727,6 +1730,12 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
gfp |= __GFP_ZERO;
+ if (domain) {
+ cpu_addr = pisa_alloc_coherent(&page, dev, domain, size, gfp, attrs);
+ if (cpu_addr)
+ goto map;
+ }
+
if (gfpflags_allow_blocking(gfp) &&
!(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) {
return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs);
@@ -1744,6 +1753,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
return NULL;
}
+map:
*handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot,
dev->coherent_dma_mask);
if (*handle == DMA_MAPPING_ERROR) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index b0d968aa9181c..25e4645310678 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -10,6 +10,8 @@
#include <linux/bitops.h>
#include <linux/cache.h>
#include <linux/device.h>
+#include <linux/dma-direct.h>
+#include <linux/dma-map-ops.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
#include <linux/export.h>
@@ -697,6 +699,150 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
return i;
}
+/* Find and reserve @nblocks aligned contiguous blocks in @meta under @pa->lock. */
+static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblocks,
+ unsigned long align_mask)
+{
+ DECLARE_BITMAP(busy, PISA_MAX_BLOCKS);
+ struct pisa_alloc *pa = meta->pa;
+ unsigned int bpp = PISA_BLOCKS(0), i;
+ bool poisoned = false;
+ unsigned long off;
+ struct page *page;
+
+ if (pisa_meta_avail(meta) < nblocks)
+ return NULL;
+
+ bitmap_or(busy, meta->clean_bitmap, meta->dirty_bitmap, bpp);
+ bitmap_complement(busy, busy, bpp);
+
+ off = 0;
+ for (;;) {
+ off = bitmap_find_next_zero_area(busy, bpp, off, nblocks, align_mask);
+ if (off >= bpp) {
+ if (poisoned) {
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+ pisa_place_meta(meta);
+ }
+ return NULL;
+ }
+
+ page = pfn_to_page(pisa_meta_to_pfn(meta) + off);
+ for (i = 0; i < nblocks; i++) {
+ if (!pisa_block_poisoned(pa, page + i))
+ continue;
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap))
+ meta->nr_clean--;
+ else if (__test_and_clear_bit(off + i, meta->dirty_bitmap))
+ meta->nr_dirty--;
+ __set_bit(off + i, busy);
+ poisoned = true;
+ off += i + 1;
+ break;
+ }
+ if (i == nblocks)
+ break;
+ }
+
+ if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
+ meta->on_list == PISA_LIST_IDLE_DIRTY)
+ pa->num_idle_pages--;
+
+ for (i = 0; i < nblocks; i++) {
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
+ meta->nr_clean--;
+ } else {
+ __clear_bit(off + i, meta->dirty_bitmap);
+ meta->nr_dirty--;
+ }
+ pisa_block_prep(page + i);
+ page_ref_unfreeze(page + i, 1);
+ }
+ pisa_place_meta(meta);
+ return page;
+}
+
+/*
+ * Specialized memory provider for iommu_dma_alloc() and dma_direct_alloc():
+ * allocate a zeroed, naturally aligned contiguous range of order-0 pages for
+ * @size bytes (<= PISA_SIZE) from @dev's per-node coherent allocator into
+ * *@pagep and return its kernel virtual address, or NULL.
+ */
+void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
+ struct iommu_domain *domain, size_t size,
+ gfp_t gfp, unsigned long attrs)
+{
+ bool should_release = false;
+ struct list_head *lists[3];
+ unsigned long align_mask, flags;
+ unsigned int nblocks, i;
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ struct page *page = NULL;
+
+ size = PAGE_ALIGN(size);
+ if (!pagep || !dev || !READ_ONCE(dev->pisa_coherent) ||
+ !dev_is_dma_coherent(dev) || !size || size > PISA_SIZE)
+ return NULL;
+ if (min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) <= DMA_BIT_MASK(32))
+ return NULL;
+ if ((current_gfp_context(gfp) & ~(__GFP_ZERO | __GFP_NOWARN)) != GFP_KERNEL ||
+ (attrs & ~DMA_ATTR_NO_WARN))
+ return NULL;
+
+ if (domain) {
+ struct pisa_window *win = pisa_dma_window(domain);
+
+ if (!win || pisa_window_declined(win) || !(domain->pgsize_bitmap & PISA_SIZE))
+ return NULL;
+ } else if (is_swiotlb_for_alloc(dev) ||
+ force_dma_unencrypted(dev) ||
+ !dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
+ return NULL;
+ }
+
+ pa = pisa_node_pa(dev_to_node(dev), 0, false);
+ if (!pa)
+ return NULL;
+
+ nblocks = size >> PAGE_SHIFT;
+ align_mask = (1UL << get_order(size)) - 1;
+ lists[0] = &pa->partial;
+ lists[1] = &pa->partial_dirty;
+ lists[2] = &pa->idle;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < ARRAY_SIZE(lists) && !page; i++) {
+ list_for_each_entry(meta, lists[i], list) {
+ page = pisa_take_contig(meta, nblocks, align_mask);
+ if (page)
+ break;
+ }
+ }
+ if (!page) {
+ spin_unlock_irqrestore(&pa->lock, flags);
+ meta = pisa_alloc_split_page(0, gfp, pa->nid);
+ if (IS_ERR(meta))
+ return NULL;
+ spin_lock_irqsave(&pa->lock, flags);
+ meta->pa = pa;
+ meta->prot = pa->prot;
+ WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ page = pisa_take_contig(meta, nblocks, align_mask);
+ if (unlikely(!page))
+ should_release = pisa_place_meta(meta);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+ if (!page)
+ return NULL;
+ *pagep = page;
+ return memset(page_address(page), 0, size);
+}
/**
* __pisa_free_page - Recycle a freed block back into its PISA allocator
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index ab64657ad5c58..72ee8986b4802 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -352,6 +352,72 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
}
#endif /* CONFIG_IOMMU_DMA */
+static void test_coherent_alloc_and_free(struct kunit *test)
+{
+ struct device dev = { .init_name = "pisa_kunit", .numa_node = NUMA_NO_NODE,
+ .coherent_dma_mask = DMA_BIT_MASK(64),
+ .pisa_coherent = true };
+ struct page *p1, *p2, *p3;
+ void *v1, *v2, *v3;
+#ifdef CONFIG_IOMMU_DMA
+ struct pisa_window *win;
+ dma_addr_t d1, d2;
+#endif
+
+ dev.dma_mask = &dev.coherent_dma_mask;
+ /* NUMA_NO_NODE would follow the node of the CPU the test runs on. */
+ set_dev_node(&dev, first_memory_node);
+#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \
+ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL)
+ dev.dma_coherent = true;
+#endif
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, NULL, NULL, SZ_4K, GFP_KERNEL, 0));
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, NULL, SZ_4K,
+ GFP_KERNEL | __GFP_ACCOUNT, 0));
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, NULL, PISA_SIZE + SZ_4K,
+ GFP_KERNEL, 0));
+
+#ifdef CONFIG_IOMMU_DMA
+ /* Uninitialized domain has no DMA cookie/window yet. */
+ KUNIT_EXPECT_NULL(test, pisa_alloc_coherent(&p1, &dev, &fake_domain, SZ_4K,
+ GFP_KERNEL | __GFP_NOWARN, 0));
+
+ KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+ win = pisa_dma_window(&fake_domain);
+
+ v1 = pisa_alloc_coherent(&p1, &dev, &fake_domain, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v1);
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d1, &dev, &fake_domain, page_to_phys(p1), SZ_64K,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_TRUE(test, is_pisa_iova(win, d1));
+
+ v2 = pisa_alloc_coherent(&p2, &dev, &fake_domain, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v2);
+ KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+ KUNIT_ASSERT_EQ(test,
+ pisa_map_phys(&d2, &dev, &fake_domain, page_to_phys(p2), SZ_64K,
+ IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE), 0);
+ KUNIT_EXPECT_EQ(test, d2, d1 + SZ_64K);
+#else
+ v1 = pisa_alloc_coherent(&p1, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v1);
+ v2 = pisa_alloc_coherent(&p2, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v2);
+ KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+#endif
+
+ /* Freeing p1 returns its 16 order-0 pages; next 64K alloc reuses them. */
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p1, SZ_64K));
+ v3 = pisa_alloc_coherent(&p3, &dev, NULL, SZ_64K, GFP_KERNEL, 0);
+ KUNIT_ASSERT_NOT_NULL(test, v3);
+ KUNIT_EXPECT_PTR_EQ(test, p3, p1);
+
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p2, SZ_64K));
+ KUNIT_EXPECT_TRUE(test, pisa_free_coherent(p3, SZ_64K));
+}
+
static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
const char *name, bool *field)
{
@@ -431,6 +497,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_window_helpers),
KUNIT_CASE(test_map_and_domain_lifecycle),
#endif
+ KUNIT_CASE(test_coherent_alloc_and_free),
KUNIT_CASE(test_dev_pisa_attrs),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 92482fce943e9..a6556aa403a09 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -13,6 +13,7 @@
#include <linux/rcupdate.h>
struct device;
+struct iommu_domain;
#ifdef CONFIG_PISA
@@ -78,6 +79,21 @@ static inline bool pisa_free_page(struct page *page)
return ret;
}
+void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
+ struct iommu_domain *domain, size_t size,
+ gfp_t gfp, unsigned long attrs);
+
+static inline bool pisa_free_coherent(struct page *page, size_t size)
+{
+ unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+
+ if (!page || !is_pisa_page(page))
+ return false;
+ while (count--)
+ __free_page(page++);
+ return true;
+}
+
unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int preferred_nid, nodemask_t *nodemask,
int nr_pages, struct page **page_array,
@@ -109,6 +125,20 @@ static inline bool pisa_free_page(struct page *page)
return false;
}
+static inline void *pisa_alloc_coherent(struct page **pagep,
+ struct device *dev,
+ struct iommu_domain *domain,
+ size_t size, gfp_t gfp,
+ unsigned long attrs)
+{
+ return NULL;
+}
+
+static inline bool pisa_free_coherent(struct page *page, size_t size)
+{
+ return false;
+}
+
static inline unsigned long
pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 25e9a86835fd3..11052b65e573b 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -15,6 +15,7 @@
#include <linux/slab.h>
#include <linux/pci-p2pdma.h>
#include <linux/cc_platform.h>
+#include <linux/pisa.h>
#include "direct.h"
@@ -244,6 +245,12 @@ void *dma_direct_alloc(struct device *dev, size_t size,
if (attrs & DMA_ATTR_NO_WARN)
gfp |= __GFP_NOWARN;
+ cpu_addr = pisa_alloc_coherent(&page, dev, NULL, size, gfp, attrs);
+ if (cpu_addr) {
+ *dma_handle = phys_to_dma_direct(dev, page_to_phys(page), false);
+ return cpu_addr;
+ }
+
if (dma_direct_use_no_mapping(dev, attrs))
return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp);
@@ -424,6 +431,8 @@ void dma_direct_free(struct device *dev, size_t size,
if (is_vmalloc_addr(cpu_addr)) {
vunmap(cpu_addr);
+ } else if (pisa_free_coherent(virt_to_page(cpu_addr), size)) {
+ return;
} else {
if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED))
arch_dma_clear_uncached(cpu_addr, size);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (10 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
` (13 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Every block allocation and free takes the allocator's lock. Highly
concurrent users e.g. socket tx fragments, can cause significant lock
contention.
Add pisa_enable_pcp(), an opt-in per-CPU front cache, similar
to the page allocator's per-CPU lists:
- Each per-CPU cache keeps up to 64 free blocks, and refills it from
the main allocator in batches of 32.
- returned blocks belonging to the local NUMA node are kept in the local
cache.
- returned blocks belonging to a remote NUMA node are returned to the
proper per-node allocator.
The cache is fully integrated with the shrinker and CPU offlining to
drain the cache.
Enable per-CPU caches for the per-node order-0 RW and SKB_FRAG_PAGE_ORDER
RO allocators.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 342 +++++++++++++++++++++++++++++++++++--
drivers/iommu/pisa-kunit.c | 78 +++++++++
drivers/iommu/pisa-priv.h | 35 +++-
include/linux/pisa.h | 6 +
4 files changed, 448 insertions(+), 13 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 25e4645310678..913706a503341 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/cpuhotplug.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
#include <linux/dma-map-ops.h>
@@ -24,6 +25,7 @@
#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/page_owner.h>
+#include <linux/percpu.h>
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
@@ -60,6 +62,10 @@ module_param_named(max_idle, pisa_max_idle, uint, 0644);
static unsigned int pisa_idle_ms __read_mostly = 200;
module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
+/* Stop recycling partial-page blocks when free blocks exceed @pisa_compact PISA pages. */
+static unsigned int pisa_compact __read_mostly;
+module_param_named(compact, pisa_compact, uint, 0644);
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -90,7 +96,11 @@ static void pisa_release_page_rcu(struct rcu_head *head)
struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
struct page *block, *page = pfn_to_page(pisa_meta_to_pfn(meta));
unsigned int i, order = meta->pa->order, slot = meta->slot;
+ unsigned long flags;
+ spin_lock_irqsave(&meta->pa->lock, flags);
+ meta->pa->stats.page_free_cnt++;
+ spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
/* Possibly still mapped by a device: leak it, still charged. */
@@ -355,9 +365,223 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
__set_bit(idx, meta->dirty_bitmap);
meta->nr_dirty++;
}
+ pa->blocks_in_use--;
*release |= pisa_place_meta(meta);
}
+/**
+ * pisa_page_can_recycle - Check if a block can stay cached in a caller pool
+ * @page: Block head page
+ *
+ * Called on the recycle fast path (per-CPU cache, TX page_frag, page_pool).
+ * Fast-path cost is a static-key branch plus a READ_ONCE() of module parameters
+ * (~1-2 ns when compaction is off); when enabled, it does a lockless RCU +
+ * metadata array lookup (~5-10 ns).
+ *
+ * Return: true if @page can be recycled in place, false if it should be
+ * released back to its allocator so sparse 2MB pages can drain.
+ */
+bool pisa_page_can_recycle(const struct page *page)
+{
+ unsigned long bpp, in_use, free_blocks, pfn;
+ const struct pisa_meta *meta;
+ const struct pisa_alloc *pa;
+ unsigned int compact;
+ bool ret = true;
+ u8 on_list;
+ long held;
+
+ if (!static_branch_unlikely(&pisa_key))
+ return true;
+
+ compact = READ_ONCE(pisa_compact);
+ if (!compact)
+ return true;
+
+ pfn = page_to_pfn(page);
+ rcu_read_lock();
+ if (!is_pisa_pfn(pfn))
+ goto out;
+
+ meta = pisa_meta_of_pfn(pfn);
+ on_list = READ_ONCE(meta->on_list);
+ if (on_list == PISA_LIST_FULL || on_list == PISA_LIST_NONE)
+ goto out;
+
+ pa = meta->pa;
+ if (!pa)
+ goto out;
+
+ /* Do not drain the partial page currently being filled at list head. */
+ if (READ_ONCE(pa->partial_dirty.next) == &meta->list ||
+ READ_ONCE(pa->partial.next) == &meta->list)
+ goto out;
+
+ bpp = PISA_BLOCKS(pa->order);
+ /* Do not drain dense pages that still have more than half their blocks in use. */
+ if (READ_ONCE(meta->nr_clean) + READ_ONCE(meta->nr_dirty) < (bpp >> 1))
+ goto out;
+
+ held = pisa_alloc_held_pages(pa);
+ in_use = READ_ONCE(pa->blocks_in_use);
+ free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
+ (unsigned long)held * bpp - in_use : 0;
+
+ ret = free_blocks <= (unsigned long)compact * bpp;
+out:
+ rcu_read_unlock();
+ return ret;
+}
+EXPORT_SYMBOL_GPL(pisa_page_can_recycle);
+
+/* Return @n freed blocks to @pa under a single acquisition of @pa->lock. */
+static void pisa_free_bulk(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+{
+ bool release = false;
+ unsigned long flags;
+ unsigned int i;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < n; i++) {
+ if (likely(!pisa_block_poisoned(pa, blocks[i])))
+ pisa_return_block_locked(pa, blocks[i], &release);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (release)
+ pisa_schedule_reclaim();
+}
+
+/*
+ * Per-CPU block cache: each CPU caches up to PISA_PCP_HIGH free blocks
+ * (refcount 0, not in bitmaps), refilling and flushing in PISA_PCP_BATCH
+ * batches. Drained by the shrinker and CPU hotplug.
+ */
+
+/* Max blocks moved per per-CPU cache refill or flush (1..PISA_PCP_BATCH). */
+static unsigned int pisa_pcp_batch __read_mostly = PISA_PCP_BATCH;
+
+static int pisa_pcp_batch_set(const char *val, const struct kernel_param *kp)
+{
+ return param_set_uint_minmax(val, kp, 1, PISA_PCP_BATCH);
+}
+
+static const struct kernel_param_ops pisa_pcp_batch_ops = {
+ .set = pisa_pcp_batch_set,
+ .get = param_get_uint,
+};
+module_param_cb(pcp_batch, &pisa_pcp_batch_ops, &pisa_pcp_batch, 0644);
+
+/* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
+static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+{
+ unsigned int i, batch = READ_ONCE(pisa_pcp_batch);
+ struct pisa_pcp *pcp;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ pcp = this_cpu_ptr(pa->pcp);
+ spin_lock(&pcp->lock);
+ for (i = 0; i < n; i++) {
+ if (pcp->count == PISA_PCP_HIGH) {
+ /* Keep the most recently freed, likely still in cache. */
+ pisa_free_bulk(pa, pcp->blocks, batch);
+ pcp->count -= batch;
+ memmove(pcp->blocks, pcp->blocks + batch,
+ pcp->count * sizeof(*pcp->blocks));
+ }
+ pcp->blocks[pcp->count++] = blocks[i];
+ }
+ spin_unlock(&pcp->lock);
+ local_irq_restore(flags);
+}
+
+/* Take up to @nr blocks, at refcount 0, from this CPU's cache. */
+static unsigned int pisa_pcp_pop(struct pisa_alloc *pa, unsigned int nr, struct page **out)
+{
+ struct pisa_pcp *pcp;
+ unsigned int got = 0;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ pcp = this_cpu_ptr(pa->pcp);
+ spin_lock(&pcp->lock);
+ while (got < nr && pcp->count)
+ out[got++] = pcp->blocks[--pcp->count];
+ spin_unlock(&pcp->lock);
+ local_irq_restore(flags);
+
+ return got;
+}
+
+/* Return the blocks @cpu caches for @pa. Process context. */
+static void pisa_pcp_drain_cpu(struct pisa_alloc *pa, int cpu)
+{
+ struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+ unsigned long flags;
+
+ spin_lock_irqsave(&pcp->lock, flags);
+ if (pcp->count) {
+ pisa_free_bulk(pa, pcp->blocks, pcp->count);
+ pcp->count = 0;
+ }
+ spin_unlock_irqrestore(&pcp->lock, flags);
+}
+
+/* Drain every non-empty CPU cache of @pa. */
+void pisa_pcp_drain(struct pisa_alloc *pa)
+{
+ int cpu;
+
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (!smp_load_acquire(&pa->pcp))
+ return;
+ for_each_possible_cpu(cpu) {
+ if (!READ_ONCE(per_cpu_ptr(pa->pcp, cpu)->count))
+ continue;
+ pisa_pcp_drain_cpu(pa, cpu);
+ }
+}
+
+/* Blocks in @pa's caches, for shrinker and debugfs. */
+static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+{
+ unsigned long n = 0;
+ int cpu;
+
+ if (!pa->pcp)
+ return 0;
+ for_each_possible_cpu(cpu) {
+ /* A racy sum is good enough for reclaim and statistics. */
+ n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+ }
+ return n;
+}
+
+/**
+ * pisa_enable_pcp - Put per-CPU block caches in front of an allocator
+ * @pa: Allocator to cache
+ *
+ * Return: 0, or -ENOMEM; @pa works without the cache.
+ */
+int pisa_enable_pcp(struct pisa_alloc *pa)
+{
+ struct pisa_pcp __percpu *pcp;
+ int cpu;
+
+ if (!pa || pa->pcp)
+ return 0;
+ pcp = alloc_percpu(struct pisa_pcp);
+ if (!pcp)
+ return -ENOMEM;
+ for_each_possible_cpu(cpu)
+ spin_lock_init(&per_cpu_ptr(pcp, cpu)->lock);
+ /* Publish initialized locks; readers depend on the pointer. */
+ if (cmpxchg_release(&pa->pcp, NULL, pcp))
+ free_percpu(pcp);
+ return 0;
+}
+
/*
* Return the predefined per-node allocator for (@nid, @order, @ro), or NULL.
*/
@@ -412,7 +636,9 @@ bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int n
return false;
pa = pisa_node_pa(nid, order, false);
return pa &&
- (!list_empty_careful(&pa->partial) ||
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ ((smp_load_acquire(&pa->pcp) && raw_cpu_read(pa->pcp->count)) ||
+ !list_empty_careful(&pa->partial) ||
!list_empty_careful(&pa->partial_dirty) ||
!list_empty_careful(&pa->idle));
}
@@ -532,6 +758,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
break;
}
*countp -= used;
+ pa->blocks_in_use += used;
return got;
}
@@ -638,6 +865,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
meta->prot = pa->prot;
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ pa->stats.page_alloc_cnt++;
}
spin_unlock_irqrestore(&pa->lock, flags);
@@ -656,6 +884,49 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
return n;
}
+/* Allocate @nr_pages blocks through this CPU's cache, refilling in batches on miss. */
+static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+ struct page **page_array, bool frozen)
+{
+ unsigned int pcp_batch = READ_ONCE(pisa_pcp_batch);
+ unsigned long got, n, i, j;
+
+ got = pisa_pcp_pop(pa, nr_pages, page_array);
+ if (got < nr_pages) {
+ if (nr_pages - got >= pcp_batch) {
+ got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
+ page_array + got, false, true);
+ } else {
+ struct page *batch[PISA_PCP_BATCH];
+
+ n = __pisa_alloc_bulk(pa, gfp, pcp_batch, batch, false, true);
+ for (i = 0; i < n && got < nr_pages; i++)
+ page_array[got++] = batch[i];
+ if (i < n)
+ pisa_pcp_push(pa, batch + i, n - i);
+ }
+ }
+
+ for (i = j = 0; i < got; i++) {
+ struct page *block = page_array[i];
+
+ page_array[i] = NULL;
+ /* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
+ if (pisa_block_poisoned(pa, block))
+ continue;
+ pisa_block_prep(block);
+ if (want_init_on_alloc(gfp)) {
+ memset(page_address(block), 0, PAGE_SIZE << pa->order);
+ if (frozen) /* Order zeroing before caller publishes the page. */
+ smp_wmb();
+ }
+ if (!frozen)
+ page_ref_unfreeze(block, 1);
+ page_array[j++] = block;
+ }
+ return j;
+}
+
/**
* pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
* @pa: Allocator to allocate from
@@ -681,11 +952,16 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
i++;
continue;
}
+
/* Fill the run of @n NULL entries at @i. */
for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
;
- got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
- want_init_on_alloc(gfp), frozen);
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp))
+ got = pisa_pcp_alloc(pa, gfp, n, page_array + i, frozen);
+ else
+ got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+ want_init_on_alloc(gfp), frozen);
i += got;
if (got < n)
break;
@@ -733,10 +1009,13 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
for (i = 0; i < nblocks; i++) {
if (!pisa_block_poisoned(pa, page + i))
continue;
- if (__test_and_clear_bit(off + i, meta->clean_bitmap))
+ if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
meta->nr_clean--;
- else if (__test_and_clear_bit(off + i, meta->dirty_bitmap))
+ pa->blocks_in_use++;
+ } else if (__test_and_clear_bit(off + i, meta->dirty_bitmap)) {
meta->nr_dirty--;
+ pa->blocks_in_use++;
+ }
__set_bit(off + i, busy);
poisoned = true;
off += i + 1;
@@ -760,6 +1039,7 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
pisa_block_prep(page + i);
page_ref_unfreeze(page + i, 1);
}
+ pa->blocks_in_use += nblocks;
pisa_place_meta(meta);
return page;
}
@@ -806,6 +1086,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
pa = pisa_node_pa(dev_to_node(dev), 0, false);
if (!pa)
return NULL;
+ pisa_pcp_drain(pa);
nblocks = size >> PAGE_SHIFT;
align_mask = (1UL << get_order(size)) - 1;
@@ -830,6 +1111,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
meta->pa = pa;
meta->prot = pa->prot;
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
+ pa->stats.page_alloc_cnt++;
page = pisa_take_contig(meta, nblocks, align_mask);
if (unlikely(!page))
should_release = pisa_place_meta(meta);
@@ -876,6 +1158,17 @@ bool __pisa_free_page(struct page *page)
if (want_init_on_free())
memset(page_address(page), 0, PAGE_SIZE << pa->order);
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
+ page_to_nid(page) == numa_mem_id()) {
+ if (pisa_page_can_recycle(page)) {
+ pisa_pcp_push(pa, &page, 1);
+ return true;
+ }
+ if (this_cpu_read(pa->pcp->count))
+ pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
+ }
+
spin_lock_irqsave(&pa->lock, flags);
pisa_return_block_locked(pa, page, &should_release);
spin_unlock_irqrestore(&pa->lock, flags);
@@ -889,15 +1182,17 @@ bool __pisa_free_page(struct page *page)
/* Shrinker: releases reservoir pages first, then idle allocator pages. */
static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
{
- unsigned long nr = pisa_reservoir_count();
+ unsigned long nr = pisa_reservoir_count() << PISA_ORDER;
struct pisa_alloc *pa;
read_lock(&pisa_allocs_lock);
- list_for_each_entry(pa, &pisa_allocs, node)
- nr += READ_ONCE(pa->num_idle_pages);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
+ nr += pisa_pcp_count(pa) << pa->order;
+ }
read_unlock(&pisa_allocs_lock);
- return nr << PISA_ORDER;
+ return nr;
}
/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
@@ -930,6 +1225,8 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
if (freed < sc->nr_to_scan) {
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
+ /* Cached blocks keep PISA pages from going idle. */
+ pisa_pcp_drain(pa);
freed += pisa_shrink_list(&pa->lock, &pa->idle,
&pa->num_idle_pages, &release_list,
sc->nr_to_scan - freed);
@@ -952,15 +1249,30 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
return freed ?: SHRINK_STOP;
}
+/* Return the blocks a dead CPU cached, which nothing else would. */
+static int pisa_cpu_dead(unsigned int cpu)
+{
+ struct pisa_alloc *pa;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node)
+ /* Pairs with smp_store_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp))
+ pisa_pcp_drain_cpu(pa, cpu);
+ read_unlock(&pisa_allocs_lock);
+ return 0;
+}
+
static int __init pisa_init(void)
{
const struct {
u8 order;
bool ro;
+ bool pcp;
} cfg[] = {
- { 0, false },
- { 2, true },
- { SKB_FRAG_PAGE_ORDER, true },
+ { 0, false, true },
+ { 2, true, false },
+ { SKB_FRAG_PAGE_ORDER, true, true },
};
struct shrinker *shrink;
int nid, i;
@@ -986,6 +1298,8 @@ static int __init pisa_init(void)
if (!pa)
continue;
pisa_set_name(pa, "pisa%s_o%u_n%d", ro ? "_ro" : "", order, nid);
+ if (cfg[i].pcp)
+ pisa_enable_pcp(pa);
pisa_nodes[nid].pa[ro][order] = pa;
}
}
@@ -995,6 +1309,10 @@ static int __init pisa_init(void)
if (!pisa_wq)
pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+ if (cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN, "iommu/pisa:dead", NULL,
+ pisa_cpu_dead) < 0)
+ pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
+
shrink = shrinker_alloc(0, "pisa");
if (!shrink) {
pr_warn("pisa: shrinker registration failed\n");
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 72ee8986b4802..5bbb540eb505e 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -482,6 +482,82 @@ static void test_dev_pisa_attrs(struct kunit *test)
device_unregister(dev);
}
+static void test_pcp_cache(struct kunit *test)
+{
+ struct page *blocks[PISA_PCP_HIGH + 1];
+ struct pisa_alloc *pa;
+ struct page *p1, *p2;
+ unsigned int i, n = 0;
+
+ KUNIT_EXPECT_EQ(test, pisa_enable_pcp(NULL), 0);
+ pa = pisa_node_pa(first_memory_node, 0, false);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ KUNIT_ASSERT_EQ(test, pisa_enable_pcp(pa), 0);
+
+ p1 = pisa_alloc(pa, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, p1);
+ __free_page(p1);
+ p2 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+ KUNIT_EXPECT_PTR_EQ(test, p2, p1);
+ if (p2)
+ __free_page(p2);
+
+ /* Overflowing PISA_PCP_HIGH flushes a batch back to the allocator. */
+ for (i = 0; i < ARRAY_SIZE(blocks); i++) {
+ blocks[i] = pisa_alloc(pa, GFP_KERNEL);
+ if (!blocks[i])
+ break;
+ n++;
+ }
+ KUNIT_EXPECT_EQ(test, n, (unsigned int)ARRAY_SIZE(blocks));
+ for (i = 0; i < n; i++)
+ __free_page(blocks[i]);
+}
+
+static void test_gfp_pisa(struct kunit *test)
+{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct device dev = { .numa_node = first_memory_node };
+ struct page *p, *p_ro, *bulk[4] = {};
+ unsigned long n;
+ int i;
+
+ pisa_shrink_scan(NULL, &sc);
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(NULL), 0);
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), 0);
+ KUNIT_EXPECT_FALSE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+
+ dev.pisa_rx_bufs = true;
+ KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), __GFP_PISA_RW);
+
+ p = alloc_pages_node(first_memory_node, GFP_KERNEL | pisa_rx_gfp(&dev), 0);
+ KUNIT_ASSERT_NOT_NULL(test, p);
+ if (p) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p));
+ KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p))->prot,
+ IOMMU_READ | IOMMU_WRITE);
+ __free_page(p);
+ KUNIT_EXPECT_TRUE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+ }
+
+ p_ro = alloc_pages(GFP_KERNEL | __GFP_COMP | __GFP_PISA_RO, 3);
+ KUNIT_ASSERT_NOT_NULL(test, p_ro);
+ if (p_ro) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(p_ro));
+ KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p_ro))->prot,
+ IOMMU_READ);
+ __free_pages(p_ro, 3);
+ }
+
+ n = alloc_pages_bulk_node(GFP_KERNEL | __GFP_PISA_RW, first_memory_node,
+ ARRAY_SIZE(bulk), bulk);
+ KUNIT_EXPECT_EQ(test, n, ARRAY_SIZE(bulk));
+ for (i = 0; i < n; i++) {
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(bulk[i]));
+ __free_page(bulk[i]);
+ }
+}
+
static void test_reservoir_and_scrubber(struct kunit *test)
{
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -499,6 +575,8 @@ static struct kunit_case pisa_meta_test_cases[] = {
#endif
KUNIT_CASE(test_coherent_alloc_and_free),
KUNIT_CASE(test_dev_pisa_attrs),
+ KUNIT_CASE(test_pcp_cache),
+ KUNIT_CASE(test_gfp_pisa),
KUNIT_CASE(test_reservoir_and_scrubber),
{}
};
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index e5ca95d2654cd..8f87bdf7d4ee2 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -53,6 +53,7 @@ enum {
* Lock hierarchy:
* - pisa_allocs_lock (rwlock): outermost; protects pisa_allocs list.
* - pisa_meta_mutex (mutex): serializes pisa_meta_init() and page-table alloc.
+ * - pcp->lock (spinlock, irqsave): per-CPU block cache.
* - pa->lock (spinlock, irqsave): per-allocator lists, bitmaps, and stats.
* - pisa_domains[].map_lock (spinlock, irqsave): first-time iommu_map() in __pisa_map_phys().
* - pisa_domains_lock (spinlock, irqsave): leaf lock over pisa_domains[]/windows.
@@ -105,11 +106,28 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
return meta->nr_clean + meta->nr_dirty;
}
+/* Per-CPU front cache of free blocks at refcount 0, see pisa_enable_pcp(). */
+#define PISA_PCP_HIGH 64U /* blocks cached per CPU at most */
+#define PISA_PCP_BATCH 32U /* blocks moved per refill or flush */
+
+struct pisa_pcp {
+ spinlock_t lock; /* by this CPU, or by a drain from any CPU */
+ unsigned int count;
+ struct page *blocks[PISA_PCP_HIGH];
+};
+
+/* Per-allocator statistics, see pisa_allocs_show(). */
+struct pisa_alloc_stats {
+ u64 page_alloc_cnt;
+ u64 page_free_cnt;
+};
+
/**
* struct pisa_alloc - Allocator managing 2MB-backed blocks of a fixed order
- * @lock: Spinlock protecting lists and block bitmaps
+ * @lock: Spinlock protecting lists, block bitmaps, and u64 @stats
* @order: Block order managed by this allocator (<= PISA_ORDER)
* @prot: IOMMU_READ and/or IOMMU_WRITE from @dir passed to pisa_create()
+ * @pcp: Per-CPU block caches, or NULL (see pisa_enable_pcp())
* @partial: Partially used PISA pages with @nr_clean > 0
* @partial_dirty: Partially used PISA pages with @nr_clean == 0 and @nr_dirty > 0
* @idle: Completely free PISA pages (reclaimable under pressure)
@@ -117,15 +135,18 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
* @num_idle_pages: Length of @idle
* @max_idle_pages: High watermark of @idle before async release to the buddy allocator
* @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @blocks_in_use: Blocks handed out and not yet freed
* @nid: NUMA node id
* @node: Node in the global pisa_allocs list
* @name: Debug name (defaults to caller of pisa_create())
+ * @stats: Per-allocator statistics
*/
struct pisa_alloc {
/* First cacheline: hot fields touched on block alloc and free. */
spinlock_t lock;
u8 order;
u8 prot;
+ struct pisa_pcp __percpu *pcp;
struct list_head partial;
struct list_head partial_dirty;
struct list_head idle;
@@ -133,12 +154,22 @@ struct pisa_alloc {
unsigned int num_idle_pages;
unsigned int max_idle_pages;
unsigned long idle_jiffies;
+ unsigned long blocks_in_use;
/* Cold. */
int nid;
struct list_head node;
char name[48];
+ struct pisa_alloc_stats stats;
} ____cacheline_aligned;
+/* Active non-idle PISA pages held by @pa, for soft compaction checks. */
+static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
+{
+ return (long)(READ_ONCE(pa->stats.page_alloc_cnt) -
+ READ_ONCE(pa->stats.page_free_cnt)) -
+ READ_ONCE(pa->num_idle_pages);
+}
+
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
@@ -202,6 +233,8 @@ struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direct
struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool ro);
__printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+int pisa_enable_pcp(struct pisa_alloc *pa);
+void pisa_pcp_drain(struct pisa_alloc *pa);
void pisa_release_page(struct pisa_meta *meta);
unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
unsigned long nr_pages, struct page **page_array, bool frozen);
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index a6556aa403a09..cdf176464d74a 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -100,6 +100,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
bool frozen);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
+bool pisa_page_can_recycle(const struct page *page);
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
@@ -139,6 +140,11 @@ static inline bool pisa_free_coherent(struct page *page, size_t size)
return false;
}
+static inline bool pisa_page_can_recycle(const struct page *page)
+{
+ return true;
+}
+
static inline unsigned long
pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill()
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (11 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
` (12 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Transmit socket buffers use memory coming from skb_page_frag_refill().
Sockets whose leaf device has enabled pisa_tx_bufs should use PISA
memory.
Add a dedicated per-task page_frag, current->task_frag_pisa, populated
with read-only PISA blocks via __GFP_PISA_RO, and select it on TX when
the socket's (or cork's) route goes out in the clear through a PISA-enabled
device.
Because walking lower devices can be expensive (e.g. bonded devices),
cache the leaf device check on the net_device and refresh it at most once
every ~250ms. Special paths that hold plaintext before encryption (IPsec,
kTLS) or do not use a DMA device keep using regular pages;
ESP output uses skb_page_frag_refill_dev() to opt in when the
egress device has pisa_tx_bufs enabled.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 24 +++++++---
drivers/iommu/pisa-alloc.c | 57 ++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 3 +-
include/linux/device.h | 9 ++++
include/linux/netdevice.h | 4 ++
include/linux/pisa.h | 20 +++++++++
include/linux/sched.h | 4 ++
include/linux/skbuff.h | 2 +
include/net/sock.h | 89 ++++++++++++++++++++++++++++++++++++++
kernel/exit.c | 2 +
kernel/fork.c | 2 +
net/core/sock.c | 41 ++++++++++++++++++
net/ipv4/esp4.c | 4 +-
net/ipv4/ip_output.c | 11 +++--
net/ipv4/tcp.c | 3 +-
net/ipv6/esp6.c | 4 +-
net/ipv6/ip6_output.c | 11 +++--
17 files changed, 270 insertions(+), 20 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 97cd2d03ff77c..ade0c94b02ae7 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2909,7 +2909,8 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
static const DEVICE_ATTR_RO(removable);
/* Per-device configuration to selectively enable PISA */
-static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
+static ssize_t pisa_attr_store(struct device *dev, const char *buf, size_t count,
+ bool *field, int (*setter)(struct device *, bool))
{
bool val;
int ret = kstrtobool(buf, &val);
@@ -2921,11 +2922,17 @@ static ssize_t pisa_attr_store(const char *buf, size_t count, bool *field)
if (ret < 0)
return ret;
}
- WRITE_ONCE(*field, val);
+ if (setter) {
+ ret = setter(dev, val);
+ if (ret < 0)
+ return ret;
+ } else {
+ WRITE_ONCE(*field, val);
+ }
return count;
}
-#define PISA_BOOL_ATTR(name) \
+#define PISA_SET_ATTR(name, setter) \
static ssize_t name##_show(struct device *dev, \
struct device_attribute *attr, char *buf) \
{ \
@@ -2935,13 +2942,15 @@ static ssize_t name##_store(struct device *dev, \
struct device_attribute *attr, \
const char *buf, size_t count) \
{ \
- return pisa_attr_store(buf, count, &dev->name); \
+ return pisa_attr_store(dev, buf, count, &dev->name, setter); \
} \
static DEVICE_ATTR_RW(name)
+#define PISA_BOOL_ATTR(name) PISA_SET_ATTR(name, NULL)
+
PISA_BOOL_ATTR(pisa_coherent);
PISA_BOOL_ATTR(pisa_rx_bufs);
-PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
PISA_BOOL_ATTR(pisa_tx_skb_heads);
static struct attribute *dev_attr_pisa[] = {
@@ -3100,8 +3109,11 @@ static void device_remove_attrs(struct device *dev)
const struct class *class = dev->class;
const struct device_type *type = dev->type;
- if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+ if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
device_remove_group(dev, &dev_attr_pisa_group);
+ if (dev->pisa_tx_bufs)
+ net_pisa_tx_bufs_set(dev, false);
+ }
if (dev->physical_location) {
device_remove_group(dev, &dev_attr_physical_location_group);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 913706a503341..7b9d0a7acc101 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -644,6 +644,63 @@ bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int n
}
EXPORT_SYMBOL(pisa_has_free_blocks_rx);
+#ifdef CONFIG_NET
+/* Enabled while at least one device has pisa_tx_bufs set. */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+EXPORT_SYMBOL(net_pisa_tx_bufs_key);
+
+int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+ static DEFINE_MUTEX(lock);
+
+ mutex_lock(&lock);
+ if (dev->pisa_tx_bufs != val) {
+ WRITE_ONCE(dev->pisa_tx_bufs, val);
+ if (val)
+ static_branch_inc(&net_pisa_tx_bufs_key);
+ else
+ static_branch_dec(&net_pisa_tx_bufs_key);
+ }
+ mutex_unlock(&lock);
+ return 0;
+}
+
+/*
+ * The sysfs attribute pisa_tx_bufs lives on the physical DMA parent
+ * (dev->dev.parent), whereas a socket route (dst->dev) may point to an upper
+ * virtual net_device (bond, VLAN, ipvlan, etc.). Walking lower devices via
+ * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
+ * expensive, so dev->pisa_tx_stamp caches the walk result on the net_device
+ * and refreshes it at most once every 256 jiffies:
+ * bit 0 (PISA_TX_STAMP_OK): pisa_tx_bufs enabled on lower device(s)
+ * bits 31..8 (PISA_TX_STAMP_MASK): upper 24 bits of (u32)jiffies
+ */
+static int netdev_pisa_tx_bufs_walk(struct net_device *dev, struct netdev_nested_priv *priv)
+{
+ return dev->dev.parent && READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+}
+
+/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
+bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+{
+ bool ok;
+
+ if (dev->dev.parent) {
+ ok = READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ } else {
+ rcu_read_lock();
+ ok = netdev_walk_all_lower_dev_rcu(dev,
+ netdev_pisa_tx_bufs_walk,
+ NULL) != 0;
+ rcu_read_unlock();
+ }
+ now |= ok ? PISA_TX_STAMP_OK : 0;
+ WRITE_ONCE(dev->pisa_tx_stamp, now);
+ return now & PISA_TX_STAMP_OK;
+}
+EXPORT_SYMBOL(netdev_tx_pisa_refresh);
+#endif /* CONFIG_NET */
+
/*
* Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
* pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5bbb540eb505e..5f6b2ebcab1d9 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -477,7 +477,8 @@ static void test_dev_pisa_attrs(struct kunit *test)
}
kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
- kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+ if (IS_ENABLED(CONFIG_NET))
+ kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
device_unregister(dev);
}
diff --git a/include/linux/device.h b/include/linux/device.h
index b3c11e32116ca..08f511b978ded 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -814,6 +814,15 @@ struct device {
DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
};
+#if defined(CONFIG_NET) && defined(CONFIG_PISA)
+int net_pisa_tx_bufs_set(struct device *dev, bool val);
+#else
+static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+ return -EOPNOTSUPP;
+}
+#endif
+
#define __create_dev_flag_accessors(accessor_name, flag_name) \
static inline bool dev_##accessor_name(const struct device *dev) \
{ \
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e5..41c999223a15e 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2161,6 +2161,7 @@ enum netdev_reg_state {
* @neighbours: List heads pointing to this device's neighbours'
* dev_list, one per address-family.
* @hwprov: Tracks which PTP performs hardware packet time stamping.
+ * @pisa_tx_stamp: Cached leaf pisa flags and timestamp.
*
* FIXME: cleanup struct net_device such that network protocol info
* moves out.
@@ -2276,6 +2277,9 @@ struct net_device {
} adj_list;
/* Read-mostly cache-line for fast-path access */
+#ifdef CONFIG_PISA
+ u32 pisa_tx_stamp;
+#endif
xdp_features_t xdp_features;
const struct xdp_metadata_ops *xdp_metadata_ops;
const struct xsk_tx_metadata_ops *xsk_tx_metadata_ops;
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index cdf176464d74a..822e1556d78ef 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -107,6 +107,18 @@ static inline gfp_t pisa_rx_gfp(const struct device *dev)
return dev && READ_ONCE(dev->pisa_rx_bufs) ? __GFP_PISA_RW : 0;
}
+/* current->task_frag_pisa lifetime, see task_page_frag(). */
+static inline void pisa_task_frag_init(struct task_struct *tsk)
+{
+ tsk->task_frag_pisa.page = NULL;
+}
+
+static inline void pisa_task_frag_release(struct task_struct *tsk)
+{
+ if (tsk->task_frag_pisa.page)
+ put_page(tsk->task_frag_pisa.page);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -164,6 +176,14 @@ static inline bool pisa_has_free_blocks_rx(const struct device *dev,
return false;
}
+static inline void pisa_task_frag_init(struct task_struct *tsk)
+{
+}
+
+static inline void pisa_task_frag_release(struct task_struct *tsk)
+{
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 1c93da75226e3..1c7ef0d90dc35 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1453,6 +1453,10 @@ struct task_struct {
struct pipe_inode_info *splice_pipe;
struct page_frag task_frag;
+#ifdef CONFIG_PISA
+ /* TX frag that may hold PISA pages, see sk_page_frag_dst(). */
+ struct page_frag task_frag_pisa;
+#endif
#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
struct lazy_mmu_state lazy_mmu_state;
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 6787fcf6ac3e4..2fce6eafa1cb3 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -3808,6 +3808,8 @@ static inline void skb_frag_page_copy(skb_frag_t *fragto,
}
bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t prio);
+bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
+ const struct sk_buff *skb);
/**
* __skb_frag_dma_map - maps a paged fragment via the DMA API
diff --git a/include/net/sock.h b/include/net/sock.h
index 173b37b2b8bb0..73906351d11db 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -2670,6 +2670,95 @@ static inline struct page_frag *sk_page_frag(struct sock *sk)
return &sk->sk_frag;
}
+/* 256-jiffy cache in dev->pisa_tx_stamp; see netdev_tx_pisa_refresh(). */
+#define PISA_TX_STAMP_OK BIT(0)
+#define PISA_TX_STAMP_MASK (~0xffU)
+
+#ifdef CONFIG_PISA
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+
+static inline bool net_pisa_tx_bufs_enabled(void)
+{
+ return static_branch_unlikely(&net_pisa_tx_bufs_key);
+}
+
+bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+{
+ u32 stamp, now;
+
+ if (!dev)
+ return false;
+ stamp = READ_ONCE(dev->pisa_tx_stamp);
+ now = (u32)jiffies & PISA_TX_STAMP_MASK;
+ if (likely((stamp & PISA_TX_STAMP_MASK) == now))
+ return stamp & PISA_TX_STAMP_OK;
+ return netdev_tx_pisa_refresh((struct net_device *)dev, now);
+}
+#else
+static inline bool net_pisa_tx_bufs_enabled(void)
+{
+ return false;
+}
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+{
+ return false;
+}
+#endif
+
+/*
+ * May TX payload routed through @dst be copied into PISA pages? Those pages
+ * stay readable by every device that mapped them for as long as they are
+ * active, so only data that leaves the host in clear qualifies: IPsec payload
+ * is still plaintext here.
+ */
+static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+{
+ bool ok = false;
+
+ if (!net_pisa_tx_bufs_enabled() || !dst)
+ return false;
+
+ rcu_read_lock();
+ if (!dst_xfrm(dst))
+ ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst));
+ rcu_read_unlock();
+ return ok;
+}
+
+/* Is @pfrag the task's frag reserved for PISA-eligible data? */
+static inline bool is_pisa_pfrag(const struct page_frag *pfrag)
+{
+#ifdef CONFIG_PISA
+ return pfrag == ¤t->task_frag_pisa;
+#else
+ return false;
+#endif
+}
+
+/* current->task_frag, or its PISA twin if @dst qualifies. */
+static inline struct page_frag *task_page_frag(const struct dst_entry *dst)
+{
+#ifdef CONFIG_PISA
+ if (dst_tx_pisa_ok(dst))
+ return ¤t->task_frag_pisa;
+#endif
+ return ¤t->task_frag;
+}
+
+/*
+ * sk_page_frag() for data routed through @dst. PISA pages only ever enter
+ * current->task_frag_pisa, so data of sockets that do not qualify (or that use
+ * sk_page_frag() directly: kTLS, sockmap, KCM, MPTCP) never lands in them.
+ */
+static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
+ const struct dst_entry *dst)
+{
+ return sk->sk_use_task_frag ? task_page_frag(dst) : &sk->sk_frag;
+}
+
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag);
static inline bool __sock_writeable(const struct sock *sk, int wmem_alloc)
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804a..ac7ca1fb180c3 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -6,6 +6,7 @@
*/
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/slab.h>
#include <linux/sched/autogroup.h>
#include <linux/sched/mm.h>
@@ -1009,6 +1010,7 @@ void __noreturn do_exit(long code)
if (tsk->task_frag.page)
put_page(tsk->task_frag.page);
+ pisa_task_frag_release(tsk);
exit_task_stack_account(tsk);
diff --git a/kernel/fork.c b/kernel/fork.c
index 5efceb8d58a78..d235d7ab04687 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -44,6 +44,7 @@
#include <linux/mmu_notifier.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/mm_inline.h>
#include <linux/memblock.h>
#include <linux/nsproxy.h>
@@ -982,6 +983,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
#endif
tsk->splice_pipe = NULL;
tsk->task_frag.page = NULL;
+ pisa_task_frag_init(tsk);
tsk->wake_q.next = NULL;
tsk->worker_private = NULL;
diff --git a/net/core/sock.c b/net/core/sock.c
index e2401d735c5a7..695fefe316925 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -115,6 +115,7 @@
#include <linux/memcontrol.h>
#include <linux/prefetch.h>
#include <linux/compat.h>
+#include <linux/pisa.h>
#include <linux/mroute.h>
#include <linux/mroute6.h>
#include <linux/icmpv6.h>
@@ -3257,13 +3258,20 @@ DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key);
*/
bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
{
+ /* PISA pages only go into current->task_frag_pisa, see sk_page_frag_dst(). */
+ if (is_pisa_pfrag(pfrag))
+ gfp |= __GFP_PISA_RO;
+
if (pfrag->page) {
+ if ((gfp & __GFP_PISA_RO) && !pisa_page_can_recycle(pfrag->page))
+ goto drop_page;
if (page_ref_count(pfrag->page) == 1) {
pfrag->offset = 0;
return true;
}
if (pfrag->offset + sz <= pfrag->size)
return true;
+drop_page:
put_page(pfrag->page);
}
@@ -3289,6 +3297,39 @@ bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
}
EXPORT_SYMBOL(skb_page_frag_refill);
+/**
+ * skb_page_frag_refill_dev - GFP_ATOMIC skb_page_frag_refill() for @skb TX
+ * @sz: minimum size of the fragment we want to get
+ * @pfrag: pointer to page_frag, private to the caller
+ * @skb: packet being transmitted
+ *
+ * Pages come from the per-node DMA_TO_DEVICE PISA allocators when they are
+ * useful for @skb's egress device (__netdev_tx_pisa_ok()), otherwise from the
+ * buddy allocator. PISA pages are readable by every device that maps them
+ * while they stay active, so @pfrag must only ever hold data meant for devices,
+ * e.g. ciphertext; it may mix PISA and ordinary pages.
+ */
+bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
+ const struct sk_buff *skb)
+{
+ gfp_t gfp = GFP_ATOMIC;
+
+ /*
+ * Before ip_finish_output2() populates skb->dev (e.g. esp_output),
+ * the egress device is carried by skb_dst(skb).
+ */
+ if (net_pisa_tx_bufs_enabled()) {
+ const struct dst_entry *dst = skb_dst(skb);
+
+ rcu_read_lock();
+ if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL)))
+ gfp |= __GFP_PISA_RO;
+ rcu_read_unlock();
+ }
+ return skb_page_frag_refill(sz, pfrag, gfp);
+}
+EXPORT_SYMBOL(skb_page_frag_refill_dev);
+
bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
{
if (likely(skb_page_frag_refill(32U, pfrag, sk->sk_allocation)))
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index ae701561d9f12..feecf094f582c 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -451,7 +451,7 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto cow;
}
@@ -560,7 +560,7 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto error_free;
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 3396f177a18d0..a5ea530bc8b5b 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -986,7 +986,10 @@ static int __ip_append_data(struct sock *sk,
exthdrlen = !skb ? rt->dst.header_len : 0;
mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize;
- paged = !!cork->gso_size;
+ paged = !!cork->gso_size ||
+ (sk->sk_type != SOCK_RAW &&
+ (rt->dst.dev->features & NETIF_F_SG) &&
+ is_pisa_pfrag(pfrag));
hh_len = LL_RESERVED_SPACE(rt->dst.dev);
@@ -1383,8 +1386,8 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4,
}
return __ip_append_data(sk, fl4, &sk->sk_write_queue, &inet->cork.base,
- sk_page_frag(sk), getfrag,
- from, length, transhdrlen, flags);
+ sk_page_frag_dst(sk, inet->cork.base.dst),
+ getfrag, from, length, transhdrlen, flags);
}
static void ip_cork_release(struct inet_cork *cork)
@@ -1592,7 +1595,7 @@ struct sk_buff *ip_make_skb(struct sock *sk,
return ERR_PTR(err);
err = __ip_append_data(sk, fl4, &queue, cork,
- ¤t->task_frag, getfrag,
+ task_page_frag(cork->dst), getfrag,
from, length, transhdrlen, flags);
if (err) {
__ip_flush_pending_frames(sk, &queue, cork);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index b9de7c302ba83..b8d44a0638914 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1290,9 +1290,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
copy = msg_data_left(msg);
if (zc == 0) {
+ const struct dst_entry *dst = __sk_dst_get(sk);
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
- struct page_frag *pfrag = sk_page_frag(sk);
+ struct page_frag *pfrag = sk_page_frag_dst(sk, dst);
if (!sk_page_frag_refill(sk, pfrag))
goto wait_for_space;
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index d654f0f442388..cdedda548749a 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -481,7 +481,7 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto cow;
}
@@ -592,7 +592,7 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
spin_lock_bh(&x->lock);
- if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+ if (unlikely(!skb_page_frag_refill_dev(allocsize, pfrag, skb))) {
spin_unlock_bh(&x->lock);
goto error_free;
}
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 25fe0ba98b1a6..957c38a08680c 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1486,7 +1486,10 @@ static int __ip6_append_data(struct sock *sk,
dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len;
}
- paged = !!cork->gso_size;
+ paged = !!cork->gso_size ||
+ (sk->sk_type != SOCK_RAW &&
+ (rt->dst.dev->features & NETIF_F_SG) &&
+ is_pisa_pfrag(pfrag));
mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize;
orig_mtu = mtu;
@@ -1889,8 +1892,8 @@ int ip6_append_data(struct sock *sk,
}
return __ip6_append_data(sk, &sk->sk_write_queue, &inet->cork,
- sk_page_frag(sk), getfrag,
- from, length, transhdrlen, flags);
+ sk_page_frag_dst(sk, inet->cork.base.dst),
+ getfrag, from, length, transhdrlen, flags);
}
EXPORT_SYMBOL_GPL(ip6_append_data);
@@ -2092,7 +2095,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk,
}
err = __ip6_append_data(sk, &queue, cork,
- ¤t->task_frag, getfrag, from,
+ task_page_frag(cork->base.dst), getfrag, from,
length + exthdrlen, transhdrlen + exthdrlen,
flags);
if (err) {
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (12 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
` (11 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When a page_pool performs DMA mapping (PP_FLAG_DMA_MAP) on a device with
pisa_rx_bufs enabled, pass __GFP_PISA_RW to use PISA memory.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
net/core/page_pool.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index d7c88c0b67e53..91cae5ba78373 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
#include <net/page_pool/helpers.h>
#include <net/page_pool/memory_provider.h>
#include <net/xdp.h>
+#include <linux/pisa.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
@@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(struct page_pool *pool
*/
if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
gfp |= __GFP_NOWARN;
+ if (dma_map)
+ gfp |= pisa_rx_gfp(pool->p.dev);
/* Don't support bulk alloc for high-order pages */
if (unlikely(pp_order))
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (13 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
` (10 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
In Confidential Computing (CoCo) guests (AMD SEV-SNP, Intel TDX, ARM CCA),
guest memory is private/encrypted by default, and DMA buffers must be
converted to shared/decrypted in the host page tables before a device
can access them.
Doing that hypercall per-packet on 4KB pages is prohibitively expensive,
so guests normally bounce DMA through swiotlb. With PISA, 2MB pages are
decrypted once upfront when allocated and re-encrypted only when retired
back to the buddy allocator, avoiding both per-packet hypercalls and
bounce-buffer copies.
Add pisa.decrypt (auto-enabled when CC_ATTR_GUEST_MEM_ENCRYPT is active)
and the PISA_DECRYPTED flag to manage 2MB page decryption/re-encryption
and bypass swiotlb bouncing for decrypted PISA pages.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 117 ++++++++++++++++++++++++++++++++-----
drivers/iommu/pisa-kunit.c | 11 ++++
drivers/iommu/pisa-map.c | 2 +-
drivers/iommu/pisa-meta.c | 13 +++++
drivers/iommu/pisa-node.c | 86 ++++++++++++++++-----------
drivers/iommu/pisa-priv.h | 16 ++++-
include/linux/pisa.h | 16 +++++
kernel/dma/direct.c | 5 +-
8 files changed, 212 insertions(+), 54 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 7b9d0a7acc101..b4a25c65bf97e 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -9,6 +9,7 @@
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/cache.h>
+#include <linux/cc_platform.h>
#include <linux/cpuhotplug.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
@@ -29,6 +30,7 @@
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
#include <linux/sched/mm.h>
+#include <linux/set_memory.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
#include <linux/slab.h>
@@ -49,6 +51,10 @@ static LLIST_HEAD(pisa_free_list);
/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
static struct workqueue_struct *pisa_wq __ro_after_init;
+/* Mark active PISA pages decrypted for DMA (auto-enabled on CoCo guests). */
+static bool pisa_decrypt __read_mostly;
+module_param_named(decrypt, pisa_decrypt, bool, 0444);
+
/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */
atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp;
unsigned long pisa_max_pages __read_mostly;
@@ -80,6 +86,51 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
static void pisa_schedule_reclaim(void);
+/* Return true if a new PISA page should be decrypted. */
+bool pisa_want_decrypt(void)
+{
+ if (!pisa_decrypt)
+ return false;
+
+ /* Explicit page decryption is for CoCo guests, not bare-metal SME hosts. */
+ return !WARN_ON_ONCE(cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT));
+}
+
+/* Decrypt and zero a new PISA page. Must be called from sleepable context. */
+int pisa_page_prepare(struct page *page, struct pisa_meta *meta)
+{
+ void *vaddr = page_address(page);
+ int ret;
+
+ ret = set_memory_decrypted((unsigned long)vaddr, 1 << PISA_ORDER);
+ if (ret) {
+ /* Leak partially shared page rather than returning it to the buddy allocator. */
+ pr_warn_ratelimited("pisa: leaking PISA page that could not be decrypted\n");
+ return ret;
+ }
+ /* Initialize cleartext cachelines after flipping encryption state. */
+ memset(vaddr, 0, PISA_SIZE);
+ meta->flags |= PISA_DECRYPTED;
+ return 0;
+}
+
+/* Undo pisa_page_prepare(). */
+int pisa_page_unprepare(struct pisa_meta *meta)
+{
+ struct page *page;
+
+ if (!(meta->flags & PISA_DECRYPTED))
+ return 0;
+ page = pfn_to_page(pisa_meta_to_pfn(meta));
+ if (set_memory_encrypted((unsigned long)page_address(page), 1 << PISA_ORDER)) {
+ pr_warn_ratelimited("pisa: leaking PISA page that could not be re-encrypted\n");
+ return -EIO;
+ }
+ WRITE_ONCE(meta->flags, meta->flags & ~PISA_DECRYPTED);
+ return 0;
+}
+
+
/*
* Idle PISA page release proceeds in three stages:
* 1. Detach onto lockless pisa_free_list and schedule pisa_reclaim_work.
@@ -103,12 +154,13 @@ static void pisa_release_page_rcu(struct rcu_head *head)
spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
- /* Possibly still mapped by a device: leak it, still charged. */
- if (meta->flags & PISA_QUARANTINED)
- return;
-
/* The page has been unmapped from every domain by pisa_release_page() */
- pisa_slot_free(slot);
+ if (!(meta->flags & PISA_QUARANTINED))
+ pisa_slot_free(slot);
+
+ /* Still shared with the host, or possibly still mapped by a device. */
+ if (meta->flags & (PISA_DECRYPTED | PISA_QUARANTINED))
+ return;
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -120,7 +172,12 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
void pisa_release_page(struct pisa_meta *meta)
{
+ u8 flags;
+
if (!meta->pa) {
+ /* Unbound (reservoir) page: leaked if re-encryption fails. */
+ if (pisa_page_unprepare(meta))
+ return;
__free_pages(pfn_to_page(pisa_meta_to_pfn(meta)), PISA_ORDER);
atomic_long_dec(&pisa_nr_pages);
return;
@@ -128,9 +185,15 @@ void pisa_release_page(struct pisa_meta *meta)
/* Quarantined on unmap failure and leaked in pisa_release_page_rcu(). */
pisa_unmap_all(meta);
+ flags = meta->flags & ~PISA_ACTIVE;
/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
- WRITE_ONCE(meta->flags, meta->flags & ~PISA_ACTIVE);
+ WRITE_ONCE(meta->flags, flags);
+ if ((flags & (PISA_DECRYPTED | PISA_QUARANTINED)) == PISA_DECRYPTED) {
+ synchronize_rcu();
+ pisa_page_unprepare(meta);
+ }
+
call_rcu(&meta->rcu, pisa_release_page_rcu);
}
@@ -741,15 +804,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
unsigned int bpp = PISA_BLOCKS(order);
struct pisa_meta *meta;
struct page *page;
+ int ret;
/* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
- meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+ meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, pisa_want_decrypt(), &clean);
if (IS_ERR(meta))
return meta;
if (unlikely(pisa_slot_alloc(meta))) {
- pisa_release_page(meta);
- return ERR_PTR(-ENOSPC);
+ ret = -ENOSPC;
+ goto err_free;
}
page = pfn_to_page(pisa_meta_to_pfn(meta));
@@ -757,9 +821,9 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
if (split_page_compound(page, PISA_ORDER, order)) {
pisa_slot_free(meta->slot);
- pisa_release_page(meta);
pisa_alloc_backoff(nid, can_block);
- return ERR_PTR(-EBUSY);
+ ret = -EBUSY;
+ goto err_free;
}
INIT_LIST_HEAD(&meta->list);
@@ -773,6 +837,16 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
}
return meta;
+
+err_free:
+ if (!can_block && (meta->flags & PISA_DECRYPTED)) {
+ /* set_memory_encrypted() may sleep; defer to reclaim worker. */
+ llist_add(&meta->llnode, &pisa_free_list);
+ pisa_schedule_reclaim();
+ } else {
+ pisa_release_page(meta);
+ }
+ return ERR_PTR(ret);
}
/* Reset fields normally cleared by prep_new_page()/post_alloc_hook(). */
@@ -1135,7 +1209,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
if (!win || pisa_window_declined(win) || !(domain->pgsize_bitmap & PISA_SIZE))
return NULL;
} else if (is_swiotlb_for_alloc(dev) ||
- force_dma_unencrypted(dev) ||
+ (force_dma_unencrypted(dev) && !pisa_want_decrypt()) ||
!dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
return NULL;
}
@@ -1294,10 +1368,15 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
read_unlock(&pisa_allocs_lock);
}
- /* Retiring must not run under @pa->lock. */
+ /* Unmap outside locks; defer decrypted pages to workqueue outside fs_reclaim. */
list_for_each_entry_safe(meta, tmp, &release_list, list) {
list_del_init(&meta->list);
- pisa_release_page(meta);
+ if (meta->flags & PISA_DECRYPTED) {
+ llist_add(&meta->llnode, &pisa_free_list);
+ pisa_schedule_reclaim();
+ } else {
+ pisa_release_page(meta);
+ }
}
/* Charge do_shrink_slab() for the full PISA pages reclaimed. */
@@ -1342,6 +1421,16 @@ static int __init pisa_init(void)
pisa_slots_init();
+ /*
+ * CoCo guests require decrypted pages and cannot decrypt in atomic
+ * context, so ensure the per-node reservoir is enabled to serve them.
+ */
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
+ pisa_decrypt = true;
+ if (!pisa_reservoir_max)
+ pisa_reservoir_max = 16;
+ }
+
/* Preallocate the supported per-node allocators. */
for_each_node_state(nid, N_MEMORY) {
for (i = 0; i < ARRAY_SIZE(cfg); i++) {
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 5f6b2ebcab1d9..b63cd6585b673 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -63,6 +63,7 @@ static void test_meta_active_toggle(struct kunit *test)
unsigned long pfn, base_pfn;
struct pisa_meta *meta;
struct page *page;
+ phys_addr_t pa;
KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -80,16 +81,26 @@ static void test_meta_active_toggle(struct kunit *test)
}
KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
meta = pisa_meta_of_pfn(pfn);
+ pa = page_to_phys(page);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
pisa_meta_set_active(meta, true);
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
+ KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), pa);
+
+ WRITE_ONCE(meta->flags, PISA_ACTIVE | PISA_DECRYPTED);
+ KUNIT_EXPECT_TRUE(test, is_pisa_direct(pa, PAGE_SIZE));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PISA_SIZE + PAGE_SIZE));
+ KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), dma_addr_unencrypted(pa));
pisa_meta_set_active(meta, false);
KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+ KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
__free_pages(page, PISA_ORDER);
}
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 2c481e514575c..fc58c639f3288 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -470,7 +470,7 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
phys_addr_t pisa_phys = phys & PISA_MASK;
ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
- pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+ pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
if (likely(!ret)) {
/* Ensure leaf PTE is visible before setting the domain bit. */
smp_mb__before_atomic();
diff --git a/drivers/iommu/pisa-meta.c b/drivers/iommu/pisa-meta.c
index 00939c366ba6d..3acdec815935f 100644
--- a/drivers/iommu/pisa-meta.c
+++ b/drivers/iommu/pisa-meta.c
@@ -78,6 +78,19 @@ struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
return &pisa_meta_array[frame];
}
+bool __is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ u8 flags = READ_ONCE(pisa_meta_of_pfn(PHYS_PFN(phys))->flags);
+
+ /*
+ * Only decrypted active pages can bypass swiotlb bouncing, and the
+ * transfer must not straddle the end of the PISA frame.
+ */
+ return (flags & (PISA_ACTIVE | PISA_DECRYPTED)) ==
+ (PISA_ACTIVE | PISA_DECRYPTED) &&
+ (phys & (PISA_SIZE - 1)) + size <= PISA_SIZE;
+}
+
/* Base PFN of @meta's PISA frame, or ULONG_MAX if out-of-range/sink. */
unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
{
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 0f23b94a7a6f8..dfde9c021fe05 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -62,7 +62,7 @@ static unsigned int pisa_memalloc_save(gfp_t gfp)
/* Allocate a PISA page from the buddy allocator and initialize its metadata. */
static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
- bool *clean)
+ bool decrypt, bool *clean)
{
struct pisa_meta *meta;
unsigned int memflags;
@@ -83,6 +83,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
* must not dip into emergency reserves or trigger OOM/warnings.
*/
gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+ if (decrypt)
+ gfp &= ~__GFP_ZERO;
gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
if (can_block)
@@ -101,14 +103,21 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
__free_pages(page, PISA_ORDER);
return ERR_PTR(-ERANGE);
}
- if (can_block)
- memalloc_flags_restore(memflags);
meta = pisa_meta_of_pfn(page_to_pfn(page));
memset(meta, 0, sizeof(*meta));
atomic_long_inc(&pisa_nr_pages);
- *clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+ /* On decrypt failure the page is leaked and stays charged to budget. */
+ if (decrypt && pisa_page_prepare(page, meta)) {
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ return ERR_PTR(-EIO);
+ }
+ if (can_block)
+ memalloc_flags_restore(memflags);
+ *clean = (gfp & __GFP_ZERO) || (meta->flags & PISA_DECRYPTED) ||
+ want_init_on_alloc(0) || want_init_on_free();
return meta;
}
@@ -124,11 +133,12 @@ static void pisa_reservoir_refill_node(int nid)
unsigned long flags;
bool clean;
- meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+ meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
+ pisa_want_decrypt(), &clean);
if (IS_ERR(meta))
break;
- /* init_on_alloc/init_on_free pages are already zeroed. */
+ /* Decrypted and init_on_alloc/init_on_free pages are already zeroed. */
if (!clean)
memset(page_address(pfn_to_page(pisa_meta_to_pfn(meta))), 0, PISA_SIZE);
@@ -145,47 +155,48 @@ static void pisa_reservoir_refill_node(int nid)
static void pisa_reservoir_trim_node(int nid, unsigned int max)
{
struct pisa_node *res = &pisa_nodes[nid];
- struct pisa_meta *meta, *tmp;
- LIST_HEAD(free_list);
- unsigned long flags;
- spin_lock_irqsave(&res->lock, flags);
- while (res->nr_pages > max && !list_empty(&res->pages)) {
- list_move(res->pages.prev, &free_list);
- res->nr_pages--;
- }
- spin_unlock_irqrestore(&res->lock, flags);
+ while (res->nr_pages > max && res->nr_pages > READ_ONCE(pisa_reservoir_max)) {
+ struct pisa_meta *meta;
+ unsigned long flags;
+
+ spin_lock_irqsave(&res->lock, flags);
+ meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
+ if (meta) {
+ list_del_init(&meta->list);
+ res->nr_pages--;
+ }
+ spin_unlock_irqrestore(&res->lock, flags);
+ if (!meta)
+ break;
- list_for_each_entry_safe(meta, tmp, &free_list, list) {
- list_del_init(&meta->list);
pisa_release_page(meta);
cond_resched();
}
}
+/* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
static void pisa_node_work_fn(struct work_struct *work)
{
- struct pisa_node *res = container_of(work, struct pisa_node, work);
- int nid = res - pisa_nodes;
- unsigned int max = READ_ONCE(pisa_reservoir_max);
-
- if (unlikely(pisa_meta_init()))
- return;
+ struct pisa_node *node = container_of(work, struct pisa_node, work);
+ int nid = node - pisa_nodes;
- if (READ_ONCE(res->nr_pages) < max)
+ pisa_reservoir_trim_node(nid, READ_ONCE(pisa_reservoir_max));
+ if (READ_ONCE(pisa_reservoir_max) && static_branch_unlikely(&pisa_key))
pisa_reservoir_refill_node(nid);
- else if (READ_ONCE(res->nr_pages) > max)
- pisa_reservoir_trim_node(nid, max);
}
-/* Queue @nid's reservoir worker on a CPU belonging to @nid. */
+/* Queue node @nid's work if its reservoir is below or above the target. */
static void pisa_schedule_reservoir_node(int nid)
{
- if (nid >= 0 && nid < MAX_NUMNODES && node_state(nid, N_MEMORY))
- queue_work_node(nid, system_dfl_wq, &pisa_nodes[nid].work);
+ struct pisa_node *res = &pisa_nodes[nid];
+
+ if (node_state(nid, N_MEMORY) &&
+ READ_ONCE(res->nr_pages) != READ_ONCE(pisa_reservoir_max) &&
+ !work_pending(&res->work))
+ queue_work_node(nid, system_dfl_wq, &res->work);
}
-/* Queue reservoir refill/trim workers on all memory nodes. */
void pisa_schedule_reservoir(void)
{
int nid;
@@ -195,7 +206,7 @@ void pisa_schedule_reservoir(void)
}
/* Pop one pre-prepared PISA page from node @nid's reservoir under @res->lock. */
-static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool decrypt)
{
struct pisa_node *res = &pisa_nodes[nid];
struct pisa_meta *meta;
@@ -207,10 +218,12 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
spin_lock_irqsave(&res->lock, flags);
meta = list_first_entry_or_null(&res->pages, struct pisa_meta, list);
- if (meta) {
+ if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
list_del_init(&meta->list);
res->nr_pages--;
refill = res->nr_pages <= max / 2;
+ } else {
+ meta = NULL;
}
spin_unlock_irqrestore(&res->lock, flags);
@@ -224,7 +237,7 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
* Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
* refill when <= half full or empty), or allocate one inline on miss.
*/
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean)
{
bool can_block = gfpflags_allow_blocking(gfp);
unsigned int max = READ_ONCE(pisa_reservoir_max);
@@ -232,7 +245,7 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
nid = pisa_sanitize_nid(nid);
if (max) {
- meta = pisa_reservoir_pop_node(nid, max);
+ meta = pisa_reservoir_pop_node(nid, max, decrypt);
if (meta) {
*clean = true;
return meta;
@@ -244,8 +257,11 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
/* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
if (!can_block && time_before(jiffies, READ_ONCE(pisa_nodes[nid].next_alloc)))
return ERR_PTR(-EAGAIN);
+ /* Page decryption may sleep; atomic callers must rely on the reservoir. */
+ if (decrypt && !can_block)
+ return ERR_PTR(-EAGAIN);
- meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+ meta = __pisa_page_alloc(gfp, nid, can_block, decrypt, clean);
if (IS_ERR(meta))
pisa_alloc_backoff(nid, can_block);
return meta;
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 8f87bdf7d4ee2..f08388f003fad 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -7,6 +7,7 @@
#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/llist.h>
+#include <linux/mem_encrypt.h>
#include <linux/nodemask.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
@@ -62,7 +63,7 @@ enum {
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
- * @flags: PISA_ACTIVE / PISA_QUARANTINED
+ * @flags: PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED
* @prot: Owning allocator's IOMMU_READ/WRITE protection bits
* @nr_clean: Zeroed available blocks in @clean_bitmap
* @nr_dirty: Dirty available blocks in @dirty_bitmap
@@ -101,6 +102,14 @@ struct pisa_meta {
static_assert(offsetof(struct pisa_meta, flags) == 0);
static_assert(sizeof(struct pisa_meta) == PISA_META_SIZE);
+/* Physical address for IOMMU PTE of @meta at @phys (shared alias if decrypted). */
+static inline phys_addr_t pisa_pte_phys(const struct pisa_meta *meta, phys_addr_t phys)
+{
+ if (meta->flags & PISA_DECRYPTED)
+ return dma_addr_unencrypted(phys);
+ return phys;
+}
+
static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
{
return meta->nr_clean + meta->nr_dirty;
@@ -227,6 +236,9 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
if (!can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
+bool pisa_want_decrypt(void);
+int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
+int pisa_page_unprepare(struct pisa_meta *meta);
/* pisa-alloc.c, also used by the per-node code in pisa-node.c */
struct pisa_alloc *pisa_create(int nid, unsigned int order, enum dma_data_direction dir);
@@ -256,7 +268,7 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 822e1556d78ef..89eb0722d90bc 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -33,6 +33,7 @@ struct iommu_domain;
enum {
PISA_ACTIVE = BIT(0), /* page is owned by a pisa_alloc */
PISA_QUARANTINED = BIT(1), /* IOMMU unmap failed: never reuse */
+ PISA_DECRYPTED = BIT(2), /* page was set_memory_decrypted() */
};
/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
@@ -45,6 +46,8 @@ extern unsigned long *pisa_chunk_bitmap;
int pisa_meta_init(void);
+bool __is_pisa_direct(phys_addr_t phys, size_t size);
+
/* fast membership check for pisa pfn/pages */
static inline bool is_pisa_pfn(unsigned long pfn)
{
@@ -61,6 +64,14 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return READ_ONCE(*flags) & PISA_ACTIVE;
}
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ if (!static_branch_unlikely(&pisa_key))
+ return false;
+
+ return __is_pisa_direct(phys, size);
+}
+
static inline bool is_pisa_page(const struct page *page);
bool __pisa_free_page(struct page *page);
@@ -133,6 +144,11 @@ static inline bool is_pisa_pfn(unsigned long pfn)
return false;
}
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+ return false;
+}
+
static inline bool pisa_free_page(struct page *page)
{
return false;
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 11052b65e573b..a7fa1efc0ba09 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -707,8 +707,9 @@ dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
if (is_swiotlb_force_bounce(dev)) {
if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT))
return DMA_MAPPING_ERROR;
-
- return swiotlb_map(dev, phys, size, dir, attrs);
+ if (!is_pisa_direct(phys, size))
+ return swiotlb_map(dev, phys, size, dir, attrs);
+ attrs |= DMA_ATTR_CC_SHARED;
}
if (attrs & DMA_ATTR_CC_SHARED)
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (14 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
` (9 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Blocks returned to a pisa_alloc are marked dirty and need to be zeroed
before they can serve __GFP_ZERO requests. Zeroing blocks inline at
allocation time adds CPU cost and latency.
Add an optional background scrubber (enabled via pisa.scrub_max) that
runs on the allocator's home NUMA node work item, zeroes dirty blocks
outside pa->lock in batches, and moves them to meta->clean_bitmap[].
Scrubbing is demand-driven by the number of dirty blocks that __GFP_ZERO
allocations had to zero inline.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 188 ++++++++++++++++++++++++++++++++++++-
drivers/iommu/pisa-kunit.c | 49 +++++++++-
drivers/iommu/pisa-node.c | 31 ++++++
drivers/iommu/pisa-priv.h | 11 ++-
4 files changed, 272 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index b4a25c65bf97e..4b61ec84ed6d1 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -29,6 +29,7 @@
#include <linux/percpu.h>
#include <linux/pgalloc_tag.h>
#include <linux/rcupdate.h>
+#include <linux/sched/clock.h>
#include <linux/sched/mm.h>
#include <linux/set_memory.h>
#include <linux/shrinker.h>
@@ -72,6 +73,10 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
+/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
+unsigned int pisa_scrub_max __read_mostly;
+#define PISA_SCRUB_MAX_LIMIT 65536U
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -86,6 +91,24 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
static void pisa_schedule_reclaim(void);
+static int pisa_scrub_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (ret)
+ return ret;
+ WRITE_ONCE(pisa_scrub_max, min(pisa_scrub_max, PISA_SCRUB_MAX_LIMIT));
+ /* Serve demand accumulated while the scrubber was off (once initialized). */
+ if (pisa_scrub_max && static_branch_unlikely(&pisa_key))
+ pisa_schedule_scrub_all();
+ return 0;
+}
+
+static const struct kernel_param_ops pisa_scrub_ops = {
+ .set = pisa_scrub_set,
+ .get = param_get_uint,
+};
+module_param_cb(scrub_max, &pisa_scrub_ops, &pisa_scrub_max, 0644);
/* Return true if a new PISA page should be decrypted. */
bool pisa_want_decrypt(void)
{
@@ -263,6 +286,7 @@ static void pisa_schedule_reclaim(void)
else
queue_delayed_work(wq, &pisa_reclaim_work, msecs_to_jiffies(idle_ms));
}
+
/*
* Update @meta's position on @pa's lists under @pa->lock. When all blocks
* become available (avail == bpp), either queues @meta to @pisa_free_list (and
@@ -353,6 +377,148 @@ static bool pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
return false;
}
+/*
+ * Background page scrubber: pre-zeroes dirty blocks in @pa->nid's work item
+ * to match inline-zeroed demand (@pa->scrub_demand) when scrub_max > 0.
+ */
+static struct pisa_meta *pisa_pick_dirty_meta(struct pisa_alloc *pa)
+{
+ struct pisa_meta *meta;
+
+ meta = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+ if (meta)
+ return meta;
+
+ list_for_each_entry(meta, &pa->partial, list) {
+ if (meta->nr_dirty)
+ return meta;
+ }
+
+ if (!list_empty(&pa->idle)) {
+ meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+ if (meta->nr_dirty) {
+ pa->num_idle_pages--;
+ return meta;
+ }
+ }
+
+ return NULL;
+}
+
+/* Dirty blocks per allocator, inline-zeroed since the last run, that queue a run. */
+#define PISA_SCRUB_TRIGGER 32U
+/* Cap on @pa->scrub_demand, so a long-off scrubber does not owe too much. */
+#define PISA_SCRUB_DEMAND_MAX 4096U
+/* Bytes zeroed per batch: a few microseconds of memset(). */
+/* XXX assumes PAGE_SIZE <= SZ_128K (32 * 4KB pages). */
+#define PISA_SCRUB_BYTES SZ_128K
+#define PISA_SCRUB_BATCH (PISA_SCRUB_BYTES >> PAGE_SHIFT)
+/* Time after which a run stops and requeues. */
+#define PISA_SCRUB_RUN_NS (50 * NSEC_PER_USEC)
+
+/*
+ * Scrub one batch of dirty blocks in @pa outside @pa->lock.
+ * Returns the number of PAGE_SIZE pages taken off the dirty bitmap.
+ */
+static unsigned long pisa_scrub_alloc(struct pisa_alloc *pa)
+{
+ unsigned int bpp, order, idx, count = 0, used = 0, want;
+ u16 idxs[PISA_SCRUB_BATCH];
+ unsigned long base_pfn, flags;
+ bool should_release;
+ struct pisa_meta *meta;
+
+ spin_lock_irqsave(&pa->lock, flags);
+ if (!pa->scrub_demand) {
+ spin_unlock_irqrestore(&pa->lock, flags);
+ return 0;
+ }
+
+ meta = pisa_pick_dirty_meta(pa);
+ if (!meta) {
+ pa->scrub_demand = 0;
+ spin_unlock_irqrestore(&pa->lock, flags);
+ return 0;
+ }
+
+ order = pa->order;
+ bpp = PISA_BLOCKS(order);
+ base_pfn = pisa_meta_to_pfn(meta);
+ want = min(max(PISA_SCRUB_BATCH >> order, 1U), pa->scrub_demand);
+
+ for_each_set_bit(idx, meta->dirty_bitmap, bpp) {
+ struct page *block = pfn_to_page(base_pfn + (idx << order));
+
+ __clear_bit(idx, meta->dirty_bitmap);
+ used++;
+ if (!pisa_block_poisoned(pa, block))
+ idxs[count++] = idx;
+ if (used == meta->nr_dirty || count == want)
+ break;
+ }
+ meta->nr_dirty -= used;
+ pa->blocks_in_use += used - count;
+ pisa_place_meta(meta);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (unlikely(!count))
+ return (unsigned long)used << order;
+
+ for (idx = 0; idx < count; idx++) {
+ struct page *block = pfn_to_page(base_pfn + (idxs[idx] << order));
+
+ memset(page_address(block), 0, PAGE_SIZE << order);
+ }
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (idx = 0; idx < count; idx++)
+ __set_bit(idxs[idx], meta->clean_bitmap);
+ meta->nr_clean += count;
+ pa->scrub_demand -= min(count, pa->scrub_demand);
+ should_release = pisa_place_meta(meta);
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return (unsigned long)used << order;
+}
+
+/*
+ * Scrub dirty blocks for allocators homed on @nid up to @pisa_scrub_max pages or
+ * PISA_SCRUB_RUN_NS. Returns true if stopped early with demand remaining.
+ */
+bool pisa_scrub_node(int nid)
+{
+ unsigned long budget = READ_ONCE(pisa_scrub_max), done = 0, n;
+ u64 deadline = local_clock() + PISA_SCRUB_RUN_NS;
+ struct pisa_alloc *pa;
+ bool more = false;
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ if (pa->nid != nid || !READ_ONCE(pa->scrub_demand))
+ continue;
+ read_unlock(&pisa_allocs_lock);
+
+ while ((n = pisa_scrub_alloc(pa))) {
+ done += n;
+ if (done >= budget || local_clock() >= deadline) {
+ more = true;
+ break;
+ }
+ cond_resched();
+ }
+
+ read_lock(&pisa_allocs_lock);
+ if (more)
+ break;
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ return more;
+}
+
/**
* pisa_create - Create a PISA page allocator
* @nid: Home NUMA node for this allocator
@@ -952,6 +1118,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
for (i = got; i < got + dirty; i++)
set_page_private(out[i], 1);
got += dirty;
+ pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
} else {
/* First choice are dirty blocks, followed by clean */
got += pisa_scan_bitmap(meta, true, want, out, !frozen);
@@ -973,6 +1140,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
struct page **pages, bool zero, bool frozen)
{
+ bool should_scrub = false;
unsigned long n, flags, i;
struct pisa_meta *meta;
unsigned int want;
@@ -986,6 +1154,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
continue;
/* Fallback to a new allocation */
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
/* With @zero, fresh dirty blocks are zeroed below. */
meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
@@ -998,6 +1167,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
}
+ /* Queue the scrubber when the inline zeroing crosses the trigger. */
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
out:
@@ -1012,6 +1183,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
else
page_ref_unfreeze(pages[i], 1);
}
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
return n;
}
@@ -1112,7 +1285,7 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
{
DECLARE_BITMAP(busy, PISA_MAX_BLOCKS);
struct pisa_alloc *pa = meta->pa;
- unsigned int bpp = PISA_BLOCKS(0), i;
+ unsigned int bpp = PISA_BLOCKS(0), i, dirty = 0;
bool poisoned = false;
unsigned long off;
struct page *page;
@@ -1166,11 +1339,13 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
} else {
__clear_bit(off + i, meta->dirty_bitmap);
meta->nr_dirty--;
+ dirty++;
}
pisa_block_prep(page + i);
page_ref_unfreeze(page + i, 1);
}
pa->blocks_in_use += nblocks;
+ pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
pisa_place_meta(meta);
return page;
}
@@ -1185,7 +1360,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
struct iommu_domain *domain, size_t size,
gfp_t gfp, unsigned long attrs)
{
- bool should_release = false;
+ bool should_scrub = false, should_release = false;
struct list_head *lists[3];
unsigned long align_mask, flags;
unsigned int nblocks, i;
@@ -1234,10 +1409,14 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
}
}
if (!page) {
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
meta = pisa_alloc_split_page(0, gfp, pa->nid);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
return NULL;
+ }
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
@@ -1247,10 +1426,13 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
if (unlikely(!page))
should_release = pisa_place_meta(meta);
}
+ should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
spin_unlock_irqrestore(&pa->lock, flags);
if (should_release)
pisa_schedule_reclaim();
+ if (should_scrub)
+ pisa_schedule_scrub(pa->nid);
if (!page)
return NULL;
*pagep = page;
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index b63cd6585b673..40bf7801183da 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -572,8 +572,53 @@ static void test_gfp_pisa(struct kunit *test)
static void test_reservoir_and_scrubber(struct kunit *test)
{
- KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
- KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+ unsigned int old_res = READ_ONCE(pisa_reservoir_max);
+ unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+ struct pisa_meta *meta;
+ struct pisa_alloc *pa;
+ struct page *p;
+ int nid;
+
+ pisa_shrink_scan(NULL, &sc);
+
+ /* Refill reservoir to 1 PISA page per memory node, then consume one. */
+ WRITE_ONCE(pisa_reservoir_max, 1);
+ pisa_schedule_reservoir();
+ for_each_node_state(nid, N_MEMORY)
+ flush_work(&pisa_nodes[nid].work);
+ KUNIT_EXPECT_GE(test, pisa_reservoir_count(), 1UL);
+
+ pa = pisa_node_pa(first_memory_node, 2, true);
+ KUNIT_ASSERT_NOT_NULL(test, pa);
+ p = pisa_alloc(pa, GFP_KERNEL);
+ if (p) {
+ meta = pisa_meta_of_pfn(page_to_pfn(p));
+ /* Reservoir pages come pre-zeroed (all remaining blocks clean). */
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ __free_pages(p, 2);
+ if (!want_init_on_free()) {
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 1U);
+
+ /* Scrub dirty block back into clean_bitmap. */
+ pa->scrub_demand = 1;
+ WRITE_ONCE(pisa_scrub_max, 512);
+ pisa_scrub_node(pa->nid);
+ KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ }
+ }
+ pisa_shrink_scan(NULL, &sc);
+
+ /* Trim reservoirs back to 0. */
+ WRITE_ONCE(pisa_reservoir_max, 0);
+ pisa_schedule_reservoir();
+ for_each_node_state(nid, N_MEMORY)
+ flush_work(&pisa_nodes[nid].work);
+ KUNIT_EXPECT_EQ(test, pisa_reservoir_count(), 0UL);
+
+ WRITE_ONCE(pisa_reservoir_max, old_res);
+ pisa_schedule_reservoir();
+ WRITE_ONCE(pisa_scrub_max, old_scrub);
}
static struct kunit_case pisa_meta_test_cases[] = {
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index dfde9c021fe05..cd29c146c281e 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -5,6 +5,7 @@
*/
#include <linux/atomic.h>
+#include <linux/bitops.h>
#include <linux/pisa.h>
#include <linux/err.h>
#include <linux/gfp.h>
@@ -32,6 +33,9 @@
*/
unsigned int pisa_reservoir_max __read_mostly = 16;
+/* Scrub the allocators whose home node (@pa->nid) is this one. */
+#define PISA_NODE_SCRUB 0
+
static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
{
int ret = param_set_uint(val, kp);
@@ -175,6 +179,30 @@ static void pisa_reservoir_trim_node(int nid, unsigned int max)
}
}
+/* Schedule node @nid's work item to scrub allocators homed on @nid. */
+void pisa_schedule_scrub(int nid)
+{
+ struct pisa_node *node;
+
+ if (!READ_ONCE(pisa_scrub_max) || nid == NUMA_NO_NODE)
+ return;
+
+ node = &pisa_nodes[nid];
+ set_bit(PISA_NODE_SCRUB, &node->flags);
+ /* Pairs with the clearing of PENDING before the work runs. */
+ smp_mb__after_atomic();
+ if (!work_pending(&node->work))
+ queue_work_node(nid, system_dfl_wq, &node->work);
+}
+
+void pisa_schedule_scrub_all(void)
+{
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY)
+ pisa_schedule_scrub(nid);
+}
+
/* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
static void pisa_node_work_fn(struct work_struct *work)
{
@@ -184,6 +212,9 @@ static void pisa_node_work_fn(struct work_struct *work)
pisa_reservoir_trim_node(nid, READ_ONCE(pisa_reservoir_max));
if (READ_ONCE(pisa_reservoir_max) && static_branch_unlikely(&pisa_key))
pisa_reservoir_refill_node(nid);
+
+ if (test_and_clear_bit(PISA_NODE_SCRUB, &node->flags) && pisa_scrub_node(nid))
+ pisa_schedule_scrub(nid);
}
/* Queue node @nid's work if its reservoir is below or above the target. */
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index f08388f003fad..ef3abb1db9049 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -144,8 +144,9 @@ struct pisa_alloc_stats {
* @num_idle_pages: Length of @idle
* @max_idle_pages: High watermark of @idle before async release to the buddy allocator
* @idle_jiffies: Timestamp (jiffies) when @idle last exceeded @max_idle_pages
+ * @scrub_demand: Dirty blocks zeroed inline since the scrubber last ran
* @blocks_in_use: Blocks handed out and not yet freed
- * @nid: NUMA node id
+ * @nid: NUMA node whose worker scrubs this allocator
* @node: Node in the global pisa_allocs list
* @name: Debug name (defaults to caller of pisa_create())
* @stats: Per-allocator statistics
@@ -162,6 +163,7 @@ struct pisa_alloc {
struct list_head full;
unsigned int num_idle_pages;
unsigned int max_idle_pages;
+ unsigned int scrub_demand;
unsigned long idle_jiffies;
unsigned long blocks_in_use;
/* Cold. */
@@ -187,7 +189,8 @@ struct pisa_node {
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
unsigned int nr_pages;
- struct work_struct work; /* refill/trim, on this node */
+ unsigned long flags; /* PISA_NODE_* requests */
+ struct work_struct work; /* refill/trim/scrub, on this node */
} ____cacheline_aligned_in_smp;
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
@@ -263,11 +266,15 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
unsigned int *nr, struct list_head *release_list,
unsigned long nr_to_scan);
+extern unsigned int pisa_scrub_max;
+bool pisa_scrub_node(int nid);
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
void pisa_node_init(void);
void pisa_schedule_reservoir(void);
+void pisa_schedule_scrub(int nid);
+void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (15 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
` (8 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
skb heads are provided by a slab allocator that is fed by alloc_pages().
Add SLAB_CACHE_PISA_RO so that a slab cache can back its slabs with
zeroed, read-only PISA folios (__GFP_PISA_RO | __GFP_ZERO) via
pisa_alloc_folio(), clamping the slab order to
[2, PAGE_ALLOC_COSTLY_ORDER].
Slabs are still released with free_frozen_pages(), so pages from PISA
return to their allocator via __free_pages_prepare(). Caches with
SLAB_CACHE_PISA_RO skip KFENCE and are never merged.
Existing caches are unaffected.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 27 +++++++++++++++++++++++++++
drivers/iommu/pisa-kunit.c | 19 +++++++++++++++++++
include/linux/pisa.h | 7 +++++++
include/linux/slab.h | 10 ++++++++++
mm/slab.h | 3 ++-
mm/slab_common.c | 5 ++++-
mm/slub.c | 14 +++++++++++++-
7 files changed, 82 insertions(+), 3 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 4b61ec84ed6d1..3b5c49d143a7e 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -848,6 +848,33 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred
return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
}
+/**
+ * pisa_alloc_folio - Allocate a frozen folio for SLUB (SLAB_CACHE_PISA_RO)
+ * @gfp: GFP flags (must have __GFP_PISA_RW or __GFP_PISA_RO set)
+ * @order: Folio order
+ * @nid: Target NUMA node (or NUMA_NO_NODE)
+ *
+ * Drops __GFP_THISNODE before calling the per-node allocator and verifies
+ * node placement afterwards.
+ *
+ * Return: Frozen folio of @order, or NULL.
+ */
+struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+ struct page *page;
+
+ page = pisa_alloc_pages(gfp & ~__GFP_THISNODE, order, nid, NULL);
+ if (!page)
+ return NULL;
+ if (unlikely((gfp & __GFP_THISNODE) && nid != NUMA_NO_NODE &&
+ page_to_nid(page) != nid)) {
+ page_ref_unfreeze(page, 1);
+ __free_pages(page, order);
+ return NULL;
+ }
+ return page_folio(page);
+}
+
/**
* pisa_has_free_blocks_rx - Check if @dev has free PISA RX blocks of @order on @nid
* @dev: Device to check
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 40bf7801183da..9cc1328893ace 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
+#include <linux/slab.h>
#include <net/sock.h>
#include "dma-iommu.h"
@@ -621,6 +622,23 @@ static void test_reservoir_and_scrubber(struct kunit *test)
WRITE_ONCE(pisa_scrub_max, old_scrub);
}
+static void test_slub_page_alloc(struct kunit *test)
+{
+ struct kmem_cache *s;
+ void *obj;
+
+ s = kmem_cache_create("pisa_kunit_slab", 1024, NULL,
+ SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+ KUNIT_ASSERT_NOT_NULL(test, s);
+
+ obj = kmem_cache_alloc(s, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, obj);
+ KUNIT_EXPECT_TRUE(test, is_pisa_page(virt_to_page(obj)));
+ kmem_cache_free(s, obj);
+
+ kmem_cache_destroy(s);
+}
+
static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_meta_init_and_roundtrip),
KUNIT_CASE(test_meta_invalid_phys),
@@ -635,6 +653,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
KUNIT_CASE(test_pcp_cache),
KUNIT_CASE(test_gfp_pisa),
KUNIT_CASE(test_reservoir_and_scrubber),
+ KUNIT_CASE(test_slub_page_alloc),
{}
};
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 89eb0722d90bc..b9c5197f2d88d 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -110,6 +110,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
int nr_pages, struct page **page_array,
bool frozen);
+struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
bool pisa_page_can_recycle(const struct page *page);
@@ -181,6 +182,12 @@ pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order, int preferred_nid,
return 0;
}
+static inline struct folio *
+pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+ return NULL;
+}
+
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
return 0;
diff --git a/include/linux/slab.h b/include/linux/slab.h
index cda126def67af..df19c5898e9fe 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -66,6 +66,9 @@ enum _slab_flag_bits {
#endif
#endif
_SLAB_NO_SHEAVES,
+#ifdef CONFIG_PISA
+ _SLAB_CACHE_PISA_RO,
+#endif
_SLAB_FLAGS_LAST_BIT
};
@@ -260,6 +263,13 @@ enum _slab_flag_bits {
#define SLAB_OBJ_EXT_IN_OBJ __SLAB_FLAG_UNUSED
#endif
+/* Back slabs with zeroed read-only PISA folios (__GFP_PISA_RO | __GFP_ZERO) */
+#ifdef CONFIG_PISA
+#define SLAB_CACHE_PISA_RO __SLAB_FLAG_BIT(_SLAB_CACHE_PISA_RO)
+#else
+#define SLAB_CACHE_PISA_RO __SLAB_FLAG_UNUSED
+#endif
+
/*
* ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
*
diff --git a/mm/slab.h b/mm/slab.h
index e94ac63d10dc0..53620e40b216b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -473,7 +473,8 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
SLAB_NOLEAKTRACE | SLAB_RECLAIM_ACCOUNT | \
SLAB_TEMPORARY | SLAB_ACCOUNT | \
- SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE)
+ SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE | \
+ SLAB_CACHE_PISA_RO)
#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
SLAB_TRACE | SLAB_CONSISTENCY_CHECKS)
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 01c64ef661e7f..d6c00047b1623 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -49,7 +49,7 @@ struct kmem_cache *kmem_cache;
*/
#define SLAB_NEVER_MERGE (SLAB_DEBUG_FLAGS | SLAB_TYPESAFE_BY_RCU | \
SLAB_NOLEAKTRACE | SLAB_FAILSLAB | SLAB_NO_MERGE | \
- SLAB_OBJ_EXT_IN_OBJ)
+ SLAB_OBJ_EXT_IN_OBJ | SLAB_CACHE_PISA_RO)
#define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
SLAB_CACHE_DMA32 | SLAB_ACCOUNT | SLAB_MAY_ACCOUNT)
@@ -377,6 +377,9 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
if (s)
goto out_unlock;
+ if (flags & SLAB_CACHE_PISA_RO)
+ flags |= SLAB_SKIP_KFENCE;
+
cache_name = kstrdup_const(name, GFP_KERNEL);
if (!cache_name) {
err = -ENOMEM;
diff --git a/mm/slub.c b/mm/slub.c
index 45e077c7526c2..63609768f4e3f 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -46,6 +46,7 @@
#include <linux/prandom.h>
#include <kunit/test.h>
#include <kunit/test-bug.h>
+#include <linux/pisa.h>
#include <linux/sort.h>
#include <linux/irq_work.h>
#include <linux/kprobes.h>
@@ -3378,6 +3379,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,
if (unlikely(!allow_spin))
page = alloc_frozen_pages_nolock(0/* __GFP_COMP is implied */,
node, order);
+ else if (unlikely(flags & __GFP_PISA_RO))
+ page = (struct page *)pisa_alloc_folio(flags, order, node);
else if (node == NUMA_NO_NODE)
page = alloc_frozen_pages(flags, order);
else
@@ -3484,7 +3487,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
gfp_t alloc_gfp;
void *start;
- flags &= gfp_allowed_mask;
+ flags &= gfp_allowed_mask & ~__GFP_PISA_RW;
flags |= s->allocflags;
@@ -3495,9 +3498,13 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
if ((alloc_gfp & __GFP_DIRECT_RECLAIM) && oo_order(oo) > oo_order(s->min))
alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;
+ if (unlikely(s->flags & SLAB_CACHE_PISA_RO))
+ alloc_gfp = (flags | __GFP_NOWARN) & ~__GFP_NOFAIL;
slab = alloc_slab_page(alloc_gfp, node, oo, allow_spin);
if (unlikely(!slab)) {
+ if (s->flags & SLAB_CACHE_PISA_RO)
+ return NULL;
oo = s->min;
alloc_gfp = flags;
/*
@@ -8182,6 +8189,11 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
if (s->flags & SLAB_RECLAIM_ACCOUNT)
s->allocflags |= __GFP_RECLAIMABLE;
+ if (s->flags & SLAB_CACHE_PISA_RO) {
+ s->allocflags |= __GFP_PISA_RO | __GFP_ZERO;
+ order = clamp_t(unsigned int, order, 2, PAGE_ALLOC_COSTLY_ORDER);
+ }
+
/*
* For kmalloc caches we enable sheaves later by
* bootstrap_kmalloc_sheaves() to avoid recursion.
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (16 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
` (7 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
The linear part of most transmitted skbs is a small head from the
skbuff_small_head cache. To avoid per-packet IOMMU map/unmap operations
or bounce-buffer copies for headers, allow backing small TX skb heads
with PISA memory.
When per-device pisa_tx_skb_heads is enabled, create a parallel
skbuff_small_head_pisa slab cache backed by zeroed, read-only PISA
folios (SLAB_CACHE_PISA_RO). Eligible TX paths passing __GFP_PISA_RO
allocate small heads from this cache and fall back to skbuff_small_head
on failure.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/base/core.c | 4 +-
drivers/iommu/pisa-alloc.c | 133 ++++++++++++++++++++++++++++++++-----
drivers/iommu/pisa-kunit.c | 5 +-
include/linux/device.h | 6 ++
include/linux/pisa.h | 6 ++
include/net/sock.h | 74 ++++++++++++++++++---
net/core/skbuff.c | 39 ++++++++---
net/core/sock.c | 23 +++++--
net/ipv4/ip_output.c | 8 ++-
net/ipv4/tcp.c | 2 +-
net/ipv4/tcp_output.c | 12 ++--
net/ipv6/ip6_output.c | 8 ++-
12 files changed, 263 insertions(+), 57 deletions(-)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index ade0c94b02ae7..efc1f9b9d255d 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2951,7 +2951,7 @@ static DEVICE_ATTR_RW(name)
PISA_BOOL_ATTR(pisa_coherent);
PISA_BOOL_ATTR(pisa_rx_bufs);
PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
-PISA_BOOL_ATTR(pisa_tx_skb_heads);
+PISA_SET_ATTR(pisa_tx_skb_heads, net_pisa_tx_skb_heads_set);
static struct attribute *dev_attr_pisa[] = {
&dev_attr_pisa_coherent.attr,
@@ -3113,6 +3113,8 @@ static void device_remove_attrs(struct device *dev)
device_remove_group(dev, &dev_attr_pisa_group);
if (dev->pisa_tx_bufs)
net_pisa_tx_bufs_set(dev, false);
+ if (dev->pisa_tx_skb_heads)
+ net_pisa_tx_skb_heads_set(dev, false);
}
if (dev->physical_location) {
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 3b5c49d143a7e..7df2ecde32916 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -37,6 +37,7 @@
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
+#include <net/hotdata.h>
#include <net/sock.h>
#include "dma-iommu.h"
@@ -922,37 +923,133 @@ int net_pisa_tx_bufs_set(struct device *dev, bool val)
}
/*
- * The sysfs attribute pisa_tx_bufs lives on the physical DMA parent
- * (dev->dev.parent), whereas a socket route (dst->dev) may point to an upper
- * virtual net_device (bond, VLAN, ipvlan, etc.). Walking lower devices via
- * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
- * expensive, so dev->pisa_tx_stamp caches the walk result on the net_device
- * and refreshes it at most once every 256 jiffies:
- * bit 0 (PISA_TX_STAMP_OK): pisa_tx_bufs enabled on lower device(s)
+ * dev->pisa_tx_skb_heads: when set on an egress device, small TX skb heads
+ * (__GFP_PISA_RO) come from a copy of skbuff_small_head whose slab pages are
+ * PISA blocks. The linear part of most transmitted skbs is then mapped
+ * through the PISA window without per-driver bounce buffers. Several heads
+ * share a slab page and each one ends with its skb_shared_info, so:
+ * - the allocators are TX-only (DMA_TO_DEVICE): their PISA pages are mapped device
+ * read-only, and a DMA mapping that wants device writes gets a per-buffer
+ * IOMMU mapping, as for any kmalloc() buffer;
+ * - slab pages are zeroed when taken from the allocator, so the device can read
+ * skb heads (live or freed) but never older contents of the page.
+ * Heads are freed with kfree() whichever cache they came from, so the key
+ * can be toggled at any time. The cache and allocators are created on the first
+ * write and never destroyed.
+ */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
+EXPORT_SYMBOL(net_pisa_tx_skb_heads_key);
+
+static struct kmem_cache *pisa_skb_small_head_cache __read_mostly;
+
+/* Lazily initialize skbuff_small_head_pisa on first enable. */
+static int pisa_skb_head_init(void)
+{
+ unsigned int size = kmem_cache_size(net_hotdata.skb_small_head_cache);
+ struct kmem_cache_args args = {
+ /* As skbuff_small_head, see skb_init(). */
+ .useroffset = 0,
+ .usersize = SKB_WITH_OVERHEAD(size),
+ .sheaf_capacity = 32,
+ };
+ static DEFINE_MUTEX(mutex);
+ struct kmem_cache *s;
+ int ret = 0;
+
+ mutex_lock(&mutex);
+ if (pisa_skb_small_head_cache)
+ goto out;
+
+ /* No slab is allocated before the cache is published below. */
+ s = kmem_cache_create("skbuff_small_head_pisa", size, &args,
+ SLAB_HWCACHE_ALIGN | SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+ if (!s) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ WRITE_ONCE(pisa_skb_small_head_cache, s);
+out:
+ mutex_unlock(&mutex);
+ return ret;
+}
+
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+ static DEFINE_MUTEX(lock);
+
+ if (val) {
+ int ret = pisa_skb_head_init();
+
+ if (ret)
+ return ret;
+ }
+
+ mutex_lock(&lock);
+ if (dev->pisa_tx_skb_heads != val) {
+ WRITE_ONCE(dev->pisa_tx_skb_heads, val);
+ if (val)
+ static_branch_inc(&net_pisa_tx_skb_heads_key);
+ else
+ static_branch_dec(&net_pisa_tx_skb_heads_key);
+ }
+ mutex_unlock(&lock);
+ return 0;
+}
+
+void *pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+ if (!static_branch_unlikely(&net_pisa_tx_skb_heads_key))
+ return NULL;
+ return kmem_cache_alloc_node(pisa_skb_small_head_cache,
+ flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node);
+}
+
+/*
+ * The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the
+ * physical DMA parent (dev->dev.parent), whereas a socket route (dst->dev)
+ * may point to an upper virtual net_device (bond, VLAN, ipvlan, etc.).
+ * Walking lower devices via netdev_walk_all_lower_dev_rcu() on every TX
+ * allocation would be too expensive, so dev->pisa_tx_stamp caches the walk
+ * result on the net_device and refreshes it at most once every 256 jiffies:
+ * bit 0 (PISA_TX_STAMP_BUFS): pisa_tx_bufs enabled on lower device(s)
+ * bit 1 (PISA_TX_STAMP_HEADS): pisa_tx_skb_heads enabled on lower device(s)
* bits 31..8 (PISA_TX_STAMP_MASK): upper 24 bits of (u32)jiffies
*/
-static int netdev_pisa_tx_bufs_walk(struct net_device *dev, struct netdev_nested_priv *priv)
+static u32 netdev_pisa_tx_flags(const struct net_device *dev)
{
- return dev->dev.parent && READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ struct device *parent = dev->dev.parent;
+ u32 flags = 0;
+
+ if (parent) {
+ if (READ_ONCE(parent->pisa_tx_bufs))
+ flags |= PISA_TX_STAMP_BUFS;
+ if (READ_ONCE(parent->pisa_tx_skb_heads))
+ flags |= PISA_TX_STAMP_HEADS;
+ }
+ return flags;
}
-/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
-bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+static int netdev_pisa_tx_walk(struct net_device *dev, struct netdev_nested_priv *priv)
{
- bool ok;
+ priv->data = (void *)((uintptr_t)priv->data | netdev_pisa_tx_flags(dev));
+ return 0;
+}
+/* Slow path of __netdev_tx_pisa_ok(): refresh dev->pisa_tx_stamp. */
+u32 netdev_tx_pisa_refresh(struct net_device *dev, u32 now)
+{
if (dev->dev.parent) {
- ok = READ_ONCE(dev->dev.parent->pisa_tx_bufs);
+ now |= netdev_pisa_tx_flags(dev);
} else {
+ struct netdev_nested_priv priv = {};
+
rcu_read_lock();
- ok = netdev_walk_all_lower_dev_rcu(dev,
- netdev_pisa_tx_bufs_walk,
- NULL) != 0;
+ netdev_walk_all_lower_dev_rcu(dev, netdev_pisa_tx_walk, &priv);
rcu_read_unlock();
+ now |= (uintptr_t)priv.data;
}
- now |= ok ? PISA_TX_STAMP_OK : 0;
WRITE_ONCE(dev->pisa_tx_stamp, now);
- return now & PISA_TX_STAMP_OK;
+ return now;
}
EXPORT_SYMBOL(netdev_tx_pisa_refresh);
#endif /* CONFIG_NET */
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 9cc1328893ace..3fc17024bc9d0 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -489,9 +489,10 @@ static void test_dev_pisa_attrs(struct kunit *test)
}
kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
- if (IS_ENABLED(CONFIG_NET))
+ if (IS_ENABLED(CONFIG_NET)) {
kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
- kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+ }
device_unregister(dev);
}
diff --git a/include/linux/device.h b/include/linux/device.h
index 08f511b978ded..d588ae27c960c 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -816,11 +816,17 @@ struct device {
#if defined(CONFIG_NET) && defined(CONFIG_PISA)
int net_pisa_tx_bufs_set(struct device *dev, bool val);
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val);
#else
static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
{
return -EOPNOTSUPP;
}
+
+static inline int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+ return -EOPNOTSUPP;
+}
#endif
#define __create_dev_flag_accessors(accessor_name, flag_name) \
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index b9c5197f2d88d..452b8fa20c8d7 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -113,6 +113,7 @@ unsigned long pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
struct folio *pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid);
bool pisa_has_free_blocks_rx(const struct device *dev, unsigned int order, int nid);
bool pisa_page_can_recycle(const struct page *page);
+void *pisa_skb_head_alloc_node(gfp_t flags, int node);
static inline gfp_t pisa_rx_gfp(const struct device *dev)
{
@@ -169,6 +170,11 @@ static inline bool pisa_free_coherent(struct page *page, size_t size)
return false;
}
+static inline void *pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+ return NULL;
+}
+
static inline bool pisa_page_can_recycle(const struct page *page)
{
return true;
diff --git a/include/net/sock.h b/include/net/sock.h
index 73906351d11db..78f16b76dfc3b 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1900,6 +1900,10 @@ int sk_getsockopt(struct sock *sk, int level, int optname,
sockptr_t optval, sockptr_t optlen);
int sock_gettstamp(struct socket *sock, void __user *userstamp,
bool timeval, bool time32);
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order,
+ gfp_t gfp);
struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
unsigned long data_len, int noblock,
int *errcode, int max_page_order);
@@ -2671,20 +2675,27 @@ static inline struct page_frag *sk_page_frag(struct sock *sk)
}
/* 256-jiffy cache in dev->pisa_tx_stamp; see netdev_tx_pisa_refresh(). */
-#define PISA_TX_STAMP_OK BIT(0)
+#define PISA_TX_STAMP_BUFS BIT(0)
+#define PISA_TX_STAMP_HEADS BIT(1)
#define PISA_TX_STAMP_MASK (~0xffU)
#ifdef CONFIG_PISA
DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
static inline bool net_pisa_tx_bufs_enabled(void)
{
return static_branch_unlikely(&net_pisa_tx_bufs_key);
}
-bool netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+static inline bool net_pisa_tx_skb_heads_enabled(void)
+{
+ return static_branch_unlikely(&net_pisa_tx_skb_heads_key);
+}
-static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+u32 netdev_tx_pisa_refresh(struct net_device *dev, u32 now);
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev, u32 flag)
{
u32 stamp, now;
@@ -2692,9 +2703,9 @@ static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
return false;
stamp = READ_ONCE(dev->pisa_tx_stamp);
now = (u32)jiffies & PISA_TX_STAMP_MASK;
- if (likely((stamp & PISA_TX_STAMP_MASK) == now))
- return stamp & PISA_TX_STAMP_OK;
- return netdev_tx_pisa_refresh((struct net_device *)dev, now);
+ if (unlikely((stamp & PISA_TX_STAMP_MASK) != now))
+ stamp = netdev_tx_pisa_refresh((struct net_device *)dev, now);
+ return stamp & flag;
}
#else
static inline bool net_pisa_tx_bufs_enabled(void)
@@ -2702,7 +2713,12 @@ static inline bool net_pisa_tx_bufs_enabled(void)
return false;
}
-static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
+static inline bool net_pisa_tx_skb_heads_enabled(void)
+{
+ return false;
+}
+
+static inline bool __netdev_tx_pisa_ok(const struct net_device *dev, u32 flag)
{
return false;
}
@@ -2714,20 +2730,58 @@ static inline bool __netdev_tx_pisa_ok(const struct net_device *dev)
* active, so only data that leaves the host in clear qualifies: IPsec payload
* is still plaintext here.
*/
-static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+static inline bool __dst_tx_pisa_ok(const struct dst_entry *dst, u32 flag)
{
bool ok = false;
- if (!net_pisa_tx_bufs_enabled() || !dst)
+ if (!dst)
return false;
rcu_read_lock();
if (!dst_xfrm(dst))
- ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst));
+ ok = __netdev_tx_pisa_ok(dst_dev_rcu(dst), flag);
rcu_read_unlock();
return ok;
}
+static inline bool dst_tx_pisa_ok(const struct dst_entry *dst)
+{
+ return net_pisa_tx_bufs_enabled() &&
+ __dst_tx_pisa_ok(dst, PISA_TX_STAMP_BUFS);
+}
+
+static inline gfp_t pisa_tx_gfp_dst(const struct dst_entry *dst)
+{
+ if (net_pisa_tx_skb_heads_enabled() &&
+ __dst_tx_pisa_ok(dst, PISA_TX_STAMP_HEADS))
+ return __GFP_PISA_RO;
+ return 0;
+}
+
+static inline gfp_t pisa_tx_gfp_sk(const struct sock *sk)
+{
+ if (net_pisa_tx_skb_heads_enabled() && sk) {
+ bool ok;
+
+ rcu_read_lock();
+ ok = __dst_tx_pisa_ok(rcu_dereference(sk->sk_dst_cache),
+ PISA_TX_STAMP_HEADS);
+ rcu_read_unlock();
+ if (ok)
+ return __GFP_PISA_RO;
+ }
+ return 0;
+}
+
+static inline struct sk_buff *sock_alloc_send_skb_dst(struct sock *sk,
+ unsigned long size,
+ int noblock, int *errcode,
+ const struct dst_entry *dst)
+{
+ return __sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0,
+ pisa_tx_gfp_dst(dst));
+}
+
/* Is @pfrag the task's frag reserved for PISA-eligible data? */
static inline bool is_pisa_pfrag(const struct page_frag *pfrag)
{
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 8da53d5b76fc2..04ae3bb4bd905 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -93,6 +93,7 @@
#include <linux/user_namespace.h>
#include <linux/indirect_call_wrapper.h>
#include <linux/textsearch.h>
+#include <linux/pisa.h>
#include "dev.h"
#include "devmem.h"
@@ -607,16 +608,23 @@ static void *kmalloc_pfmemalloc(size_t obj_size, gfp_t flags, int node)
static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node,
struct sk_buff *skb)
{
+ bool pisa = flags & __GFP_PISA_RO;
size_t obj_size;
void *obj;
+ flags &= ~__GFP_PISA_RW;
obj_size = SKB_HEAD_ALIGN(*size);
if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE &&
!(flags & KMALLOC_NOT_NORMAL_BITS)) {
+ *size = SKB_SMALL_HEAD_CACHE_SIZE;
+ if (unlikely(pisa)) {
+ obj = pisa_skb_head_alloc_node(flags, node);
+ if (likely(obj))
+ goto out;
+ }
obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache,
flags | __GFP_NOMEMALLOC | __GFP_NOWARN,
node);
- *size = SKB_SMALL_HEAD_CACHE_SIZE;
if (likely(obj))
goto out;
/* Try again but now we are using pfmemalloc reserves */
@@ -2144,6 +2152,15 @@ void skb_copy_header(struct sk_buff *new, const struct sk_buff *old)
}
EXPORT_SYMBOL(skb_copy_header);
+static inline gfp_t skb_head_pisa_gfp(const struct sk_buff *skb)
+{
+ if (net_pisa_tx_skb_heads_enabled() &&
+ !skb->head_frag &&
+ is_pisa_page(virt_to_page(skb->head)))
+ return __GFP_PISA_RO;
+ return 0;
+}
+
static inline int skb_alloc_rx_flag(const struct sk_buff *skb)
{
if (skb_pfmemalloc(skb))
@@ -2182,7 +2199,7 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
headerlen = skb_headroom(skb);
size = skb_end_offset(skb) + skb->data_len;
- n = __alloc_skb(size, gfp_mask,
+ n = __alloc_skb(size, gfp_mask | skb_head_pisa_gfp(skb),
skb_alloc_rx_flag(skb), NUMA_NO_NODE);
if (!n)
return NULL;
@@ -2226,7 +2243,8 @@ struct sk_buff *__pskb_copy_fclone(struct sk_buff *skb, int headroom,
{
unsigned int size = skb_headlen(skb) + headroom;
int flags = skb_alloc_rx_flag(skb) | (fclone ? SKB_ALLOC_FCLONE : 0);
- struct sk_buff *n = __alloc_skb(size, gfp_mask, flags, NUMA_NO_NODE);
+ struct sk_buff *n = __alloc_skb(size, gfp_mask | skb_head_pisa_gfp(skb),
+ flags, NUMA_NO_NODE);
if (!n)
goto out;
@@ -2315,7 +2333,8 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
goto nodata;
size = SKB_WITH_OVERHEAD(size);
@@ -2524,7 +2543,8 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
oldheadroom = skb_headroom(skb);
n = __alloc_skb(newheadroom + skb->len + newtailroom,
- gfp_mask, skb_alloc_rx_flag(skb),
+ gfp_mask | skb_head_pisa_gfp(skb),
+ skb_alloc_rx_flag(skb),
NUMA_NO_NODE);
if (!n)
return NULL;
@@ -5025,7 +5045,8 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
hsize = len;
nskb = __alloc_skb(hsize + doffset + headroom,
- GFP_ATOMIC, skb_alloc_rx_flag(head_skb),
+ GFP_ATOMIC | skb_head_pisa_gfp(head_skb),
+ skb_alloc_rx_flag(head_skb),
NUMA_NO_NODE);
if (unlikely(!nskb))
@@ -6974,7 +6995,8 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
return -ENOMEM;
size = SKB_WITH_OVERHEAD(size);
@@ -7096,7 +7118,8 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off,
if (skb_pfmemalloc(skb))
gfp_mask |= __GFP_MEMALLOC;
- data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+ data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb),
+ NUMA_NO_NODE, NULL);
if (!data)
return -ENOMEM;
size = SKB_WITH_OVERHEAD(size);
diff --git a/net/core/sock.c b/net/core/sock.c
index 695fefe316925..0ddb5e57f4969 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -3061,9 +3061,10 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
* Generic send/receive buffer handlers
*/
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
- unsigned long data_len, int noblock,
- int *errcode, int max_page_order)
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order,
+ gfp_t gfp)
{
struct sk_buff *skb;
long timeo;
@@ -3092,7 +3093,8 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
timeo = sock_wait_for_wmem(sk, timeo);
}
skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
- errcode, sk->sk_allocation);
+ errcode,
+ sk->sk_allocation | (!data_len ? gfp : 0));
if (skb)
skb_set_owner_w(skb, sk);
return skb;
@@ -3103,6 +3105,16 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
*errcode = err;
return NULL;
}
+EXPORT_SYMBOL(__sock_alloc_send_pskb);
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+ unsigned long data_len, int noblock,
+ int *errcode, int max_page_order)
+{
+ return __sock_alloc_send_pskb(sk, header_len, data_len, noblock,
+ errcode, max_page_order,
+ pisa_tx_gfp_sk(sk));
+}
EXPORT_SYMBOL(sock_alloc_send_pskb);
int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
@@ -3322,7 +3334,8 @@ bool skb_page_frag_refill_dev(unsigned int sz, struct page_frag *pfrag,
const struct dst_entry *dst = skb_dst(skb);
rcu_read_lock();
- if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL)))
+ if (__netdev_tx_pisa_ok(skb->dev ?: (dst ? dst_dev_rcu(dst) : NULL),
+ PISA_TX_STAMP_BUFS))
gfp |= __GFP_PISA_RO;
rcu_read_unlock();
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index a5ea530bc8b5b..c49a33355ea45 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1134,14 +1134,16 @@ static int __ip_append_data(struct sock *sk,
alloclen += alloc_extra;
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk, alloclen,
- (flags & MSG_DONTWAIT), &err);
+ skb = sock_alloc_send_skb_dst(sk, alloclen,
+ flags & MSG_DONTWAIT,
+ &err, &rt->dst);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
skb = alloc_skb(alloclen,
- sk->sk_allocation);
+ sk->sk_allocation |
+ pisa_tx_gfp_dst(&rt->dst));
if (unlikely(!skb))
err = -ENOBUFS;
}
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index b8d44a0638914..038063717951c 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -931,7 +931,7 @@ struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp,
{
struct sk_buff *skb;
- skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp);
+ skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp | pisa_tx_gfp_sk(sk));
if (likely(skb)) {
bool mem_scheduled;
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index cf3ddc055d48d..b7f04a9df199a 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -3892,8 +3892,8 @@ void tcp_send_fin(struct sock *sk)
}
} else {
skb = alloc_skb_fclone(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC |
- __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) |
+ pisa_tx_gfp_sk(sk));
if (unlikely(!skb))
return;
@@ -3921,7 +3921,7 @@ void tcp_send_active_reset(struct sock *sk, enum sk_rst_reason reason)
TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS);
/* NOTE: No TCP options attached and we never retransmit this. */
- skb = alloc_skb(MAX_TCP_HEADER, priority);
+ skb = alloc_skb(MAX_TCP_HEADER, priority | pisa_tx_gfp_sk(sk));
if (!skb) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED);
return;
@@ -4011,7 +4011,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
int mss;
u64 now;
- skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC);
+ skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC | pisa_tx_gfp_dst(dst));
if (unlikely(!skb)) {
dst_release(dst);
return NULL;
@@ -4548,7 +4548,7 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt, u16 flags)
* sock.
*/
buff = alloc_skb(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) | pisa_tx_gfp_sk(sk));
if (unlikely(!buff)) {
struct inet_connection_sock *icsk = inet_csk(sk);
unsigned long delay;
@@ -4601,7 +4601,7 @@ static int tcp_xmit_probe_skb(struct sock *sk, int urgent, int mib)
/* We don't queue it, tcp_transmit_skb() sets ownership. */
skb = alloc_skb(MAX_TCP_HEADER,
- sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+ sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) | pisa_tx_gfp_sk(sk));
if (!skb)
return -1;
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 957c38a08680c..47a6fea3456fe 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1695,14 +1695,16 @@ static int __ip6_append_data(struct sock *sk,
goto error;
}
if (transhdrlen) {
- skb = sock_alloc_send_skb(sk, alloclen,
- (flags & MSG_DONTWAIT), &err);
+ skb = sock_alloc_send_skb_dst(sk, alloclen,
+ flags & MSG_DONTWAIT,
+ &err, &rt->dst);
} else {
skb = NULL;
if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
2 * sk->sk_sndbuf)
skb = alloc_skb(alloclen,
- sk->sk_allocation);
+ sk->sk_allocation |
+ pisa_tx_gfp_dst(&rt->dst));
if (unlikely(!skb))
err = -ENOBUFS;
}
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (17 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
` (6 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
MSG_ZEROCOPY sends map each user page with a per-buffer IOMMU mapping
and IOTLB flush. Moving user TX buffers into PISA blocks once allows
subsequent zerocopy sends to reuse pre-mapped 2MB IOMMU entries.
Add drivers/iommu/pisa-user.c to migrate private anonymous order-0 user
pages into per-node DMA_TO_DEVICE PISA allocators, controlled by
pisa.user_max_4K_pages (0 = disabled):
- Explicit migration via madvise(MADV_PISA) and MADV_NOPISA (also
supported via process_madvise()).
- Automatic best-effort migration on MSG_ZEROCOPY sends in
zerocopy_fill_skb_from_iter() when the egress device has pisa_tx_bufs
enabled. For TCP, pages that cannot be migrated fall back to the
regular copy path into PISA TX page_frags.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/Makefile | 2 +-
drivers/iommu/pisa-alloc.c | 2 +
drivers/iommu/pisa-priv.h | 2 +
drivers/iommu/pisa-user.c | 278 +++++++++++++++++++++++++
include/linux/pisa.h | 28 +++
include/linux/swap.h | 1 +
include/uapi/asm-generic/mman-common.h | 2 +
mm/madvise.c | 11 +
net/core/datagram.c | 82 +++++++-
net/core/skbuff.c | 3 +-
net/ipv4/tcp.c | 12 +-
11 files changed, 416 insertions(+), 7 deletions(-)
create mode 100644 drivers/iommu/pisa-user.c
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 125cf7e813cdc..cd14f80a594a7 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
-obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o
+obj-$(CONFIG_PISA) += pisa-meta.o pisa-alloc.o pisa-node.o pisa-user.o
ifeq ($(CONFIG_IOMMU_DMA),y)
obj-$(CONFIG_PISA) += pisa-map.o
endif
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 7df2ecde32916..1930135880d2b 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -1756,6 +1756,8 @@ static int __init pisa_init(void)
}
}
+ pisa_user_boot_enable();
+
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
if (!pisa_wq)
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index ef3abb1db9049..fe1bff0bba322 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -185,6 +185,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
+ struct pisa_alloc *user_pa; /* user page migration pa */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -278,6 +279,7 @@ void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+void pisa_user_boot_enable(void);
#endif /* CONFIG_PISA */
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
new file mode 100644
index 0000000000000..a16909d429aca
--- /dev/null
+++ b/drivers/iommu/pisa-user.c
@@ -0,0 +1,278 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA user and page-cache folios: allocate or migrate order-0 folios into
+ * per-node PISA allocators so zero-copy TX and block I/O use 2MB IOMMU mappings.
+ * Disabled on CoCo guests (user PTEs map pages encrypted).
+ */
+#include <linux/cc_platform.h>
+#include <linux/pisa.h>
+#include <linux/gfp.h>
+#include <linux/migrate.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/nodemask.h>
+#include <linux/pagewalk.h>
+#include <linux/swap.h>
+
+#include "pisa-priv.h"
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+
+/* Soft cap on 4KB user pages held in PISA blocks (0 = disabled). */
+static unsigned long pisa_user_max_4k_pages __read_mostly;
+DEFINE_STATIC_KEY_FALSE(pisa_user_key);
+
+static bool pisa_user_init(void);
+
+static int pisa_user_max_set(const char *val, const struct kernel_param *kp)
+{
+ unsigned long max_pages;
+ int ret;
+
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ return -EOPNOTSUPP;
+ ret = kstrtoul(val, 0, &max_pages);
+ if (ret)
+ return ret;
+ WRITE_ONCE(pisa_user_max_4k_pages, max_pages);
+ if (max_pages) {
+ if (!pisa_nodes[first_memory_node].pa[0][0])
+ return 0;
+ if (!pisa_user_init())
+ return -ENOMEM;
+ static_branch_enable(&pisa_user_key);
+ } else {
+ static_branch_disable(&pisa_user_key);
+ }
+ return 0;
+}
+
+static const struct kernel_param_ops pisa_user_max_ops = {
+ .set = pisa_user_max_set,
+ .get = param_get_ulong,
+};
+module_param_cb(user_max_4K_pages, &pisa_user_max_ops, &pisa_user_max_4k_pages, 0644);
+
+/* True once per-node user allocators have been allocated, serialized by pisa_user_lock. */
+static bool pisa_user_ready __read_mostly;
+static DEFINE_MUTEX(pisa_user_lock);
+
+/* Lazily create per-node user allocators on first use (process context). */
+static bool pisa_user_init(void)
+{
+ int nid;
+
+ if (!READ_ONCE(pisa_user_max_4k_pages) || cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+ return false;
+ /* Pairs with smp_store_release() below: the allocators are visible. */
+ if (smp_load_acquire(&pisa_user_ready))
+ return true;
+ if (pisa_meta_init())
+ return false;
+ mutex_lock(&pisa_user_lock);
+ for_each_node_state(nid, N_MEMORY) {
+ if (!pisa_nodes[nid].user_pa)
+ pisa_nodes[nid].user_pa = pisa_create(nid, 0, DMA_TO_DEVICE);
+ if (!pisa_nodes[nid].user_pa)
+ goto out; /* retried on the next call */
+ }
+ /* Publish pisa_nodes[].user_pa to lockless readers. */
+ smp_store_release(&pisa_user_ready, true);
+out:
+ mutex_unlock(&pisa_user_lock);
+ return pisa_user_ready;
+}
+
+void pisa_user_boot_enable(void)
+{
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) {
+ WRITE_ONCE(pisa_user_max_4k_pages, 0);
+ return;
+ }
+ if (READ_ONCE(pisa_user_max_4k_pages) && pisa_user_init())
+ static_branch_enable(&pisa_user_key);
+}
+
+/* User pages currently held in PISA blocks. */
+static unsigned long pisa_user_in_use(void)
+{
+ unsigned long used = 0;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ const struct pisa_alloc *pa = pisa_nodes[nid].user_pa;
+
+ used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ }
+ return used;
+}
+
+/* Pages that can still be moved in before reaching pisa.user_max_4K_pages. */
+static long pisa_user_budget(void)
+{
+ return (long)(READ_ONCE(pisa_user_max_4k_pages) - pisa_user_in_use());
+}
+
+/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
+static bool pisa_user_want(struct folio *folio, bool to_pisa)
+{
+ if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
+ folio_test_ksm(folio) || folio_test_large(folio) ||
+ folio_maybe_mapped_shared(folio))
+ return false;
+ return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+}
+
+/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
+static bool pisa_user_busy(struct folio *folio, int extra)
+{
+ return folio_ref_count(folio) > folio_expected_ref_count(folio) + extra;
+}
+
+struct pisa_user_mtc {
+ gfp_t gfp;
+ long budget;
+};
+
+/* new_folio_t for migration into PISA memory. */
+static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
+{
+ struct pisa_user_mtc *mtc = (void *)private;
+ int nid = folio_nid(src);
+ struct page *page;
+
+ if (mtc->budget <= 0)
+ return NULL;
+ page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+ if (!page)
+ return NULL;
+ if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
+ put_page(page);
+ mtc->budget = 0;
+ return NULL;
+ }
+ mtc->budget--;
+ return page_folio(page);
+}
+
+/* new_folio_t for migration out of PISA memory: allocators never serve these. */
+static struct folio *pisa_user_old_folio(struct folio *src, unsigned long private)
+{
+ return __folio_alloc_node(GFP_HIGHUSER_MOVABLE | __GFP_NOWARN, 0, folio_nid(src));
+}
+
+/* Migrate the isolated folios on @list; failures are put back. */
+static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+{
+ struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+ enum migrate_mode mode = MIGRATE_SYNC;
+
+ if (list_empty(list))
+ return;
+ if (to_pisa) {
+ mtc.budget = pisa_user_budget();
+ if (!gfpflags_allow_blocking(gfp))
+ mode = MIGRATE_ASYNC;
+ }
+ if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
+ NULL, (unsigned long)&mtc, mode, MR_SYSCALL, NULL))
+ putback_movable_pages(list);
+}
+
+/**
+ * pisa_upgrade_user_pages - Try to move user pages about to be sent into PISA
+ * @pages: Pages from iov_iter_get_pages2(), one reference each
+ * @nr: Number of entries in @pages
+ *
+ * Best effort, process context. Pages that cannot be moved (e.g. still
+ * referenced by an in-flight skb, or in a large folio) are skipped; for TCP
+ * with pisa_tx_bufs enabled the caller copies them instead.
+ *
+ * Return: false if nothing was attempted and the references are untouched;
+ * true if all references were dropped and the caller must look the pages up
+ * again.
+ */
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+{
+ bool drained = false;
+ LIST_HEAD(list);
+ unsigned int i, want = 0;
+
+ if (!pisa_user_init())
+ return false;
+ /* Common case: all pages already in PISA, no budget check needed. */
+ for (i = 0; i < nr; i++)
+ want += pisa_user_want(page_folio(pages[i]), true);
+ if (!want)
+ return false;
+ if (pisa_user_budget() <= 0)
+ return false;
+ for (i = 0; i < nr; i++) {
+ struct folio *folio = page_folio(pages[i]);
+
+ if (!pisa_user_want(folio, true))
+ continue;
+ /* A freshly faulted page may still sit in a per-CPU LRU batch. */
+ if (!drained && !folio_test_lru(folio)) {
+ lru_add_drain();
+ drained = true;
+ }
+ /* The reference from iov_iter_get_pages2() is expected. */
+ if (pisa_user_busy(folio, 1))
+ continue;
+ isolate_folio_to_list(folio, &list);
+ }
+ if (list_empty(&list))
+ return false;
+ /* Migration fails on pages with references it does not expect. */
+ for (i = 0; i < nr; i++)
+ put_page(pages[i]);
+ pisa_user_migrate(&list, true,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ return true;
+}
+
+#define PISA_USER_BATCH 512
+
+/**
+ * pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
+ * @vma: VMA, with the mmap lock held for reading
+ * @start: Start address
+ * @end: End address
+ * @to_pisa: true to move present pages into PISA, false to move them out
+ *
+ * Return: 0, -EINVAL for unsupported mappings, or -EOPNOTSUPP if disabled.
+ */
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool to_pisa)
+{
+ gfp_t gfp = GFP_USER | __GFP_NOWARN;
+ struct folio_walk fw;
+ unsigned long addr;
+ unsigned int n = 0;
+ LIST_HEAD(list);
+
+ if (vma->vm_flags & (VM_SHARED | VM_IO | VM_PFNMAP | VM_HUGETLB))
+ return -EINVAL;
+ if (to_pisa && !pisa_user_init())
+ return -EOPNOTSUPP;
+ lru_add_drain();
+ for (addr = start; addr < end; addr += PAGE_SIZE) {
+ struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
+
+ if (folio) {
+ if (pisa_user_want(folio, to_pisa) && !pisa_user_busy(folio, 0) &&
+ isolate_folio_to_list(folio, &list))
+ n++;
+ folio_walk_end(&fw, vma);
+ }
+ if (n == PISA_USER_BATCH) {
+ pisa_user_migrate(&list, to_pisa, gfp);
+ n = 0;
+ }
+ cond_resched();
+ }
+ pisa_user_migrate(&list, to_pisa, gfp);
+ return 0;
+}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 452b8fa20c8d7..d11fd3d663dc1 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -132,6 +132,18 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
put_page(tsk->task_frag_pisa.page);
}
+/* User pages in PISA blocks, see drivers/iommu/pisa-user.c */
+DECLARE_STATIC_KEY_FALSE(pisa_user_key);
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr);
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+ unsigned long end, bool to_pisa);
+
+/* Is pisa.user_max_4K_pages non-zero? */
+static inline bool pisa_user_enabled(void)
+{
+ return static_branch_unlikely(&pisa_user_key);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -213,6 +225,22 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
{
}
+static inline bool pisa_user_enabled(void)
+{
+ return false;
+}
+
+static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+{
+ return false;
+}
+
+static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+ unsigned long end, bool to_pisa)
+{
+ return -EINVAL;
+}
+
#endif /* CONFIG_PISA */
static inline struct page *
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4f686709d71f0..3736786ccc5a3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -314,6 +314,7 @@ extern unsigned long totalreserve_pages;
/* linux/mm/folio.c */
void folio_add_lru(struct folio *folio);
void folio_mark_accessed(struct folio *folio);
+void lru_add_drain(void);
void lru_add_drain_all(void);
enum lru_cache_drained {
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index ef1c27fa3c570..c4332db8e56ff 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -81,6 +81,8 @@
#define MADV_GUARD_INSTALL 102 /* fatal signal on access to range */
#define MADV_GUARD_REMOVE 103 /* unguard range */
+#define MADV_PISA 210 /* move pages into PISA memory */
+#define MADV_NOPISA 211 /* move pages out of PISA memory */
/* compatibility flags */
#define MAP_FILE 0
diff --git a/mm/madvise.c b/mm/madvise.c
index 0214efd2a014f..0eccb44594b04 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -6,6 +6,7 @@
* Copyright (C) 2002 Christoph Hellwig
*/
+#include <linux/pisa.h>
#include <linux/mman.h>
#include <linux/pagemap.h>
#include <linux/syscalls.h>
@@ -1393,6 +1394,9 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)
return madvise_guard_install(madv_behavior);
case MADV_GUARD_REMOVE:
return madvise_guard_remove(madv_behavior);
+ case MADV_PISA:
+ case MADV_NOPISA:
+ return pisa_madvise(vma, range->start, range->end, behavior == MADV_PISA);
/* The below behaviours update VMAs via madvise_update_vma(). */
@@ -1582,6 +1586,9 @@ madvise_behavior_valid(int behavior)
case MADV_HWPOISON:
#endif
return true;
+ case MADV_PISA:
+ case MADV_NOPISA:
+ return IS_ENABLED(CONFIG_PISA);
default:
return false;
@@ -1596,6 +1603,8 @@ static bool process_madvise_remote_valid(int behavior)
case MADV_PAGEOUT:
case MADV_WILLNEED:
case MADV_COLLAPSE:
+ case MADV_PISA:
+ case MADV_NOPISA:
return true;
default:
return false;
@@ -1782,6 +1791,8 @@ static enum madvise_lock_mode get_lock_mode(struct madvise_behavior *madv_behavi
case MADV_POPULATE_READ:
case MADV_POPULATE_WRITE:
case MADV_COLLAPSE:
+ case MADV_PISA:
+ case MADV_NOPISA:
return MADVISE_MMAP_READ_LOCK;
case MADV_GUARD_INSTALL:
case MADV_GUARD_REMOVE:
diff --git a/net/core/datagram.c b/net/core/datagram.c
index ecc30a6f83023..40f20d0faa859 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -39,6 +39,7 @@
#include <linux/kernel.h>
#include <linux/uaccess.h>
#include <linux/mm.h>
+#include <linux/pisa.h>
#include <linux/interrupt.h>
#include <linux/errno.h>
#include <linux/sched.h>
@@ -632,10 +633,74 @@ int skb_copy_datagram_from_iter_full(struct sk_buff *skb, int offset,
}
EXPORT_SYMBOL(skb_copy_datagram_from_iter_full);
-int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
- struct iov_iter *from, size_t length)
+/*
+ * Upgrade user @pages to PISA memory and, for TCP over a pisa_tx_bufs device,
+ * trim non-PISA trailing pages from the batch.
+ *
+ * Return: false to proceed with attaching *@copied bytes from @pages to @skb,
+ * or true to stop and make the caller return *@ret.
+ */
+static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, size_t *length,
+ const struct sock *sk, struct page **pages, int max_pages,
+ unsigned int orig_len, size_t *start, ssize_t *copied)
+{
+ unsigned int i, nr;
+ ssize_t keep;
+
+ if (!pisa_user_enabled() || !user_backed_iter(from))
+ return false;
+
+ /* Move user pages into PISA memory, then look them up again. */
+ if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE))) {
+ iov_iter_revert(from, *copied);
+ *copied = iov_iter_get_pages2(from, pages, *length, max_pages, start);
+ if (*copied < 0) {
+ *ret = -EFAULT;
+ return true;
+ }
+ }
+
+ /* Only TCP (which sets skb->sk at transmit time) handles short appends/-EXDEV. */
+ if (!sk || skb->sk)
+ return false;
+
+ /* Only plain MSG_ZEROCOPY ubufs (not io_uring/vhost/devmem). */
+ if (skb_zcopy(skb) && skb_zcopy(skb)->ops != &msg_zerocopy_ubuf_ops)
+ return false;
+
+ /* Egress device must have pisa_tx_bufs enabled so the copy uses a PISA page_frag. */
+ if (!dst_tx_pisa_ok(__sk_dst_get(sk)))
+ return false;
+
+ /*
+ * Trim @from to the leading pages that are in PISA memory: drop the
+ * references to the others, give their bytes back to @from, and cap
+ * *@length so that the caller ends after attaching what is left.
+ */
+ nr = DIV_ROUND_UP(*start + *copied, PAGE_SIZE);
+ for (i = 0; i < nr && is_pisa_page(pages[i]); i++)
+ ;
+ if (i == nr)
+ return false;
+
+ keep = i ? i * PAGE_SIZE - *start : 0;
+ while (nr > i)
+ put_page(pages[--nr]);
+ iov_iter_revert(from, *copied - keep);
+ *copied = keep;
+ *length = keep;
+ if (keep)
+ return false;
+
+ *ret = (skb->len != orig_len) ? 0 : -EXDEV;
+ return true;
+}
+
+static int __zerocopy_fill_skb_from_iter(struct sk_buff *skb, struct iov_iter *from,
+ size_t length, const struct sock *sk)
{
int frag = skb_shinfo(skb)->nr_frags;
+ unsigned int orig_len = skb->len;
if (!skb_frags_readable(skb))
return -EFAULT;
@@ -643,7 +708,7 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
while (length && iov_iter_count(from)) {
struct page *head, *last_head = NULL;
struct page *pages[MAX_SKB_FRAGS];
- int refs, order, n = 0;
+ int refs, order, n = 0, ret;
size_t start;
ssize_t copied;
@@ -654,6 +719,9 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
MAX_SKB_FRAGS - frag, &start);
if (copied < 0)
return -EFAULT;
+ if (pisa_zc_iter(&ret, skb, from, &length, sk, pages,
+ MAX_SKB_FRAGS - frag, orig_len, &start, &copied))
+ return ret;
length -= copied;
@@ -703,6 +771,12 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
return 0;
}
+int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
+ struct iov_iter *from, size_t length)
+{
+ return __zerocopy_fill_skb_from_iter(skb, from, length, NULL);
+}
+
static int
zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from,
int length,
@@ -760,7 +834,7 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk,
else if (binding)
ret = zerocopy_fill_skb_from_devmem(skb, from, length, binding);
else
- ret = zerocopy_fill_skb_from_iter(skb, from, length);
+ ret = __zerocopy_fill_skb_from_iter(skb, from, length, sk);
truesize = skb->truesize - orig_size;
if (sk && sk->sk_type == SOCK_STREAM) {
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 04ae3bb4bd905..739d09615638c 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1935,7 +1935,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
err = __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len,
binding);
- if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {
+ if (err == -EFAULT ||
+ ((err == -EMSGSIZE || err == -EXDEV) && skb->len == orig_len)) {
struct sock *save_sk = skb->sk;
/* Streams do not free skb on error. Reset to prev state. */
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 038063717951c..9214d6213cae5 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1128,6 +1128,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
int process_backlog = 0;
int sockc_err = 0;
int zc = 0;
+ bool must_copy = false; /* copy the next chunk of a MSG_ZEROCOPY send */
long timeo;
flags = msg->msg_flags;
@@ -1289,7 +1290,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
if (copy > msg_data_left(msg))
copy = msg_data_left(msg);
- if (zc == 0) {
+ if (zc == 0 || unlikely(must_copy)) {
const struct dst_entry *dst = __sk_dst_get(sk);
bool merge = true;
int i = skb_shinfo(skb)->nr_frags;
@@ -1335,6 +1336,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
page_ref_inc(pfrag->page);
}
pfrag->offset += copy;
+ must_copy = false;
} else if (zc == MSG_ZEROCOPY) {
/* First append to a fragless skb builds initial
* pure zerocopy skb
@@ -1354,6 +1356,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
tcp_mark_push(tp, skb);
goto new_segment;
}
+ /*
+ * The next user page is not in PISA memory: copy this
+ * chunk, without reporting SO_EE_CODE_ZEROCOPY_COPIED.
+ */
+ if (unlikely(err == -EXDEV)) {
+ must_copy = true;
+ continue;
+ }
if (err < 0)
goto do_error;
copy = err;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (18 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
` (5 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When a PISA allocator holds many partially used 2MB pages with low
overall utilization, callers that recycle blocks in place (page_pool,
per-task TX page_frags, and per-CPU block caches) can keep old 2MB pages
pinned indefinitely with only a few blocks in use.
Record @alloc_jiffies in struct pisa_meta when a 2MB page is bound to an
allocator or taken from the allocator's idle list, and add two module
parameters:
- pisa.wasted_pct (1..99%, 0 disables): wasted block percentage
threshold above which compaction kicks in.
- pisa.old_ms (1..100000 ms, 0 disables): age threshold after which a
partially used 2MB page is eligible to stop recycling.
When both parameters are set and an allocator's wasted block fraction
exceeds wasted_pct, pisa_page_can_recycle() returns false for partial
2MB pages older than old_ms (except the head page actively being
filled).
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 48 ++++++++++++++++++++++++++++++++------
drivers/iommu/pisa-priv.h | 2 ++
2 files changed, 43 insertions(+), 7 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 1930135880d2b..5054a96f0f4e9 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -74,6 +74,14 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
+/* Stop recycling partial pages older than @pisa_old_ms when wasted blocks > @pisa_wasted_pct%. */
+#define PISA_OLD_MS_MAX 100000U
+static unsigned int pisa_wasted_pct __read_mostly;
+module_param_named(wasted_pct, pisa_wasted_pct, uint, 0644);
+
+static unsigned int pisa_old_ms __read_mostly;
+module_param_named(old_ms, pisa_old_ms, uint, 0644);
+
/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
unsigned int pisa_scrub_max __read_mostly;
#define PISA_SCRUB_MAX_LIMIT 65536U
@@ -399,6 +407,7 @@ static struct pisa_meta *pisa_pick_dirty_meta(struct pisa_alloc *pa)
meta = list_last_entry(&pa->idle, struct pisa_meta, list);
if (meta->nr_dirty) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
return meta;
}
}
@@ -603,7 +612,7 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
* pisa_page_can_recycle - Check if a block can stay cached in a caller pool
* @page: Block head page
*
- * Called on the recycle fast path (per-CPU cache, TX page_frag, page_pool).
+ * Called on the recycle fast path (per-CPU cache, TX page_frag).
* Fast-path cost is a static-key branch plus a READ_ONCE() of module parameters
* (~1-2 ns when compaction is off); when enabled, it does a lockless RCU +
* metadata array lookup (~5-10 ns).
@@ -613,10 +622,11 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
*/
bool pisa_page_can_recycle(const struct page *page)
{
- unsigned long bpp, in_use, free_blocks, pfn;
+ unsigned int compact, wasted_pct, old_ms;
+ unsigned long bpp, in_use, free_blocks;
const struct pisa_meta *meta;
const struct pisa_alloc *pa;
- unsigned int compact;
+ unsigned long pfn;
bool ret = true;
u8 on_list;
long held;
@@ -625,7 +635,9 @@ bool pisa_page_can_recycle(const struct page *page)
return true;
compact = READ_ONCE(pisa_compact);
- if (!compact)
+ wasted_pct = READ_ONCE(pisa_wasted_pct);
+ old_ms = READ_ONCE(pisa_old_ms);
+ if (!compact && (!wasted_pct || !old_ms))
return true;
pfn = page_to_pfn(page);
@@ -638,6 +650,13 @@ bool pisa_page_can_recycle(const struct page *page)
if (on_list == PISA_LIST_FULL || on_list == PISA_LIST_NONE)
goto out;
+ if (!compact) {
+ unsigned long max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ if (!time_after(jiffies, READ_ONCE(meta->alloc_jiffies) + max_age))
+ goto out;
+ }
+
pa = meta->pa;
if (!pa)
goto out;
@@ -657,12 +676,21 @@ bool pisa_page_can_recycle(const struct page *page)
free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
(unsigned long)held * bpp - in_use : 0;
- ret = free_blocks <= (unsigned long)compact * bpp;
+ if (compact && free_blocks > (unsigned long)compact * bpp) {
+ ret = false;
+ goto out;
+ }
+ if (wasted_pct && old_ms && held > 0 &&
+ free_blocks * 100 > (unsigned long)held * bpp * wasted_pct) {
+ unsigned long max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ ret = !time_after(jiffies, READ_ONCE(meta->alloc_jiffies) + max_age);
+ }
out:
rcu_read_unlock();
return ret;
}
-EXPORT_SYMBOL_GPL(pisa_page_can_recycle);
+EXPORT_SYMBOL(pisa_page_can_recycle);
/* Return @n freed blocks to @pa under a single acquisition of @pa->lock. */
static void pisa_free_bulk(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
@@ -1195,6 +1223,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
if (meta && meta->nr_clean) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(pa->idle_jiffies, jiffies);
return meta;
}
@@ -1218,6 +1247,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
if (!list_empty(&pa->idle)) {
meta = list_last_entry(&pa->idle, struct pisa_meta, list);
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(pa->idle_jiffies, jiffies);
return meta;
}
@@ -1287,6 +1317,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
@@ -1454,8 +1485,10 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
}
if (meta->on_list == PISA_LIST_IDLE_CLEAN ||
- meta->on_list == PISA_LIST_IDLE_DIRTY)
+ meta->on_list == PISA_LIST_IDLE_DIRTY) {
pa->num_idle_pages--;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
+ }
for (i = 0; i < nblocks; i++) {
if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
@@ -1544,6 +1577,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
+ WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
pa->stats.page_alloc_cnt++;
page = pisa_take_contig(meta, nblocks, align_mask);
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index fe1bff0bba322..4d04bb191ecea 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -74,6 +74,7 @@ enum {
* @list: Node in @pa lists or per-node reservoir (pisa_node.pages)
* @llnode: Node in lockless pisa_free_list for async release to the buddy allocator
* @rcu: RCU head used to defer release to the buddy allocator
+ * @alloc_jiffies: Timestamp (jiffies) when the page entered or re-entered use
* @clean_bitmap: Bitmap of zeroed available block indices
* @dirty_bitmap: Bitmap of dirty available block indices
*/
@@ -94,6 +95,7 @@ struct pisa_meta {
struct llist_node llnode;
struct rcu_head rcu;
};
+ unsigned long alloc_jiffies;
DECLARE_BITMAP(clean_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
DECLARE_BITMAP(dirty_bitmap, PISA_MAX_BLOCKS) ____cacheline_aligned;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (19 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
` (4 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
libeth_rx_alloc() is called both when pre-filling RX fill queues in
process context (for indices 0 .. fq->count - 2, when buf->truesize is
still 0) and when refilling RX buffers from NAPI softirq.
Use GFP_KERNEL during the initial process-context ring fill so page_pool
and PISA page allocations (including set_memory_decrypted() under CoCo)
are not forced into GFP_ATOMIC context.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
include/net/libeth/rx.h | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h
index 0e736846c5e8a..de1e5fe44aa40 100644
--- a/include/net/libeth/rx.h
+++ b/include/net/libeth/rx.h
@@ -106,10 +106,17 @@ void libeth_rx_fq_destroy(struct libeth_fq *fq);
static inline dma_addr_t libeth_rx_alloc(const struct libeth_fq_fp *fq, u32 i)
{
struct libeth_fqe *buf = &fq->fqes[i];
+ /* fq->fqes is zero-allocated and pre-filled in process context for
+ * indices [0 .. fq->count - 2]; slot fq->count - 1 is first allocated
+ * in NAPI. Use GFP_KERNEL for initial pre-fill and GFP_ATOMIC after.
+ */
+ gfp_t gfp = (buf->truesize || i == fq->count - 1) ?
+ GFP_ATOMIC : GFP_KERNEL;
buf->truesize = fq->truesize;
- buf->netmem = page_pool_dev_alloc_netmem(fq->pp, &buf->offset,
- &buf->truesize);
+ buf->netmem = page_pool_alloc_netmem(fq->pp, &buf->offset,
+ &buf->truesize,
+ gfp | __GFP_NOWARN);
if (unlikely(!buf->netmem))
return DMA_MAPPING_ERROR;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 21/25] net/gve: Use GFP_KERNEL when pre-filling RX rings
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (20 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
` (3 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When pre-filling DQO and GQI RX rings in process context before NAPI is
enabled (e.g. gve_open), use GFP_KERNEL instead of GFP_ATOMIC so page
allocation (and CoCo page decryption) can sleep instead of dipping into
atomic reserves.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c | 7 ++++++-
drivers/net/ethernet/google/gve/gve_rx.c | 8 ++++++--
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
index 6880d1531ba5e..3b13aa4999ae5 100644
--- a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
@@ -223,13 +223,18 @@ void gve_free_to_page_pool(struct gve_rx_ring *rx,
static int gve_alloc_from_page_pool(struct gve_rx_ring *rx,
struct gve_rx_buf_state_dqo *buf_state)
{
+ /*
+ * Use GFP_KERNEL when pre-filling RX rings in process context (e.g.
+ * gve_open) so PISA metadata init and CoCo page decryption can sleep.
+ */
+ gfp_t gfp = gve_get_napi_enabled(rx->gve) ? GFP_ATOMIC : GFP_KERNEL;
netmem_ref netmem;
buf_state->page_info.buf_size = rx->packet_buffer_truesize;
netmem = page_pool_alloc_netmem(rx->dqo.page_pool,
&buf_state->page_info.page_offset,
&buf_state->page_info.buf_size,
- GFP_ATOMIC);
+ gfp);
if (!netmem)
return -ENOMEM;
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 81ea800e66e94..89f287b678bf0 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -161,12 +161,16 @@ static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
union gve_rx_data_slot *data_slot,
struct gve_rx_ring *rx)
{
+ /*
+ * Use GFP_KERNEL when pre-filling RX rings in process context (e.g.
+ * gve_open) so PISA metadata init and CoCo page decryption can sleep.
+ */
+ gfp_t gfp = gve_get_napi_enabled(priv) ? GFP_ATOMIC : GFP_KERNEL;
struct page *page;
dma_addr_t dma;
int err;
- err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE,
- GFP_ATOMIC);
+ err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
if (err) {
u64_stats_update_begin(&rx->statss);
rx->rx_buf_alloc_fail++;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (21 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
` (2 subsequent siblings)
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
When dev->pisa_rx_bufs is enabled, allocate GQI-RDA RX packet buffers
with pisa_rx_gfp(dev) so they come from the per-node PISA
allocator, and evict plain pages when a free PISA RX block is available;
DQO-RDA uses page_pool, which checks dev->pisa_rx_bufs directly.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/net/ethernet/google/gve/gve_rx.c | 32 ++++++++++++++++--------
1 file changed, 22 insertions(+), 10 deletions(-)
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 89f287b678bf0..85c243bbdeb33 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -4,6 +4,7 @@
* Copyright (C) 2015-2021 Google, Inc.
*/
+#include <linux/pisa.h>
#include "gve.h"
#include "gve_adminq.h"
#include "gve_utils.h"
@@ -170,6 +171,8 @@ static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
dma_addr_t dma;
int err;
+ if (rx->data.raw_addressing)
+ gfp |= pisa_rx_gfp(dev);
err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
if (err) {
u64_stats_update_begin(&rx->statss);
@@ -499,13 +502,19 @@ static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *sl
*(slot_addr) ^= offset;
}
-static int gve_rx_can_recycle_buffer(struct gve_rx_slot_page_info *page_info)
+static int gve_rx_can_recycle_buffer(struct gve_rx_ring *rx,
+ struct gve_rx_slot_page_info *page_info)
{
int pagecount = page_count(page_info->page);
/* This page is not being used by any SKBs - reuse */
- if (pagecount == page_info->pagecnt_bias)
+ if (pagecount == page_info->pagecnt_bias) {
+ if (rx && rx->data.raw_addressing &&
+ !is_pisa_page(page_info->page) &&
+ pisa_has_free_blocks_rx(&rx->gve->pdev->dev, 0, rx->gve->numa_node))
+ return 0;
return 1;
+ }
/* This page is still being used by an SKB - we can't reuse */
else if (pagecount > page_info->pagecnt_bias)
return 0;
@@ -549,7 +558,7 @@ static struct sk_buff *gve_rx_copy_to_pool(struct gve_rx_ring *rx,
copy_page_info = &rx->qpl_copy_pool[pool_idx];
if (!copy_page_info->can_flip) {
- int recycle = gve_rx_can_recycle_buffer(copy_page_info);
+ int recycle = gve_rx_can_recycle_buffer(NULL, copy_page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(rx->gve);
@@ -669,7 +678,7 @@ static struct sk_buff *gve_rx_skb(struct gve_priv *priv, struct gve_rx_ring *rx,
u64_stats_update_end(&rx->statss);
}
} else {
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (unlikely(recycle < 0)) {
gve_schedule_reset(priv);
@@ -975,7 +984,7 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
* owns half the page it is impossible to tell which half. Either
* the whole page is free or it needs to be replaced.
*/
- int recycle = gve_rx_can_recycle_buffer(page_info);
+ int recycle = gve_rx_can_recycle_buffer(rx, page_info);
if (recycle < 0) {
if (!rx->data.raw_addressing)
@@ -987,12 +996,15 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
union gve_rx_data_slot *data_slot =
&rx->data.data_ring[idx];
struct device *dev = &priv->pdev->dev;
- gve_rx_free_buffer(dev, page_info, data_slot);
- page_info->page = NULL;
- if (gve_rx_alloc_buffer(priv, dev, page_info,
- data_slot, rx)) {
+ struct gve_rx_slot_page_info new_info;
+ union gve_rx_data_slot new_slot;
+
+ if (gve_rx_alloc_buffer(priv, dev, &new_info,
+ &new_slot, rx))
break;
- }
+ gve_rx_free_buffer(dev, page_info, data_slot);
+ *page_info = new_info;
+ *data_slot = new_slot;
}
}
fill_cnt++;
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (22 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Extend PISA user/page-cache allocators to support O_DIRECT and buffered
disk I/O, gated per block/controller device through pisa_rx_bufs (read)
and pisa_tx_bufs (write):
- Split per-node user_pa into DMA_BIDIRECTIONAL, DMA_TO_DEVICE, and
DMA_FROM_DEVICE allocators, promoting pages to DMA_BIDIRECTIONAL if
later accessed in the opposite direction.
- Upgrade pinned O_DIRECT user pages in bio_iov_iter_get_pages(), and
allocate/promote order-0 page-cache folios for buffered I/O when PISA
is enabled on the target block device.
- Bypass IOVA allocation in dma_iova_try_alloc() for PISA pages so
blk-mq uses the pre-mapped 2MB window.
- Add pisa.scrub_migrate to compact or demote lingering folios off old,
sparse user_pa 2MB pages.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
block/bio.c | 46 ++++
drivers/iommu/dma-iommu.c | 2 +-
drivers/iommu/pisa-alloc.c | 36 ++--
drivers/iommu/pisa-node.c | 3 +-
drivers/iommu/pisa-priv.h | 21 +-
drivers/iommu/pisa-user.c | 417 ++++++++++++++++++++++++++++++++-----
include/linux/pisa.h | 100 ++++++++-
mm/filemap.c | 15 +-
mm/page-writeback.c | 8 +
mm/readahead.c | 5 +-
net/core/datagram.c | 3 +-
11 files changed, 581 insertions(+), 75 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index b0fcc69de6535..5b43430b5280c 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -6,6 +6,7 @@
#include <linux/swap.h>
#include <linux/bio-integrity.h>
#include <linux/blkdev.h>
+#include <linux/pisa.h>
#include <linux/uio.h>
#include <linux/iocontext.h>
#include <linux/slab.h>
@@ -1316,6 +1317,43 @@ static inline bool bio_iov_bvec_aligned(const struct bio *bio,
}
#endif
+/*
+ * Migrate pinned order-0 user pages in @bio into PISA memory when enabled on
+ * @bio->bi_bdev; returns true if pages were released and the caller must retry.
+ */
+static bool bio_upgrade_pages_to_pisa(struct bio *bio)
+{
+ enum dma_data_direction dir;
+ struct page *pages[64];
+ unsigned int i, nr = 0;
+
+ if (!IS_ENABLED(CONFIG_PISA) || !bio_flagged(bio, BIO_PAGE_PINNED))
+ return false;
+ dir = op_is_write(bio_op(bio)) ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+ if (!is_bdev_pisa_enabled(bio->bi_bdev, dir))
+ return false;
+
+ for (i = 0; i < bio->bi_vcnt && nr < ARRAY_SIZE(pages); i++) {
+ struct page *page = bio->bi_io_vec[i].bv_page;
+
+ if (bvec_nr_pages(&bio->bi_io_vec[i]) == 1 && !is_pisa_page(page)) {
+ get_page(page);
+ pages[nr++] = page;
+ }
+ }
+ if (!nr)
+ return false;
+
+ bio_release_pages(bio, false);
+ bio->bi_vcnt = 0;
+ bio->bi_iter.bi_size = 0;
+ if (!pisa_upgrade_user_pages(pages, nr, dir, false)) {
+ while (nr)
+ put_page(pages[--nr]);
+ }
+ return true;
+}
+
/**
* bio_iov_iter_get_pages - add user or kernel pages to a bio
* @bio: bio to add pages to
@@ -1346,6 +1384,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
{
iov_iter_extraction_t flags = 0;
struct iov_iter saved_iter;
+ bool tried_pisa = false;
int ret;
if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
@@ -1367,6 +1406,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue))
flags |= ITER_ALLOW_P2PDMA;
+retry:
saved_iter = *iter;
do {
ssize_t size;
@@ -1386,6 +1426,12 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
bio->bi_iter.bi_size += size;
} while (iov_iter_count(iter) && !bio_full(bio, 0));
+ if (!tried_pisa && bio_upgrade_pages_to_pisa(bio)) {
+ tried_pisa = true;
+ *iter = saved_iter;
+ goto retry;
+ }
+
if (is_pci_p2pdma_page(bio->bi_io_vec->bv_page))
bio->bi_opf |= REQ_NOMERGE;
ret = bio_iov_iter_align_down(bio, iter, len_align_mask);
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index a0c5ddab7a8b6..acedfe04bde26 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1868,7 +1868,7 @@ bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state,
dma_addr_t addr;
memset(state, 0, sizeof(*state));
- if (!use_dma_iommu(dev))
+ if (!use_dma_iommu(dev) || is_pisa_phys(phys))
return false;
domain = iommu_get_dma_domain(dev);
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index 5054a96f0f4e9..cedb1a62d20e2 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -75,11 +75,10 @@ static unsigned int pisa_compact __read_mostly;
module_param_named(compact, pisa_compact, uint, 0644);
/* Stop recycling partial pages older than @pisa_old_ms when wasted blocks > @pisa_wasted_pct%. */
-#define PISA_OLD_MS_MAX 100000U
-static unsigned int pisa_wasted_pct __read_mostly;
+unsigned int pisa_wasted_pct __read_mostly;
module_param_named(wasted_pct, pisa_wasted_pct, uint, 0644);
-static unsigned int pisa_old_ms __read_mostly;
+unsigned int pisa_old_ms __read_mostly;
module_param_named(old_ms, pisa_old_ms, uint, 0644);
/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
@@ -98,8 +97,6 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
},
};
-static void pisa_schedule_reclaim(void);
-
static int pisa_scrub_set(const char *val, const struct kernel_param *kp)
{
int ret = param_set_uint(val, kp);
@@ -229,18 +226,6 @@ void pisa_release_page(struct pisa_meta *meta)
call_rcu(&meta->rcu, pisa_release_page_rcu);
}
-/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
-enum pisa_meta_list {
- PISA_LIST_NONE,
- PISA_LIST_FULL,
- PISA_LIST_IDLE_CLEAN, /* head of @idle */
- PISA_LIST_IDLE_DIRTY, /* tail of @idle */
- PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
- PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
- PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
- PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
-};
-
static void pisa_reclaim_work_fn(struct work_struct *work);
/* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
@@ -285,7 +270,7 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
}
-static void pisa_schedule_reclaim(void)
+void pisa_schedule_reclaim(void)
{
struct workqueue_struct *wq = pisa_wq ?: system_wq;
unsigned int idle_ms = READ_ONCE(pisa_idle_ms);
@@ -301,7 +286,7 @@ static void pisa_schedule_reclaim(void)
* become available (avail == bpp), either queues @meta to @pisa_free_list (and
* returns true so the caller schedules reclaim) or moves it to @pa->idle.
*/
-static bool pisa_place_meta(struct pisa_meta *meta)
+bool pisa_place_meta(struct pisa_meta *meta)
{
struct pisa_alloc *pa = meta->pa;
unsigned int avail = pisa_meta_avail(meta), bpp = PISA_BLOCKS(pa->order);
@@ -505,6 +490,9 @@ bool pisa_scrub_node(int nid)
struct pisa_alloc *pa;
bool more = false;
+ if (!budget)
+ goto migrate;
+
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
if (pa->nid != nid || !READ_ONCE(pa->scrub_demand))
@@ -526,6 +514,16 @@ bool pisa_scrub_node(int nid)
}
read_unlock(&pisa_allocs_lock);
+migrate:
+ if (READ_ONCE(pisa_scrub_migrate)) {
+ unsigned long rem = !budget ? 512 : (budget > done ? budget - done : 0);
+
+ if (rem && local_clock() < deadline)
+ more |= pisa_user_scrub_node(nid, rem, deadline);
+ else
+ more = true;
+ }
+
return more;
}
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index cd29c146c281e..7469674133620 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -184,7 +184,8 @@ void pisa_schedule_scrub(int nid)
{
struct pisa_node *node;
- if (!READ_ONCE(pisa_scrub_max) || nid == NUMA_NO_NODE)
+ if ((!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate)) ||
+ nid == NUMA_NO_NODE)
return;
node = &pisa_nodes[nid];
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 4d04bb191ecea..1ecfebbaeb381 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -61,6 +61,18 @@ enum {
* - pisa_srcu: protects pisa_domains[] readers against pisa_domain_release().
*/
+/* Value of @meta->on_list. 0 is "not on an allocator list", as for a zeroed @meta. */
+enum pisa_meta_list {
+ PISA_LIST_NONE,
+ PISA_LIST_FULL,
+ PISA_LIST_IDLE_CLEAN, /* head of @idle */
+ PISA_LIST_IDLE_DIRTY, /* tail of @idle */
+ PISA_LIST_PARTIAL_CLEAN_DENSE, /* head of @partial */
+ PISA_LIST_PARTIAL_CLEAN_SPARSE, /* tail of @partial */
+ PISA_LIST_PARTIAL_DIRTY_DENSE, /* head of @partial_dirty */
+ PISA_LIST_PARTIAL_DIRTY_SPARSE, /* tail of @partial_dirty */
+};
+
/**
* struct pisa_meta - Metadata for a single PISA page in @pisa_meta_array
* @flags: PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED
@@ -187,7 +199,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
struct pisa_alloc *pa[2][PISA_ORDER + 1]; /* [0]=RW, [1]=RO by order */
- struct pisa_alloc *user_pa; /* user page migration pa */
+ struct pisa_alloc *user_pa[3]; /* user/page-cache allocators by dir */
spinlock_t lock; /* protects @pages and @nr_pages */
struct list_head pages; /* reservoir */
@@ -243,6 +255,8 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
bool pisa_want_decrypt(void);
+bool pisa_place_meta(struct pisa_meta *meta);
+void pisa_schedule_reclaim(void);
int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
int pisa_page_unprepare(struct pisa_meta *meta);
@@ -266,11 +280,16 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
return page;
}
+#define PISA_OLD_MS_MAX 100000U
+extern unsigned int pisa_wasted_pct;
+extern unsigned int pisa_old_ms;
+extern unsigned int pisa_scrub_migrate;
unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
unsigned int *nr, struct list_head *release_list,
unsigned long nr_to_scan);
extern unsigned int pisa_scrub_max;
bool pisa_scrub_node(int nid);
+bool pisa_user_scrub_node(int nid, unsigned long budget, u64 deadline);
/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
extern unsigned int pisa_reservoir_max;
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
index a16909d429aca..7dca3bf066f0d 100644
--- a/drivers/iommu/pisa-user.c
+++ b/drivers/iommu/pisa-user.c
@@ -4,15 +4,23 @@
* per-node PISA allocators so zero-copy TX and block I/O use 2MB IOMMU mappings.
* Disabled on CoCo guests (user PTEs map pages encrypted).
*/
+#include <linux/blkdev.h>
#include <linux/cc_platform.h>
+#include <linux/device.h>
#include <linux/pisa.h>
+#include <linux/fs.h>
#include <linux/gfp.h>
+#include <linux/iommu.h>
+#include <linux/jiffies.h>
#include <linux/migrate.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
#include <linux/mutex.h>
#include <linux/nodemask.h>
+#include <linux/pagemap.h>
#include <linux/pagewalk.h>
+#include <linux/folio_batch.h>
+#include <linux/sched/clock.h>
#include <linux/swap.h>
#include "pisa-priv.h"
@@ -55,6 +63,27 @@ static const struct kernel_param_ops pisa_user_max_ops = {
};
module_param_cb(user_max_4K_pages, &pisa_user_max_ops, &pisa_user_max_4k_pages, 0644);
+/*
+ * Scrub-time migration of old sparse user_pa PISA pages when waste exceeds
+ * pisa.wasted_pct (0 = off, 1 = compact in allocator, 2 = demote to the buddy allocator).
+ */
+unsigned int pisa_scrub_migrate __read_mostly;
+
+static int pisa_scrub_migrate_set(const char *val, const struct kernel_param *kp)
+{
+ int ret = param_set_uint(val, kp);
+
+ if (!ret && pisa_scrub_migrate && static_branch_unlikely(&pisa_key))
+ pisa_schedule_scrub_all();
+ return ret;
+}
+
+static const struct kernel_param_ops pisa_scrub_migrate_ops = {
+ .set = pisa_scrub_migrate_set,
+ .get = param_get_uint,
+};
+module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 0644);
+
/* True once per-node user allocators have been allocated, serialized by pisa_user_lock. */
static bool pisa_user_ready __read_mostly;
static DEFINE_MUTEX(pisa_user_lock);
@@ -62,6 +91,7 @@ static DEFINE_MUTEX(pisa_user_lock);
/* Lazily create per-node user allocators on first use (process context). */
static bool pisa_user_init(void)
{
+ static const char * const dir_names[3] = { "bi", "tx", "rx" };
int nid;
if (!READ_ONCE(pisa_user_max_4k_pages) || cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
@@ -73,10 +103,20 @@ static bool pisa_user_init(void)
return false;
mutex_lock(&pisa_user_lock);
for_each_node_state(nid, N_MEMORY) {
- if (!pisa_nodes[nid].user_pa)
- pisa_nodes[nid].user_pa = pisa_create(nid, 0, DMA_TO_DEVICE);
- if (!pisa_nodes[nid].user_pa)
- goto out; /* retried on the next call */
+ unsigned int dir;
+
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+
+ if (!pa) {
+ pa = pisa_create(nid, 0, dir);
+ if (!pa)
+ goto out; /* retried on the next call */
+ pisa_set_name(pa, "user_%s_n%d", dir_names[dir], nid);
+ pisa_enable_pcp(pa);
+ pisa_nodes[nid].user_pa[dir] = pa;
+ }
+ }
}
/* Publish pisa_nodes[].user_pa to lockless readers. */
smp_store_release(&pisa_user_ready, true);
@@ -102,9 +142,13 @@ static unsigned long pisa_user_in_use(void)
int nid;
for_each_node_state(nid, N_MEMORY) {
- const struct pisa_alloc *pa = pisa_nodes[nid].user_pa;
+ unsigned int dir;
- used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ const struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+
+ used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+ }
}
return used;
}
@@ -115,14 +159,43 @@ static long pisa_user_budget(void)
return (long)(READ_ONCE(pisa_user_max_4k_pages) - pisa_user_in_use());
}
-/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
-static bool pisa_user_want(struct folio *folio, bool to_pisa)
+/*
+ * Return true if @folio should be migrated into (@to_pisa) or out of PISA
+ * memory, promoting existing PISA folios to DMA_BIDIRECTIONAL when needed.
+ */
+static bool pisa_user_want_dir(struct folio *folio, bool to_pisa,
+ enum dma_data_direction dir, bool allow_file)
{
- if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
- folio_test_ksm(folio) || folio_test_large(folio) ||
- folio_maybe_mapped_shared(folio))
+ unsigned long pfn = folio_pfn(folio);
+
+ if (is_zero_folio(folio) || folio_is_zone_device(folio) ||
+ folio_test_ksm(folio) || folio_test_large(folio))
return false;
- return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+ if (folio_test_anon(folio)) {
+ if (folio_maybe_mapped_shared(folio))
+ return false;
+ } else if (!allow_file || !folio_mapping(folio) ||
+ folio_test_locked(folio) || folio_test_writeback(folio)) {
+ return false;
+ }
+
+ if (!to_pisa)
+ return is_pisa_pfn(pfn);
+ if (!is_pisa_pfn(pfn))
+ return true;
+
+ if (dir == DMA_TO_DEVICE || dir == DMA_FROM_DEVICE) {
+ int need = (dir == DMA_FROM_DEVICE) ? IOMMU_WRITE : IOMMU_READ;
+ const struct pisa_meta *meta;
+ bool promote;
+
+ rcu_read_lock();
+ meta = is_pisa_pfn(pfn) ? pisa_meta_of_pfn(pfn) : NULL;
+ promote = meta && (meta->prot & need) != need;
+ rcu_read_unlock();
+ return promote;
+ }
+ return false;
}
/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
@@ -132,20 +205,26 @@ static bool pisa_user_busy(struct folio *folio, int extra)
}
struct pisa_user_mtc {
- gfp_t gfp;
- long budget;
+ gfp_t gfp;
+ long budget;
+ enum dma_data_direction dir;
+ bool same_alloc;
};
/* new_folio_t for migration into PISA memory. */
static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
{
struct pisa_user_mtc *mtc = (void *)private;
+ bool already_pisa = is_pisa_pfn(folio_pfn(src));
+ enum dma_data_direction dir = mtc->dir;
int nid = folio_nid(src);
struct page *page;
- if (mtc->budget <= 0)
+ if (!already_pisa && mtc->budget <= 0)
return NULL;
- page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+ if (already_pisa && !mtc->same_alloc)
+ dir = DMA_BIDIRECTIONAL;
+ page = pisa_alloc(pisa_nodes[nid].user_pa[dir], mtc->gfp);
if (!page)
return NULL;
if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
@@ -153,7 +232,8 @@ static struct folio *pisa_user_new_folio(struct folio *src, unsigned long privat
mtc->budget = 0;
return NULL;
}
- mtc->budget--;
+ if (!already_pisa)
+ mtc->budget--;
return page_folio(page);
}
@@ -164,78 +244,319 @@ static struct folio *pisa_user_old_folio(struct folio *src, unsigned long privat
}
/* Migrate the isolated folios on @list; failures are put back. */
-static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
+ enum dma_data_direction dir,
+ bool same_alloc, gfp_t gfp)
{
- struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+ struct pisa_user_mtc mtc = {
+ .gfp = gfp,
+ .budget = LONG_MAX,
+ .dir = dir,
+ .same_alloc = same_alloc,
+ };
enum migrate_mode mode = MIGRATE_SYNC;
+ unsigned int done = 0;
if (list_empty(list))
- return;
+ return 0;
if (to_pisa) {
mtc.budget = pisa_user_budget();
if (!gfpflags_allow_blocking(gfp))
mode = MIGRATE_ASYNC;
}
if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
- NULL, (unsigned long)&mtc, mode, MR_SYSCALL, NULL))
+ NULL, (unsigned long)&mtc, mode, MR_SYSCALL, &done))
putback_movable_pages(list);
+ return done;
}
/**
- * pisa_upgrade_user_pages - Try to move user pages about to be sent into PISA
- * @pages: Pages from iov_iter_get_pages2(), one reference each
+ * pisa_upgrade_user_pages - Try to move user pages into PISA for @dir
+ * @pages: Pages from iov_iter_get_pages2() or iov_iter_extract_pages()
* @nr: Number of entries in @pages
+ * @dir: I/O direction (DMA_TO_DEVICE or DMA_FROM_DEVICE)
+ * @pinned: true if @pages are FOLL_PIN-pinned, false if refcounted (+1)
*
- * Best effort, process context. Pages that cannot be moved (e.g. still
- * referenced by an in-flight skb, or in a large folio) are skipped; for TCP
- * with pisa_tx_bufs enabled the caller copies them instead.
- *
- * Return: false if nothing was attempted and the references are untouched;
- * true if all references were dropped and the caller must look the pages up
- * again.
+ * Return: true if references/pins were dropped and the caller must re-lookup.
*/
-bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir, bool pinned)
{
+ int extra = pinned ? GUP_PIN_COUNTING_BIAS : 1;
bool drained = false;
LIST_HEAD(list);
- unsigned int i, want = 0;
+ unsigned int i, want = 0, need_budget = 0;
if (!pisa_user_init())
return false;
- /* Common case: all pages already in PISA, no budget check needed. */
- for (i = 0; i < nr; i++)
- want += pisa_user_want(page_folio(pages[i]), true);
+ /* Common case: all pages already in PISA with matching prot. */
+ for (i = 0; i < nr; i++) {
+ struct folio *folio = page_folio(pages[i]);
+
+ if (pisa_user_want_dir(folio, true, dir, false)) {
+ want++;
+ need_budget += !is_pisa_pfn(folio_pfn(folio));
+ }
+ }
if (!want)
return false;
- if (pisa_user_budget() <= 0)
+ if (need_budget && pisa_user_budget() <= 0)
return false;
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
- if (!pisa_user_want(folio, true))
+ if (!pisa_user_want_dir(folio, true, dir, false))
continue;
/* A freshly faulted page may still sit in a per-CPU LRU batch. */
if (!drained && !folio_test_lru(folio)) {
lru_add_drain();
drained = true;
}
- /* The reference from iov_iter_get_pages2() is expected. */
- if (pisa_user_busy(folio, 1))
+ if (pisa_user_busy(folio, extra))
continue;
isolate_folio_to_list(folio, &list);
}
if (list_empty(&list))
return false;
- /* Migration fails on pages with references it does not expect. */
- for (i = 0; i < nr; i++)
- put_page(pages[i]);
- pisa_user_migrate(&list, true,
- GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ /* Migration fails on pages with references or pins it does not expect. */
+ if (pinned) {
+ unpin_user_pages(pages, nr);
+ } else {
+ for (i = 0; i < nr; i++)
+ put_page(pages[i]);
+ }
+ pisa_user_migrate_dir(&list, true, dir, false,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+ return true;
+}
+
+/**
+ * __pisa_filemap_alloc - Allocate an order-0 page-cache folio from a PISA user allocator
+ * @mapping: Page-cache address_space
+ * @gfp: GFP flags
+ * @is_write: True when allocating on the buffered write path
+ *
+ * Return: Newly allocated order-0 folio, or NULL on failure.
+ */
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+ gfp_t gfp, bool is_write)
+{
+ enum dma_data_direction dir = is_write ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+ struct page *page;
+ int nid;
+
+ if (!__pisa_mapping_enabled(mapping, dir))
+ return NULL;
+ if (is_write && __pisa_mapping_enabled(mapping, DMA_FROM_DEVICE))
+ dir = DMA_BIDIRECTIONAL;
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!smp_load_acquire(&pisa_user_ready) &&
+ ((gfp & GFP_KERNEL) != GFP_KERNEL || !pisa_user_init()))
+ return NULL;
+ if (pisa_user_budget() <= 0)
+ return NULL;
+ nid = pisa_sanitize_nid(numa_node_id());
+ page = pisa_alloc(pisa_nodes[nid].user_pa[dir], gfp);
+ if (!page)
+ return NULL;
+ if (pisa_meta_of_pfn(page_to_pfn(page))->flags & PISA_DECRYPTED) {
+ put_page(page);
+ return NULL;
+ }
+ return page_folio(page);
+}
+
+/**
+ * pisa_folio_upgrade - Migrate an unlocked order-0 folio to a PISA allocator page
+ * @folio: Folio to migrate (caller holds 1 ref, dropped only when returning true)
+ * @dir: Required DMA direction (DMA_TO_DEVICE or DMA_FROM_DEVICE)
+ * @fbatch: Optional folio_batch holding @folio's reference to release before migration
+ *
+ * Return: true if @folio was isolated and caller's reference was dropped
+ * (caller must re-lookup in mapping); false if @folio is untouched.
+ */
+bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
+ struct folio_batch *fbatch)
+{
+ LIST_HEAD(list);
+
+ if (!pisa_user_init() ||
+ !pisa_user_want_dir(folio, true, dir, true) ||
+ (!is_pisa_pfn(folio_pfn(folio)) && pisa_user_budget() <= 0))
+ return false;
+ if (!folio_test_lru(folio))
+ lru_add_drain();
+ if (pisa_user_busy(folio, 1) || !isolate_folio_to_list(folio, &list))
+ return false;
+ /* Drop caller's reference so migrate_pages() sees only expected refs. */
+ if (fbatch)
+ folio_batch_release(fbatch);
+ else
+ folio_put(folio);
+ pisa_user_migrate_dir(&list, true, dir, false,
+ GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
return true;
}
+bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir)
+{
+ struct device *dev;
+
+ if (!bdev || !bdev->bd_disk)
+ return false;
+ for (dev = disk_to_dev(bdev->bd_disk); dev; dev = dev->parent) {
+ if (dir == DMA_FROM_DEVICE ?
+ READ_ONCE(dev->pisa_rx_bufs) :
+ READ_ONCE(dev->pisa_tx_bufs))
+ return true;
+ }
+ return false;
+}
+
+bool __pisa_mapping_enabled(const struct address_space *mapping, enum dma_data_direction dir)
+{
+ struct super_block *sb;
+
+ if (!mapping || !mapping->host || !mapping->host->i_sb)
+ return false;
+ sb = mapping->host->i_sb;
+ return __pisa_bdev_enabled(sb_is_blkdev_sb(sb) ?
+ I_BDEV(mapping->host) : sb->s_bdev, dir);
+}
+
#define PISA_USER_BATCH 512
+static unsigned long pisa_user_scrub_alloc(struct pisa_alloc *pa,
+ enum dma_data_direction dir,
+ unsigned int mode,
+ unsigned long budget)
+{
+ unsigned int wasted_pct = READ_ONCE(pisa_wasted_pct);
+ unsigned int old_ms = READ_ONCE(pisa_old_ms);
+ unsigned long bpp = PISA_BLOCKS(0), in_use, free_blocks;
+ unsigned long max_age, base_pfn = 0, flags;
+ DECLARE_BITMAP(inuse_map, PISA_MAX_BLOCKS);
+ struct list_head *lists[] = { &pa->partial_dirty, &pa->partial };
+ struct pisa_meta *meta, *target = NULL;
+ unsigned int i, idx, count = 0;
+ bool should_release = false;
+ unsigned long migrated = 0;
+ LIST_HEAD(list);
+ long held;
+
+ if (!wasted_pct || !old_ms)
+ return 0;
+
+ pisa_pcp_drain(pa);
+
+ held = pisa_alloc_held_pages(pa);
+ in_use = READ_ONCE(pa->blocks_in_use);
+ free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
+ (unsigned long)held * bpp - in_use : 0;
+ if (held <= 0 || free_blocks * 100 <= (unsigned long)held * bpp * wasted_pct)
+ return 0;
+
+ max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+ spin_lock_irqsave(&pa->lock, flags);
+ for (i = 0; i < ARRAY_SIZE(lists) && !target; i++) {
+ list_for_each_entry_reverse(meta, lists[i], list) {
+ if (mode == 1 && &meta->list == lists[i]->next)
+ break;
+ if (time_after(jiffies, meta->alloc_jiffies + max_age)) {
+ target = meta;
+ break;
+ }
+ }
+ }
+ if (target) {
+ base_pfn = pisa_meta_to_pfn(target);
+ bitmap_or(inuse_map, target->clean_bitmap, target->dirty_bitmap, bpp);
+ bitmap_complement(inuse_map, inuse_map, bpp);
+ WRITE_ONCE(target->on_list, PISA_LIST_FULL);
+ list_move(&target->list, &pa->full);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ if (!target)
+ return 0;
+
+ for_each_set_bit(idx, inuse_map, bpp) {
+ unsigned long pfn = base_pfn + idx;
+ struct folio *folio;
+
+ rcu_read_lock();
+ if (!is_pisa_pfn(pfn)) {
+ rcu_read_unlock();
+ continue;
+ }
+ folio = page_folio(pfn_to_page(pfn));
+ if (!folio_try_get(folio)) {
+ rcu_read_unlock();
+ continue;
+ }
+ if (!is_pisa_pfn(pfn) || folio != page_folio(pfn_to_page(pfn))) {
+ rcu_read_unlock();
+ folio_put(folio);
+ continue;
+ }
+ rcu_read_unlock();
+
+ if (!pisa_user_want_dir(folio, false, dir, true) ||
+ pisa_user_busy(folio, 1) ||
+ !isolate_folio_to_list(folio, &list)) {
+ folio_put(folio);
+ continue;
+ }
+ folio_put(folio);
+ if (++count >= min_t(unsigned long, budget, PISA_USER_BATCH))
+ break;
+ }
+
+ if (!list_empty(&list)) {
+ migrated = pisa_user_migrate_dir(&list, mode == 1, dir, true,
+ GFP_KERNEL | __GFP_NOWARN);
+ pisa_pcp_drain(pa);
+ }
+
+ spin_lock_irqsave(&pa->lock, flags);
+ if (READ_ONCE(target->pa) == pa) {
+ WRITE_ONCE(target->alloc_jiffies, jiffies);
+ should_release = pisa_place_meta(target);
+ }
+ spin_unlock_irqrestore(&pa->lock, flags);
+ if (should_release)
+ pisa_schedule_reclaim();
+
+ return migrated;
+}
+
+bool pisa_user_scrub_node(int nid, unsigned long budget, u64 deadline)
+{
+ unsigned int mode = READ_ONCE(pisa_scrub_migrate);
+ unsigned long done = 0;
+ unsigned int dir;
+
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!mode || mode > 2 || !smp_load_acquire(&pisa_user_ready))
+ return false;
+
+ for (dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+ struct pisa_alloc *pa = pisa_nodes[nid].user_pa[dir];
+ unsigned long n;
+
+ if (!pa)
+ continue;
+ while ((n = pisa_user_scrub_alloc(pa, dir, mode, budget - done))) {
+ done += n;
+ if (done >= budget || local_clock() >= deadline)
+ return true;
+ cond_resched();
+ }
+ }
+ return false;
+}
+
/**
* pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
* @vma: VMA, with the mmap lock held for reading
@@ -262,17 +583,17 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (pisa_user_want(folio, to_pisa) && !pisa_user_busy(folio, 0) &&
- isolate_folio_to_list(folio, &list))
+ if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false) &&
+ !pisa_user_busy(folio, 0) && isolate_folio_to_list(folio, &list))
n++;
folio_walk_end(&fw, vma);
}
if (n == PISA_USER_BATCH) {
- pisa_user_migrate(&list, to_pisa, gfp);
+ pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
n = 0;
}
cond_resched();
}
- pisa_user_migrate(&list, to_pisa, gfp);
+ pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
return 0;
}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index d11fd3d663dc1..1d9b07370cc75 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -10,9 +10,13 @@
#include <linux/device.h>
#include <linux/mm.h>
#include <linux/numa.h>
+#include <linux/folio_batch.h>
#include <linux/rcupdate.h>
+struct address_space;
+struct block_device;
struct device;
+struct folio_batch;
struct iommu_domain;
#ifdef CONFIG_PISA
@@ -134,7 +138,15 @@ static inline void pisa_task_frag_release(struct task_struct *tsk)
/* User pages in PISA blocks, see drivers/iommu/pisa-user.c */
DECLARE_STATIC_KEY_FALSE(pisa_user_key);
-bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr);
+bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir, bool pinned);
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+ gfp_t gfp, bool is_write);
+bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
+ struct folio_batch *fbatch);
+bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir);
+bool __pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir);
int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa);
@@ -144,6 +156,49 @@ static inline bool pisa_user_enabled(void)
return static_branch_unlikely(&pisa_user_key);
}
+static inline bool is_bdev_pisa_enabled(struct block_device *bdev,
+ enum dma_data_direction dir)
+{
+ return pisa_user_enabled() && __pisa_bdev_enabled(bdev, dir);
+}
+
+static inline bool pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir)
+{
+ return pisa_user_enabled() && __pisa_mapping_enabled(mapping, dir);
+}
+
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+ struct folio *folio, bool write_wait)
+{
+ return write_wait && pisa_mapping_enabled(mapping, DMA_TO_DEVICE) &&
+ pisa_folio_upgrade(folio, DMA_TO_DEVICE, NULL);
+}
+
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+ struct folio *folio, pgoff_t *index,
+ struct folio_batch *fbatch)
+{
+ pgoff_t idx;
+
+ if (!pisa_mapping_enabled(mapping, DMA_TO_DEVICE))
+ return false;
+ idx = folio->index;
+ if (!pisa_folio_upgrade(folio, DMA_TO_DEVICE, fbatch))
+ return false;
+ *index = idx;
+ return true;
+}
+
+static inline struct folio *
+pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
+ unsigned int order, bool is_write)
+{
+ if (order || !pisa_user_enabled())
+ return NULL;
+ return __pisa_filemap_alloc(mapping, gfp, is_write);
+}
+
#else /* !CONFIG_PISA */
static inline int pisa_meta_init(void)
@@ -230,11 +285,52 @@ static inline bool pisa_user_enabled(void)
return false;
}
-static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr)
+static inline bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
+ enum dma_data_direction dir,
+ bool pinned)
{
return false;
}
+static inline bool pisa_folio_upgrade(struct folio *folio,
+ enum dma_data_direction dir,
+ struct folio_batch *fbatch)
+{
+ return false;
+}
+
+static inline bool is_bdev_pisa_enabled(struct block_device *bdev,
+ enum dma_data_direction dir)
+{
+ return false;
+}
+
+static inline bool pisa_mapping_enabled(const struct address_space *mapping,
+ enum dma_data_direction dir)
+{
+ return false;
+}
+
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+ struct folio *folio, bool write_wait)
+{
+ return false;
+}
+
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+ struct folio *folio, pgoff_t *index,
+ struct folio_batch *fbatch)
+{
+ return false;
+}
+
+static inline struct folio *
+pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
+ unsigned int order, bool is_write)
+{
+ return NULL;
+}
+
static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa)
{
diff --git a/mm/filemap.c b/mm/filemap.c
index b04be64a5069f..3860b0c6db4ec 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -15,6 +15,7 @@
#include <linux/dax.h>
#include <linux/fs.h>
#include <linux/sched/signal.h>
+#include <linux/pisa.h>
#include <linux/uaccess.h>
#include <linux/capability.h>
#include <linux/kernel_stat.h>
@@ -1976,6 +1977,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
pgoff_t index, fgf_t fgp_flags, gfp_t gfp, struct mempolicy *policy)
{
+ bool tried_pisa = false;
struct folio *folio;
repeat:
@@ -1985,6 +1987,13 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
if (!folio)
goto no_page;
+ if (!tried_pisa) {
+ tried_pisa = true;
+ if (pisa_filemap_upgrade(mapping, folio,
+ (fgp_flags & (FGP_WRITE | FGP_NOWAIT)) == FGP_WRITE))
+ goto repeat;
+ }
+
if (fgp_flags & FGP_LOCK) {
if (fgp_flags & FGP_NOWAIT) {
if (!folio_trylock(folio)) {
@@ -2044,7 +2053,11 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
err = -ENOMEM;
if (order > min_order)
alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
- folio = filemap_alloc_folio(alloc_gfp, order, policy);
+ folio = pisa_filemap_alloc(mapping, alloc_gfp, order,
+ fgp_flags & FGP_WRITE);
+ if (!folio)
+ folio = filemap_alloc_folio(alloc_gfp, order,
+ policy);
if (!folio)
continue;
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 0694ec4098f6d..39cc28f9762c6 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -37,6 +37,7 @@
#include <linux/sched/rt.h>
#include <linux/sched/signal.h>
#include <linux/mm_inline.h>
+#include <linux/pisa.h>
#include <linux/shmem_fs.h>
#include <trace/events/writeback.h>
@@ -2409,6 +2410,7 @@ static pgoff_t wbc_end(struct writeback_control *wbc)
static struct folio *writeback_get_folio(struct address_space *mapping,
struct writeback_control *wbc)
{
+ pgoff_t upgraded = ULONG_MAX;
struct folio *folio;
retry:
@@ -2423,6 +2425,12 @@ static struct folio *writeback_get_folio(struct address_space *mapping,
return NULL;
}
+ if (folio->index != upgraded &&
+ pisa_writeback_upgrade(mapping, folio, &wbc->index, &wbc->fbatch)) {
+ upgraded = wbc->index;
+ goto retry;
+ }
+
folio_lock(folio);
if (unlikely(!folio_prepare_writeback(mapping, wbc, folio))) {
folio_unlock(folio);
diff --git a/mm/readahead.c b/mm/readahead.c
index 6e5563290287e..f5cfc69dc9909 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -114,6 +114,7 @@
*/
#include <linux/blk_plug.h>
+#include <linux/pisa.h>
#include <linux/kernel.h>
#include <linux/dax.h>
#include <linux/gfp.h>
@@ -197,7 +198,9 @@ static struct folio *ractl_alloc_folio(struct readahead_control *ractl,
{
struct folio *folio;
- folio = filemap_alloc_folio(gfp_mask, order, NULL);
+ folio = pisa_filemap_alloc(ractl->mapping, gfp_mask, order, false);
+ if (!folio)
+ folio = filemap_alloc_folio(gfp_mask, order, NULL);
if (folio && ractl->dropbehind)
__folio_set_dropbehind(folio);
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 40f20d0faa859..8ba9c39cb05ec 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -651,7 +651,8 @@ static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, s
return false;
/* Move user pages into PISA memory, then look them up again. */
- if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE))) {
+ if (pisa_upgrade_user_pages(pages, DIV_ROUND_UP(*start + *copied, PAGE_SIZE),
+ DMA_TO_DEVICE, false)) {
iov_iter_revert(from, *copied);
*copied = iov_iter_get_pages2(from, pages, *length, max_pages, start);
if (*copied < 0) {
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (23 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
Document the PISA IOMMU 2MB-hugepage-backed DMA memory
subsystem in Documentation/core-api/pisa.rst.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
.../admin-guide/kernel-parameters.txt | 6 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/pisa.rst | 426 ++++++++++++++++++
3 files changed, 433 insertions(+)
create mode 100644 Documentation/core-api/pisa.rst
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0b3c140babd94..cb978469a7df3 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5345,6 +5345,12 @@ Kernel parameters
pirq= [SMP,APIC] Manual mp-table setup
See Documentation/arch/x86/i386/IO-APIC.rst.
+ pisa.*= [IOMMU] Pre-mapped IOVA Superpage Allocator (PISA)
+ tuning parameters (max_bits, max_pages, slots, max_idle,
+ idle_ms, reservoir_max, scrub_max, compact, wasted_pct,
+ old_ms, pcp_batch, user_max_4K_pages, scrub_migrate).
+ See Documentation/core-api/pisa.rst.
+
plip= [PPT,NET] Parallel port network link
Format: { parport<nr> | timid | 0 }
See also Documentation/admin-guide/parport.rst.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..4955f477ed797 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
dma-attributes
dma-isa-lpc
swiotlb
+ pisa
mm-api
cgroup
genalloc
diff --git a/Documentation/core-api/pisa.rst b/Documentation/core-api/pisa.rst
new file mode 100644
index 0000000000000..6ed4bd9951f8a
--- /dev/null
+++ b/Documentation/core-api/pisa.rst
@@ -0,0 +1,426 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=============================================
+PISA: Pre-mapped IOVA Superpage Allocator
+=============================================
+
+Overview
+========
+
+PISA is a memory allocator that provides DMA-able memory backed
+by large physically contiguous pages (typically 2MB), and mapped by
+IOMMU leaves of the same size.
+Devices can opt-in to use PISA memory for descriptor rings,
+header buffers, tx and rx data buffers, and zerocopy I/O.
+
+PISA has the following features:
+
+- greatly reduced IOTLB pressure by using fewer and much larger
+ IOMMU entries
+- greatly reduced DMA map/unmap costs. Mapping only happens when the first
+ block from a 2MB page is used. Unmapping is a no-op for the caller,
+ and happens in a separate worker the 2MB page is fully unused
+- much better security than DMA-FQ, because buffers are not returned
+ to the buddy allocator (where they could be used for code or sensitive
+ data) until all IOMMU and IOTLB mappings have been flushed.
+ Note that the mappings are kept active during the lifetime of buffers,
+ but this is the same thing that happens e.g. with pagepool, even in strict
+ IOMMU mode (DMA).
+- can remove the cost of copying into bounce buffers e.g. for overcommitted
+ or Confidential Computing VMs.
+
+For devices that do not opt-in to PISA, the IOMMU keeps using the
+native IOMMU mode (DMA or DMA-FQ).
+
+Note that PISA is fully integrated with relevant kernel APIs so
+devices can use it without device-specific code changes.
+
+What PISA is not
+================
+For avoidance of doubt: PISA is essentially a specialized memory allocator,
+modeled after alloc_pages() and related APIs. The main difference is that
+it enforces some properties on the memory it provides so that other operations
+(IOMMU mapping and unmapping, high rate allocations, identification...)
+can be implemented efficiently and with the desired security properties.
+
+It is not a specialized cache or pagepool or slab allocator:
+we have already plenty of them, built on top of the buddy allocator.
+PISA sits below these existing subsystems and acts as a memory provider.
+
+It is not a custom netdev feature. While network devices are the primary
+user, disks and other compute accelerators can use it, again, as a replacement
+for the alloc_pages() allocator.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+ (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+ access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+ exceeds the IOTLB size, backpressuring the bus via flow control.
+ This is one of the biggest bottlenecks for high speed devices:
+ typical network devices have very little IOVA locality, and that makes
+ the IOTLB almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA
+ map/unmap costs. This is common practice for command, completion and
+ buffer queues, as well as disk I/O and network receive buffers. Network
+ TX buffers are a noticeable exception. They normally come from
+ ``alloc_pages()`` calls unaware of the leaf device, and require
+ mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+ very expensive operation (up to several microseconds to wait for
+ completion, necessary for security), but the way it is implemented
+ in DMA-FQ compromises on security: IOTLB flush requests are run
+ periodically in batches, but the underlying memory is recycled without
+ waiting. During that window, the physical memory is still accessible,
+ opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+ blocks, e.g. 2MB large pages instead of the usual 4KB (PTE). This is not
+ currently pursued by the kernel.
+
+PISA addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map,
+ unmap, and free device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB PISA pages and 2MB IOMMU
+ mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused PISA pages
+ to the system
+
+Execution is what makes PISA practical: the implementation intercepts
+a small set of kernel APIs so that device will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since PISA intercepts allocations of device-accessible
+memory, it can transparently configure unencrypted mode, required for
+confidential computing. Under a paravirtualized IOMMU, the long-lived 2MB
+mappings also make host-side pinning of DMA memory a rare event.
+
+Limitations
+===========
+
+PISA is currently limited to systems with 2MB large pages, which
+is the vast majority of existing platforms. This is enforced at compile time.
+
+Decoupling the allocation size from 2MB can be made without much effort,
+so the mechanisms can be extended to systems with 16K/64K page size.
+
+The implementation assumes non-preemptible spinlocks, so PISA is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+PISA can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers follow the direction the allocator was created
+with. Receive pools (``page_pool``) are ``DMA_BIDIRECTIONAL``, so received
+buffers can be retransmitted (XDP_TX, forwarding) without a new mapping.
+Allocators that only hold transmit data, such as socket TX fragments, are
+``DMA_TO_DEVICE``. Allocators created
+``DMA_TO_DEVICE`` or ``DMA_FROM_DEVICE`` have their PISA pages mapped device
+read-only or write-only, and a mapping that requests other access falls back
+to a per-buffer mapping.
+
+IOMMU mappings for queues are also ``DMA_BIDIRECTIONAL``, though stricter modes
+can be implemented trivially.
+
+Implementation details
+======================
+
+Most of the code size and design complexity relates to the handling
+of exceptional events (device teardown, memory hotplug) and to
+the safe release of pages through deferred tasks, RCU and grace
+periods.
+
+The implementation lives in ``drivers/iommu/pisa-*.c`` and
+``include/linux/pisa.h``, and relies on the components described below.
+We use the name "PISA" to refer to the second-level physically contiguous
+pages (typically 2MB) used for I/O memory, but the design is not tightly
+bound to the use of a PISA page.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, socket TX page frags, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+
+and most devices will automatically inherit the benefits.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of PISA pages, based on either their Physical
+ Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to PISA page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **PISA metadata** (``struct pisa_meta``):
+
+ Identification of PISA pages could be done in principle with a flag in
+ ``struct page``, but that would not solve the management of per-page
+ metadata.
+
+ PISA addresses both requirements without depending on ``struct page``
+ using an idea similar to ``pageblock_flags``. Each PISA page used as a
+ buffer requires only 256B of metadata, or 0.01% overhead. For metadata,
+ we reserve a sparse virtual memory region of approximate size
+ ``max_possible_pfn * 256 / PISA_SIZE`` (``pisa.max_bits=N`` on the
+ command line covers ``2^N`` bytes of physical address space instead; RAM
+ above the covered range is not used by PISA), so the metadata can be accessed
+ directly using the PISA page index. A metadata page is backed when a
+ allocator first takes a PISA page in its 32MB chunk, from any context: the
+ page tables are allocated beforehand for present RAM, and by a memory
+ hotplug notifier for memory added later. A compact bitmap (1 bit per
+ 32MB chunk, or 4KB per 1TB of address space) tracks which metadata
+ pages are backed, allowing fast lockless validation and direct array
+ indexing.
+
+ ``struct pisa_meta`` has flags to mark the page status (used as
+ PISA buffer, or decrypted for confidential computing), and tracks
+ which subpages are free, which domains have an active IOMMU mapping for
+ the page, and holds list/RCU linkage to manage its lifetime.
+
+- **Per-domain reserved PISA IOVA ranges**:
+
+ Another key requirement is to quickly resolve the PA-IOVA mapping for a
+ given domain. Since a PISA page can be mapped in multiple domains (e.g.
+ a network buffer used to route packets between different NICs), it would
+ be too expensive to manage random PA-IOVA mappings created on the fly.
+ Each PISA page gets a dense slot number when it enters PISA, and is
+ mapped at the same offset, ``slot * PISA_SIZE``, in
+ every domain's window. PISA reserves on each domain an IOVA range of
+ ``pisa.slots * 2MB`` (by default 2 * ``max_pages`` slots, i.e. a
+ quarter of RAM): its size does not depend on the physical memory layout,
+ and IOVAs do not expose physical addresses. The IOVA allocation happens
+ the first time a domain is used, hence a IOVA within the reserved range
+ will uniquely identify a PISA page. Slots are handed out lowest first
+ and reused only after the page has been unmapped from every domain.
+
+- **alloc_pages() compatible pisa_alloc allocator for I/O buffers**:
+
+ PISA implements a pisa_alloc object for fixed-order page allocations
+ that is a natural fit for the allocation of NIC receive buffers and tx
+ socket buffers. The API is similar to ``alloc_pages()`` (in fact, it is an
+ almost direct replacement) and pisa_allocs are instantiated as follows:
+
+ - per node, order-0 ``DMA_BIDIRECTIONAL`` (``__GFP_PISA_RW``), with per-CPU
+ block caches, to provide RX buffer pages
+ - per node, ``DMA_TO_DEVICE`` (``__GFP_PISA_RO``), with per-CPU block caches
+ to provide order-3 (``SKB_FRAG_PAGE_ORDER``) pages to refill tx socket buffers; on a miss,
+ an ordinary order-0 page is used. Order-2 ``DMA_TO_DEVICE`` (without per-CPU
+ caches) is also provided for small skb heads on systems with few CPUs.
+
+ - per node, order-0 ``DMA_BIDIRECTIONAL`` (``pisa_alloc_coherent()``),
+ carving naturally aligned contiguous 4KB page ranges (up to ``PISA_SIZE``)
+ to back coherent ``dma_alloc_attrs()`` queues and header bounce buffers
+ when ``dev->pisa_coherent`` is enabled.
+
+
+- **Fine grained control**
+
+ Both for experimentation and production use, it is useful to have
+ some form of control on which devices want to use PISA and for
+ what. PISA exposes the following controls. Keep in mind that full
+ performance can only be achieved if all regions are mapped using PISA.
+
+ - ``/sys/devices/*/*/pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads}``
+ per-device entries that control whether a device uses PISA for coherent
+ allocations, RX buffers, TX socket page fragments, and small TX skb heads
+
+- **Safe release of memory to the system**
+
+ A key feature of PISA is that memory is not released to the system
+ while there are active IOMMU mappings. Release of buffers is managed
+ as follows
+
+ 1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to
+ PISA, IOMMU mappings are still active, and they can be recycled
+
+ 2. when all blocks in a PISA page are unused, the page is moved to the
+ allocator's "idle" list, still with active mappings and available for reuse,
+ unless the allocator already holds max_idle_pages idle pages
+
+ 3. pages above that per-allocator watermark are released by a reclaim work
+ item, and under memory pressure the shrinker also releases idle pages
+ from all allocators: the iommu mappings are removed and a synchronous IOTLB
+ flush is issued. The pages are no longer usable for allocations but not
+ returned to the buddy allocator yet
+
+ 4. once the previous step is complete, the system schedules ``call_rcu()``
+ and waits for an RCU grace period (and re-encrypts the page if it was
+ decrypted) before returning the PISA page to the buddy allocator.
+
+- **Handling domain destruction**
+
+ When a device/domain is destroyed, all I/O issued by the device must have
+ been quiesced and memory released via ``__free_pages()`` or ``dma_free*()``.
+ PISA hooks into the domain destructor (``iommu_put_dma_cookie()``) and
+ forgets the disappearing domain: its bit is cleared in every active
+ PISA page, without touching the domain's page tables, which the owner
+ frees wholesale. The domain index is reused only after an SRCU grace
+ period.
+
+- **Handling loss of the last DMA API user**
+
+ ``dma_unmap_*()`` never removes window PTEs, so when the last driver using
+ a group's default domain through the DMA API unbinds, or the group's DMA
+ ownership is taken (VFIO, iommufd), ``pisa_domain_quiesce()`` unmaps every
+ active PISA page from that domain and flushes the IOTLB. The window
+ and domain index are kept; PTEs are re-installed on demand if a driver binds
+ again. In-flight page_pool pages, which ``page_pool_scrub()`` cannot unmap,
+ are covered the same way.
+
+- **Allocation and compound splitting**:
+
+ An allocator that needs memory calls ``pisa_alloc_split_page()`` to allocate a
+ physically contiguous PISA page on the calling CPU's NUMA node, and
+ ``split_page_compound()`` to split it into independent blocks of
+ ``pa->order``. These blocks can be independently passed through the
+ networking stack.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+PISA integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of PISA memory is opt-in via per-device sysfs
+attributes or module parameters. Mapping, unmapping, and freeing automatically detects
+whether a buffer or IOVA belongs to PISA:
+
+- **Streaming DMA map and unmap** (``dma_map_page_attrs()``,
+ ``dma_map_single_attrs()``, ``dma_map_sg_attrs()`` and
+ ``dma_unmap_page_attrs()``, ``dma_unmap_single_attrs()``,
+ ``dma_unmap_sg_attrs()`` via ``drivers/iommu/dma-iommu.c`` and
+ ``kernel/dma/direct.c``):
+
+ - On map, any buffer belonging to a PISA page
+ (``is_pisa_phys(phys)``) is mapped via the domain's reserved
+ PISA IOVA window with a simple addition, typically reusing
+ existing mappings. Under ``dma-direct``, decrypted PISA pages
+ (``is_pisa_direct(phys, size)``) can be mapped directly without
+ ``swiotlb`` bounce.
+
+ On unmap, IOVAs can be identified as PISA buffers with a simple
+ range check and that results in a no-operation.
+
+- **Coherent DMA allocation and free** (``dma_alloc_attrs()`` /
+ ``dma_alloc_coherent()`` and ``dma_free_attrs()`` / ``dma_free_coherent()``
+ via ``drivers/iommu/dma-iommu.c`` and ``kernel/dma/direct.c``):
+
+ - PISA is enabled per device via ``/sys/devices/.../pisa_coherent``
+ (``dev->pisa_coherent``). When set, sleepable ``GFP_KERNEL`` coherent
+ allocations up to ``PISA_SIZE`` are rounded up to 4KB, carved out of
+ the per-node order-0 ``pisa_alloc`` via ``pisa_alloc_coherent()``, and
+ mapped through the domain's PISA IOVA window (or directly at their
+ physical/decrypted DMA address for ``dma-direct``). ``dma_free_attrs()``
+ releases the order-0 pages via ``pisa_free_coherent()``, which recycles
+ them back to the allocator.
+
+- **Page allocator release** (``__free_pages()`` / ``put_page()`` via
+ ``__free_pages_prepare()`` in ``mm/page_alloc.c``):
+
+ - ``pisa_free_page()`` checks ``is_pisa_page(page)``
+ and recycles PISA subpages back to their owning ``pisa_alloc``
+ instead of releasing them to the buddy allocator while their 2MB IOMMU
+ mappings remain active.
+
+- **IOMMU domain teardown** (``iommu_put_dma_cookie()`` in
+ ``drivers/iommu/dma-iommu.c``):
+
+ - A call to ``pisa_domain_release()`` before freeing a
+ domain's IOVA cookie and page tables clears cached domain state across
+ all allocators. Existing IOMMU mappings and IOTLB flush has already happened.
+
+- **Networking ``page_pool``** (``net/core/page_pool.c``):
+
+ - PISA is enabled per device via ``/sys/devices/.../pisa_rx_bufs``
+ (``dev->pisa_rx_bufs``). ``__page_pool_alloc_netmems_slow()`` adds
+ ``__GFP_PISA_RW`` via ``pisa_rx_gfp()`` when ``PP_FLAG_DMA_MAP`` is set,
+ allocating RX pages from the per-node ``DMA_BIDIRECTIONAL`` PISA
+ allocator and falling back to the buddy allocator on exhaustion.
+
+- **Socket TX page fragments** (``sk_page_frag_dst()`` and
+ ``skb_page_frag_refill()`` in ``net/core/sock.c``):
+
+ - When enabled per device via ``/sys/devices/.../pisa_tx_bufs``,
+ socket TX fragments are allocated from per-node ``pisa_alloc`` instances,
+ with per-CPU block caches in front of them.
+ These allocators are ``DMA_TO_DEVICE``: devices can only read them, and
+ callers that map fragments for device writes get per-buffer mappings.
+ - Only data routed to a device with a DMA parent, outside IPsec, is copied
+ into them (TCP, UDP/raw), through a dedicated
+ per-task frag. Loopback, virtual devices, kTLS, sockmap, KCM and MPTCP
+ use ordinary pages. ESP output uses the allocators through
+ ``skb_page_frag_refill_dev()``, when their egress device can use them.
+
+- **Small skb heads** (``kmalloc_reserve()`` in ``net/core/skbuff.c``):
+
+ - When enabled per device via ``/sys/devices/.../pisa_tx_skb_heads``,
+ small skb heads come from the ``skbuff_small_head_pisa`` slab cache,
+ whose slab pages are blocks of per-node TX-only ``pisa_alloc``
+ instances (created ``DMA_TO_DEVICE``), zeroed when taken from the allocator.
+ Mappings of these heads that request device write access fall back to
+ per-buffer IOMMU mappings.
+
+- **NIC driver RX buffers**
+
+ - Controlled per device by ``/sys/devices/.../pisa_rx_bufs``
+ (``dev->pisa_rx_bufs``) for queue modes managing their own page rings
+ (e.g. ``gve`` GQI-RDA).
+
+- **User pages for MSG_ZEROCOPY** (``drivers/iommu/pisa-user.c``):
+
+ - ``madvise(MADV_PISA)`` migrates the present private anonymous pages
+ of a range into blocks of per-node ``DMA_TO_DEVICE`` allocators, so that
+ zerocopy sends from them use the PISA window instead of per-send IOMMU
+ mappings or hypervisor pinning. ``MADV_NOPISA`` migrates them back to
+ ordinary pages. Both are allowed through ``process_madvise()``.
+ Freeing the pages (``munmap()``, ``MADV_DONTNEED``, exit, reclaim)
+ returns the blocks to their allocators.
+ - ``/sys/module/pisa/parameters/user_max_4K_pages``: Soft cap on user
+ pages held in PISA blocks, in 4 KB pages (``max_pages`` counts PISA
+ pages). When non-zero, eligible pages are also migrated on
+ ``MSG_ZEROCOPY`` sends (best effort, non-blocking, skipping pages still
+ referenced by in-flight skbs); 0 (default) disables both. With
+ ``pisa_tx_bufs`` also enabled, pages that are not in PISA
+ after that (cap reached, busy, large, shared or file-backed) are copied
+ by TCP into PISA socket frags instead of being sent zerocopy (other
+ senders still send them zerocopy); the copy is not reported to the
+ application.
+ - Not available on CoCo guests: decrypted blocks cannot be mapped to user
+ space, and whole pages would be shared with the host.
+
+- **Global memory limit and observability:**
+
+ - ``/sys/module/pisa/parameters/max_pages``: Global cap on PISA
+ pages held by all allocators and the per-node reservoirs
+ (defaults to 1/8th of physical RAM, at least 16 pages).
+ - ``/sys/module/pisa/parameters/reservoir_max``: Per-NUMA-node reservoir
+ target of pre-allocated PISA pages (defaults to 16 per memory node).
+ - ``/sys/module/pisa/parameters/scrub_max``: Max idle PISA pages
+ scrubbed per background pass (0 disables background scrubbing).
+ - ``/sys/kernel/debug/pisa/allocs``: Debugfs summary of global, per-node,
+ user-page, and per-allocator allocation, mapping, and fallback counters.
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread* [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs
[not found] <20261003212241.3432303-1-lrizzo@google.com>
` (24 preceding siblings ...)
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
@ 2026-11-09 18:25 ` Luigi Rizzo
25 siblings, 0 replies; 26+ messages in thread
From: Luigi Rizzo @ 2026-11-09 18:25 UTC (permalink / raw)
To: Luigi Rizzo, Joerg Roedel, Will Deacon, Robin Murphy,
Christoph Hellwig, Marek Szyprowski, Andrew Morton,
Vlastimil Babka, David Hildenbrand, Jens Axboe, David S . Miller,
Eric Dumazet, Jakub Kicinski, Paolo Abeni
Cc: Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
Jonathan Corbet, Matthew Wilcox, Lorenzo Stoakes,
Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
Alexander Lobakin, iommu, netdev, linux-mm, linux-block,
driver-core, linux-doc, linux-kernel, Luigi Rizzo
PISA allocation and mapping failures silently fall back to standard page
allocation and per-buffer DMA mappings, making performance regressions
hard to diagnose without observability.
Add per-allocator, per-node reservoir, user-page migration, and global
shrinker/reclaim counters, and expose them via
/sys/kernel/debug/pisa/allocs.
Also emit a one-shot dmesg warning on the first 2MB buddy allocation
failure.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
drivers/iommu/pisa-alloc.c | 298 +++++++++++++++++++++++++++++++++----
drivers/iommu/pisa-kunit.c | 7 +-
drivers/iommu/pisa-map.c | 79 +++++++++-
drivers/iommu/pisa-node.c | 43 +++++-
drivers/iommu/pisa-priv.h | 127 +++++++++++++++-
drivers/iommu/pisa-user.c | 77 +++++++++-
include/linux/pisa.h | 5 +
net/core/datagram.c | 2 +
8 files changed, 589 insertions(+), 49 deletions(-)
diff --git a/drivers/iommu/pisa-alloc.c b/drivers/iommu/pisa-alloc.c
index cedb1a62d20e2..91b9f00d21985 100644
--- a/drivers/iommu/pisa-alloc.c
+++ b/drivers/iommu/pisa-alloc.c
@@ -11,6 +11,7 @@
#include <linux/cache.h>
#include <linux/cc_platform.h>
#include <linux/cpuhotplug.h>
+#include <linux/debugfs.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
#include <linux/dma-map-ops.h>
@@ -31,6 +32,7 @@
#include <linux/rcupdate.h>
#include <linux/sched/clock.h>
#include <linux/sched/mm.h>
+#include <linux/seq_file.h>
#include <linux/set_memory.h>
#include <linux/shrinker.h>
#include <linux/skbuff.h>
@@ -85,6 +87,13 @@ module_param_named(old_ms, pisa_old_ms, uint, 0644);
unsigned int pisa_scrub_max __read_mostly;
#define PISA_SCRUB_MAX_LIMIT 65536U
+/* Reclaim statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats pisa_reclaim_stats;
+
+char pisa_pages_2m[16] __ro_after_init;
+char pisa_pages_4k[16] __ro_after_init;
+char pisa_no_pages_2m[16] __ro_after_init;
+
/* All live allocators, protected by pisa_allocs_lock. */
static LIST_HEAD(pisa_allocs);
static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -115,6 +124,7 @@ static const struct kernel_param_ops pisa_scrub_ops = {
.get = param_get_uint,
};
module_param_cb(scrub_max, &pisa_scrub_ops, &pisa_scrub_max, 0644);
+
/* Return true if a new PISA page should be decrypted. */
bool pisa_want_decrypt(void)
{
@@ -179,7 +189,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
unsigned long flags;
spin_lock_irqsave(&meta->pa->lock, flags);
- meta->pa->stats.page_free_cnt++;
+ pisa_stat_inc(meta->pa->stats.page_free_cnt);
spin_unlock_irqrestore(&meta->pa->lock, flags);
WRITE_ONCE(meta->pa, NULL);
@@ -190,6 +200,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
/* Still shared with the host, or possibly still mapped by a device. */
if (meta->flags & (PISA_DECRYPTED | PISA_QUARANTINED))
return;
+
atomic_long_dec(&pisa_nr_pages);
for (i = 0; i < PISA_BLOCKS(order); i++) {
block = page + (i << order);
@@ -238,6 +249,9 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
struct pisa_meta *meta, *tmp;
struct llist_node *node;
struct pisa_alloc *pa;
+ unsigned int n = 0;
+
+ pisa_stat_inc(pisa_reclaim_stats.reclaim_run);
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
@@ -263,8 +277,11 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
read_unlock(&pisa_allocs_lock);
node = llist_del_all(&pisa_free_list);
- llist_for_each_entry_safe(meta, tmp, node, llnode)
+ llist_for_each_entry_safe(meta, tmp, node, llnode) {
pisa_release_page(meta);
+ n++;
+ }
+ pisa_stat_add(pisa_reclaim_stats.reclaim_pages, n);
if (next_delay != MAX_JIFFY_OFFSET)
queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
@@ -469,6 +486,7 @@ static unsigned long pisa_scrub_alloc(struct pisa_alloc *pa)
for (idx = 0; idx < count; idx++)
__set_bit(idxs[idx], meta->clean_bitmap);
meta->nr_clean += count;
+ pisa_stat_add(pa->stats.block_scrub_cnt, count);
pa->scrub_demand -= min(count, pa->scrub_demand);
should_release = pisa_place_meta(meta);
spin_unlock_irqrestore(&pa->lock, flags);
@@ -603,6 +621,7 @@ static void pisa_return_block_locked(struct pisa_alloc *pa, struct page *block,
meta->nr_dirty++;
}
pa->blocks_in_use--;
+ pisa_stat_inc(pa->stats.block_free_cnt);
*release |= pisa_place_meta(meta);
}
@@ -729,7 +748,8 @@ static const struct kernel_param_ops pisa_pcp_batch_ops = {
module_param_cb(pcp_batch, &pisa_pcp_batch_ops, &pisa_pcp_batch, 0644);
/* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
-static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n)
+static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned int n,
+ bool is_free)
{
unsigned int i, batch = READ_ONCE(pisa_pcp_batch);
struct pisa_pcp *pcp;
@@ -738,6 +758,8 @@ static void pisa_pcp_push(struct pisa_alloc *pa, struct page **blocks, unsigned
local_irq_save(flags);
pcp = this_cpu_ptr(pa->pcp);
spin_lock(&pcp->lock);
+ if (is_free)
+ pisa_stat_add(pcp->free_cnt, n);
for (i = 0; i < n; i++) {
if (pcp->count == PISA_PCP_HIGH) {
/* Keep the most recently freed, likely still in cache. */
@@ -799,18 +821,37 @@ void pisa_pcp_drain(struct pisa_alloc *pa)
}
}
-/* Blocks in @pa's caches, for shrinker and debugfs. */
-static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+/* Aggregate statistics across @pa's per-CPU caches, for shrinker and debugfs. */
+static unsigned long pisa_pcp_stats(struct pisa_alloc *pa, u64 *alloc, u64 *miss,
+ u64 *free, u64 *ks, u64 *as)
{
+ u64 a = 0, m = 0, f = 0, s_k = 0, s_a = 0;
unsigned long n = 0;
int cpu;
- if (!pa->pcp)
- return 0;
- for_each_possible_cpu(cpu) {
- /* A racy sum is good enough for reclaim and statistics. */
- n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+ if (pa->pcp) {
+ for_each_possible_cpu(cpu) {
+ const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+
+ /* A racy sum is good enough for reclaim and statistics. */
+ n += READ_ONCE(pcp->count);
+ a += READ_ONCE(pcp->alloc_cnt);
+ m += READ_ONCE(pcp->miss_cnt);
+ f += READ_ONCE(pcp->free_cnt);
+ s_k += READ_ONCE(pcp->alloc_ks);
+ s_a += READ_ONCE(pcp->alloc_as);
+ }
}
+ if (alloc)
+ *alloc = a;
+ if (miss)
+ *miss = m;
+ if (free)
+ *free = f;
+ if (ks)
+ *ks = s_k;
+ if (as)
+ *as = s_a;
return n;
}
@@ -1206,6 +1247,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
}
*countp -= used;
pa->blocks_in_use += used;
+ pisa_stat_add(pa->stats.block_alloc_cnt, used);
return got;
}
@@ -1254,9 +1296,11 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
/* Extract up to @want blocks from @meta under @pa->lock. */
static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
- struct page **out, bool zero, bool frozen)
+ struct page **out, bool can_block, bool zero,
+ bool frozen)
{
unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
+ unsigned long base_pfn = pisa_meta_to_pfn(meta);
struct pisa_alloc *pa = meta->pa;
lockdep_assert_held(&pa->lock);
@@ -1269,12 +1313,23 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
for (i = got; i < got + dirty; i++)
set_page_private(out[i], 1);
+ if (can_block) {
+ pisa_stat_add(pa->stats.alloc_kc, got);
+ pisa_stat_add(pa->stats.alloc_ks, dirty);
+ } else {
+ pisa_stat_add(pa->stats.alloc_ac, got);
+ pisa_stat_add(pa->stats.alloc_as, dirty);
+ }
got += dirty;
pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
} else {
/* First choice are dirty blocks, followed by clean */
got += pisa_scan_bitmap(meta, true, want, out, !frozen);
got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+ if (can_block)
+ pisa_stat_add(pa->stats.alloc_kd, got);
+ else
+ pisa_stat_add(pa->stats.alloc_ad, got);
}
used = avail_before - pisa_meta_avail(meta);
@@ -1284,6 +1339,8 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
meta->nr_dirty = 0;
}
pisa_place_meta(meta);
+ if (page_to_nid(pfn_to_page(base_pfn)) != pa->nid)
+ pisa_stat_add(pa->stats.numa_mismatch_cnt, used);
return got;
}
@@ -1292,6 +1349,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
struct page **pages, bool zero, bool frozen)
{
+ bool can_block = gfpflags_allow_blocking(gfp);
bool should_scrub = false;
unsigned long n, flags, i;
struct pisa_meta *meta;
@@ -1299,7 +1357,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
/* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
spin_lock_irqsave(&pa->lock, flags);
- for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+ for (n = 0; n < nr_pages;
+ n += take_blocks(meta, want, &pages[n], can_block, zero, frozen)) {
want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
meta = pisa_pick_meta(pa, zero);
if (meta)
@@ -1310,15 +1369,31 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
spin_unlock_irqrestore(&pa->lock, flags);
/* With @zero, fresh dirty blocks are zeroed below. */
meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ long err = PTR_ERR(meta);
+
+ if (err == -EAGAIN) {
+ pisa_stat_atomic_inc(pa->stats.fail_backoff);
+ } else if (err == -ENOSPC) {
+ pisa_stat_atomic_inc(pa->stats.fail_budget);
+ } else if (err == -ENOMEM) {
+ pisa_stat_atomic_inc(pa->stats.fail_nomem);
+ pr_warn_once("pisa: order-%u allocation failed for allocator %p; further failures are counted in debugfs only\n",
+ PISA_ORDER, pa);
+ } else if (err == -EBUSY) {
+ pisa_stat_atomic_inc(pa->stats.fail_split);
+ }
+ if (!n)
+ pisa_stat_atomic_inc(pa->stats.block_alloc_fail);
goto out;
+ }
spin_lock_irqsave(&pa->lock, flags);
meta->pa = pa;
meta->prot = pa->prot;
WRITE_ONCE(meta->alloc_jiffies, jiffies);
/* Set PISA_ACTIVE after @meta is fully set up. */
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
- pa->stats.page_alloc_cnt++;
+ pisa_stat_inc(pa->stats.page_alloc_cnt);
}
/* Queue the scrubber when the inline zeroing crosses the trigger. */
should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
@@ -1346,9 +1421,11 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
struct page **page_array, bool frozen)
{
unsigned int pcp_batch = READ_ONCE(pisa_pcp_batch);
- unsigned long got, n, i, j;
+ bool zero = want_init_on_alloc(gfp);
+ unsigned long got, local, n, i, j;
got = pisa_pcp_pop(pa, nr_pages, page_array);
+ local = got;
if (got < nr_pages) {
if (nr_pages - got >= pcp_batch) {
got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
@@ -1360,7 +1437,7 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
for (i = 0; i < n && got < nr_pages; i++)
page_array[got++] = batch[i];
if (i < n)
- pisa_pcp_push(pa, batch + i, n - i);
+ pisa_pcp_push(pa, batch + i, n - i, false);
}
}
@@ -1369,10 +1446,13 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
page_array[i] = NULL;
/* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
- if (pisa_block_poisoned(pa, block))
+ if (pisa_block_poisoned(pa, block)) {
+ if (i < local)
+ local--;
continue;
+ }
pisa_block_prep(block);
- if (want_init_on_alloc(gfp)) {
+ if (zero) {
memset(page_address(block), 0, PAGE_SIZE << pa->order);
if (frozen) /* Order zeroing before caller publishes the page. */
smp_wmb();
@@ -1381,6 +1461,15 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
page_ref_unfreeze(block, 1);
page_array[j++] = block;
}
+ this_cpu_add(pa->pcp->alloc_cnt, j);
+ if (zero) {
+ if (gfpflags_allow_blocking(gfp))
+ this_cpu_add(pa->pcp->alloc_ks, j);
+ else
+ this_cpu_add(pa->pcp->alloc_as, j);
+ }
+ if (j > local)
+ this_cpu_inc(pa->pcp->miss_cnt);
return j;
}
@@ -1400,10 +1489,14 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
{
unsigned long i = 0, n, got;
- if (unlikely(!pa ||
- (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+ if (unlikely(!pa))
goto skip;
+ if (unlikely(gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))) {
+ pisa_stat_atomic_inc(pa->stats.fail_gfp);
+ goto skip;
+ }
+
while (i < nr_pages) {
if (page_array[i]) {
i++;
@@ -1500,6 +1593,9 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
page_ref_unfreeze(page + i, 1);
}
pa->blocks_in_use += nblocks;
+ pisa_stat_add(pa->stats.block_alloc_cnt, nblocks);
+ pisa_stat_add(pa->stats.alloc_kc, nblocks - dirty);
+ pisa_stat_add(pa->stats.alloc_ks, dirty);
pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
pisa_place_meta(meta);
return page;
@@ -1577,7 +1673,7 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
meta->prot = pa->prot;
WRITE_ONCE(meta->alloc_jiffies, jiffies);
WRITE_ONCE(meta->flags, meta->flags | PISA_ACTIVE);
- pa->stats.page_alloc_cnt++;
+ pisa_stat_inc(pa->stats.page_alloc_cnt);
page = pisa_take_contig(meta, nblocks, align_mask);
if (unlikely(!page))
should_release = pisa_place_meta(meta);
@@ -1591,6 +1687,11 @@ void *pisa_alloc_coherent(struct page **pagep, struct device *dev,
pisa_schedule_scrub(pa->nid);
if (!page)
return NULL;
+ /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+ if (smp_load_acquire(&pa->pcp)) {
+ this_cpu_add(pa->pcp->alloc_cnt, nblocks);
+ this_cpu_inc(pa->pcp->miss_cnt);
+ }
*pagep = page;
return memset(page_address(page), 0, size);
}
@@ -1631,9 +1732,10 @@ bool __pisa_free_page(struct page *page)
if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
page_to_nid(page) == numa_mem_id()) {
if (pisa_page_can_recycle(page)) {
- pisa_pcp_push(pa, &page, 1);
+ pisa_pcp_push(pa, &page, 1, true);
return true;
}
+ this_cpu_inc(pa->pcp->free_cnt);
if (this_cpu_read(pa->pcp->count))
pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
}
@@ -1657,7 +1759,7 @@ static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_co
read_lock(&pisa_allocs_lock);
list_for_each_entry(pa, &pisa_allocs, node) {
nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
- nr += pisa_pcp_count(pa) << pa->order;
+ nr += pisa_pcp_stats(pa, NULL, NULL, NULL, NULL, NULL) << pa->order;
}
read_unlock(&pisa_allocs_lock);
@@ -1685,11 +1787,14 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
{
struct pisa_meta *meta, *tmp;
+ unsigned long freed, res_freed;
struct pisa_alloc *pa;
- unsigned long freed;
LIST_HEAD(release_list);
freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+ res_freed = freed;
+ pisa_stat_atomic_inc(pisa_reclaim_stats.shrink_scan);
+ pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_res, res_freed);
if (freed < sc->nr_to_scan) {
read_lock(&pisa_allocs_lock);
@@ -1704,6 +1809,7 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
break;
}
read_unlock(&pisa_allocs_lock);
+ pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_idle, freed - res_freed);
}
/* Unmap outside locks; defer decrypted pages to workqueue outside fs_reclaim. */
@@ -1723,6 +1829,141 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
return freed ?: SHRINK_STOP;
}
+/* See pisa_seq_kv(). */
+void __pisa_seq_kv(struct seq_file *s, const char *title, const struct pisa_kv *st, unsigned int n)
+{
+ seq_printf(s, " %-12s", title);
+ for (; n; n--, st++)
+ seq_printf(s, " %-10s%11lld", st->name, st->val);
+ seq_putc(s, '\n');
+}
+
+/* Output global and active per-allocator statistics in /sys/kernel/debug/pisa/allocs. */
+static int pisa_allocs_show(struct seq_file *s, void *unused)
+{
+ struct pisa_reclaim_stats *rs = &pisa_reclaim_stats;
+ struct pisa_alloc *pa;
+
+ seq_puts(s, "global (allocators with no activity are omitted)\n");
+ pisa_seq_kv(s, pisa_pages_2m, { "in_use", atomic_long_read(&pisa_nr_pages) },
+ { "max", pisa_max_pages },
+ { "slots", pisa_nr_slots });
+ pisa_domains_show(s);
+ pisa_seq_kv(s, "reservoir", { pisa_pages_2m, pisa_reservoir_count() },
+ { "max_node", pisa_reservoir_max });
+ pisa_seq_kv(s, "shrinker",
+ { "scan", pisa_stat_atomic_read(rs->shrink_scan) },
+ { "from_res", pisa_stat_atomic_read(rs->pte_pages_res) },
+ { "from_idle", pisa_stat_atomic_read(rs->pte_pages_idle) });
+ pisa_seq_kv(s, "reclaim",
+ { "runs", pisa_stat_read(rs->reclaim_run) },
+ { pisa_pages_2m, pisa_stat_read(rs->reclaim_pages) },
+ { "unmap_fail", pisa_stat_atomic_read(rs->unmap_fail) });
+ pisa_reservoir_show(s);
+ pisa_user_show(s);
+
+ read_lock(&pisa_allocs_lock);
+ list_for_each_entry(pa, &pisa_allocs, node) {
+ u64 alloc_pages, free_pages, block_alloc, block_free, block_scrub;
+ u64 pages_in_use, blocks_in_use, used_pct = 0, pcp_alloc, pcp_miss;
+ u64 block_fail, fail_gfp, numa_mismatch, pcp_free, cached;
+ u64 alloc_kd, alloc_kc, alloc_ks, alloc_ad, alloc_ac, alloc_as;
+ u64 pcp_ks, pcp_as;
+ struct pisa_alloc_stats *st = &pa->stats;
+ unsigned int num_idle, max_idle;
+ unsigned long flags;
+
+ block_fail = pisa_stat_atomic_read(st->block_alloc_fail);
+ fail_gfp = pisa_stat_atomic_read(st->fail_gfp);
+ cached = pisa_pcp_stats(pa, &pcp_alloc, &pcp_miss, &pcp_free, &pcp_ks, &pcp_as);
+
+ spin_lock_irqsave(&pa->lock, flags);
+ alloc_pages = st->page_alloc_cnt;
+ free_pages = st->page_free_cnt;
+ block_alloc = st->block_alloc_cnt;
+ block_free = st->block_free_cnt;
+ block_scrub = st->block_scrub_cnt;
+ numa_mismatch = st->numa_mismatch_cnt;
+ alloc_kd = st->alloc_kd;
+ alloc_kc = st->alloc_kc;
+ alloc_ks = st->alloc_ks;
+ alloc_ad = st->alloc_ad;
+ alloc_ac = st->alloc_ac;
+ alloc_as = st->alloc_as;
+ num_idle = pa->num_idle_pages;
+ max_idle = pa->max_idle_pages;
+ spin_unlock_irqrestore(&pa->lock, flags);
+
+ /* Omit allocators that have never allocated or failed an allocation. */
+ if (!alloc_pages && !block_alloc && !block_fail && !fail_gfp)
+ continue;
+
+ pages_in_use = alloc_pages > free_pages ? alloc_pages - free_pages : 0;
+ blocks_in_use = block_alloc > block_free + cached ?
+ block_alloc - block_free - cached : 0;
+ if (pages_in_use)
+ used_pct = div64_u64((blocks_in_use << pa->order) * 100,
+ pages_in_use << PISA_ORDER);
+
+ seq_printf(s, "alloc %s order %u nid %d%s\n", pa->name, pa->order,
+ pa->nid, pa->prot == IOMMU_READ ? " tx-only" :
+ pa->prot == IOMMU_WRITE ? " rx-only" : "");
+ pisa_seq_kv(s, pisa_pages_2m, { "got", alloc_pages },
+ { "put", free_pages }, { "in_use", pages_in_use },
+ { "idle", num_idle }, { "idle_max", max_idle });
+ pisa_seq_kv(s, pisa_no_pages_2m,
+ { "backoff", pisa_stat_atomic_read(st->fail_backoff) },
+ { "budget", pisa_stat_atomic_read(st->fail_budget) },
+ { "nomem", pisa_stat_atomic_read(st->fail_nomem) },
+ { "split", pisa_stat_atomic_read(st->fail_split) });
+ pisa_seq_kv(s, "blocks", { "alloc", block_alloc },
+ { "free", block_free }, { "in_use", blocks_in_use },
+ { "used_pct", used_pct }, { "scrub", block_scrub });
+ pisa_seq_kv(s, "blocks", { "failed", block_fail },
+ { "gfp", fail_gfp }, { "numa_miss", numa_mismatch });
+ pisa_seq_kv(s, "allocations",
+ { "KD", alloc_kd }, { "KC", alloc_kc }, { "KS", alloc_ks },
+ { "AD", alloc_ad }, { "AC", alloc_ac }, { "AS", alloc_as });
+ if (pa->pcp) {
+ int cpu;
+
+ pisa_seq_kv(s, "pcpu", { "alloc", pcp_alloc },
+ { "miss", pcp_miss },
+ { "free", pcp_free }, { "cached", cached },
+ { "KS", pcp_ks }, { "AS", pcp_as });
+ for_each_possible_cpu(cpu) {
+ const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+ u64 a = READ_ONCE(pcp->alloc_cnt);
+ u64 m = READ_ONCE(pcp->miss_cnt);
+ u64 f = READ_ONCE(pcp->free_cnt);
+ unsigned int c = READ_ONCE(pcp->count);
+ char title[16];
+
+ if (!a && !f && !c)
+ continue;
+ snprintf(title, sizeof(title), "pcpu_%u", cpu);
+ pisa_seq_kv(s, title, { "alloc", a },
+ { "miss", m },
+ { "free", f }, { "cached", c });
+ }
+ }
+#ifdef CONFIG_IOMMU_DMA
+ pisa_seq_kv(s, "domains",
+ { "mapped", pisa_stat_atomic_read(st->page_map_cnt) });
+ pisa_seq_kv(s, "map_fail",
+ { "no_window", pisa_stat_atomic_read(st->fallback_nowindow) },
+ { "no_idx", pisa_stat_atomic_read(st->fallback_noidx) },
+ { "no_huge", pisa_stat_atomic_read(st->fallback_nohuge) },
+ { "map_err", pisa_stat_atomic_read(st->fallback_maperr) },
+ { "prot", pisa_stat_atomic_read(st->fallback_prot) });
+#endif
+ }
+ read_unlock(&pisa_allocs_lock);
+
+ return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(pisa_allocs);
+
/* Return the blocks a dead CPU cached, which nothing else would. */
static int pisa_cpu_dead(unsigned int cpu)
{
@@ -1751,6 +1992,10 @@ static int __init pisa_init(void)
struct shrinker *shrink;
int nid, i;
+ snprintf(pisa_pages_2m, sizeof(pisa_pages_2m), "pages_%luM", PISA_SIZE >> 20);
+ snprintf(pisa_pages_4k, sizeof(pisa_pages_4k), "pages_%luK", PAGE_SIZE >> 10);
+ snprintf(pisa_no_pages_2m, sizeof(pisa_no_pages_2m), "no_pages_%luM", PISA_SIZE >> 20);
+
pisa_node_init();
/* Default ceiling is 1/8th of RAM (floor of 16 PISA pages). */
@@ -1787,7 +2032,6 @@ static int __init pisa_init(void)
pisa_nodes[nid].pa[ro][order] = pa;
}
}
-
pisa_user_boot_enable();
/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
@@ -1799,6 +2043,10 @@ static int __init pisa_init(void)
pisa_cpu_dead) < 0)
pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
+ /* Built-in and never unloaded; debugfs stubs are no-ops when disabled. */
+ debugfs_create_file("allocs", 0444, debugfs_create_dir("pisa", NULL),
+ NULL, &pisa_allocs_fops);
+
shrink = shrinker_alloc(0, "pisa");
if (!shrink) {
pr_warn("pisa: shrinker registration failed\n");
diff --git a/drivers/iommu/pisa-kunit.c b/drivers/iommu/pisa-kunit.c
index 3fc17024bc9d0..22f4966412feb 100644
--- a/drivers/iommu/pisa-kunit.c
+++ b/drivers/iommu/pisa-kunit.c
@@ -291,9 +291,9 @@ static int fake_domain_setup(struct kunit *test)
static void test_map_and_domain_lifecycle(struct kunit *test)
{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
.coherent_dma_mask = DMA_BIT_MASK(64) };
- struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct pisa_window *win;
struct pisa_meta *meta;
struct pisa_alloc *pa;
@@ -574,14 +574,15 @@ static void test_gfp_pisa(struct kunit *test)
static void test_reservoir_and_scrubber(struct kunit *test)
{
+ struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
unsigned int old_res = READ_ONCE(pisa_reservoir_max);
unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
- struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
struct pisa_meta *meta;
struct pisa_alloc *pa;
struct page *p;
int nid;
+ KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
pisa_shrink_scan(NULL, &sc);
/* Refill reservoir to 1 PISA page per memory node, then consume one. */
@@ -607,9 +608,9 @@ static void test_reservoir_and_scrubber(struct kunit *test)
WRITE_ONCE(pisa_scrub_max, 512);
pisa_scrub_node(pa->nid);
KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+ KUNIT_EXPECT_GE(test, pa->stats.block_scrub_cnt, 1ULL);
}
}
- pisa_shrink_scan(NULL, &sc);
/* Trim reservoirs back to 0. */
WRITE_ONCE(pisa_reservoir_max, 0);
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index fc58c639f3288..2f5ea446b7de5 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -10,12 +10,11 @@
#include <linux/dma-map-ops.h>
#include <linux/dma-mapping.h>
#include <linux/pisa.h>
-#include <linux/export.h>
#include <linux/iommu.h>
#include <linux/iova.h>
-#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/moduleparam.h>
+#include <linux/seq_file.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/srcu.h>
@@ -106,6 +105,8 @@ static struct {
struct iommu_domain *domain;
dma_addr_t base;
spinlock_t map_lock; /* serialises iommu_map() in __pisa_map_phys() */
+ atomic_t slots;
+ char name[32];
} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
static DEFINE_SPINLOCK(pisa_domains_lock);
@@ -115,12 +116,56 @@ DEFINE_STATIC_SRCU(pisa_srcu);
/* Bumped under pisa_domains_lock whenever a domain index is released. */
unsigned int pisa_idx_gen __read_mostly;
+void pisa_domains_show(struct seq_file *s)
+{
+ unsigned int active = 0, draining = 0;
+ char bmap[PISA_MAX_DOMAINS];
+ unsigned long flags;
+ int idx;
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (pisa_domains[idx].domain) {
+ active++;
+ bmap[idx] = '1';
+ } else if (pisa_domains[idx].base) {
+ draining++;
+ bmap[idx] = 'd';
+ } else {
+ bmap[idx] = '.';
+ }
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+ pisa_seq_kv(s, "domains", { "active", active }, { "draining", draining },
+ { "max", PISA_MAX_DOMAINS });
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if ((idx & 63) == 0)
+ seq_printf(s, " %-12s", idx ? "" : "domains_map");
+ if ((idx & 7) == 0)
+ seq_putc(s, ' ');
+ seq_putc(s, bmap[idx]);
+ if ((idx & 63) == 63 || idx == PISA_MAX_DOMAINS - 1)
+ seq_putc(s, '\n');
+ }
+
+ spin_lock_irqsave(&pisa_domains_lock, flags);
+ for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+ if (!pisa_domains[idx].domain)
+ continue;
+ seq_printf(s, " domain_%-5d base %pad size %lluM slots %d name %s\n",
+ idx, &pisa_domains[idx].base, pisa_window_size >> 20,
+ atomic_read(&pisa_domains[idx].slots), pisa_domains[idx].name);
+ }
+ spin_unlock_irqrestore(&pisa_domains_lock, flags);
+}
/* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
static void pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
int idx, dma_addr_t iova, size_t unmapped)
{
pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
&iova, domain, idx, unmapped);
+ pisa_stat_atomic_inc(pisa_reclaim_stats.unmap_fail);
WRITE_ONCE(meta->flags, meta->flags | PISA_QUARANTINED);
}
@@ -142,6 +187,7 @@ int pisa_unmap_all(struct pisa_meta *meta)
if (!test_and_clear_bit(idx, meta->domains_mapped))
continue;
+ atomic_dec(&pisa_domains[idx].slots);
spin_lock_irqsave(&pisa_domains_lock, flags);
domain = pisa_domains[idx].domain;
@@ -217,6 +263,8 @@ void pisa_domain_release(struct iommu_domain *domain)
/* Mark slot FREE and bump @pisa_idx_gen so NOIDX domains retry. */
spin_lock_irqsave(&pisa_domains_lock, flags);
pisa_domains[idx].base = 0;
+ atomic_set(&pisa_domains[idx].slots, 0);
+ pisa_domains[idx].name[0] = '\0';
WRITE_ONCE(pisa_idx_gen, pisa_idx_gen + 1);
spin_unlock_irqrestore(&pisa_domains_lock, flags);
@@ -239,6 +287,7 @@ static unsigned int pisa_unmap_domain_locked(struct pisa_meta *meta,
if (!test_and_clear_bit(idx, meta->domains_mapped))
return 0;
+ atomic_dec(&pisa_domains[idx].slots);
unmapped = iommu_unmap_fast(domain, iova, PISA_SIZE, gather);
if (unlikely(unmapped != PISA_SIZE))
@@ -388,6 +437,9 @@ int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct p
pisa_domains[idx].domain = domain;
pisa_domains[idx].base = base;
spin_lock_init(&pisa_domains[idx].map_lock);
+ atomic_set(&pisa_domains[idx].slots, 0);
+ strscpy(pisa_domains[idx].name, dev_name(dev),
+ sizeof(pisa_domains[idx].name));
WRITE_ONCE(win->domain_idx, idx + PISA_IDX_FIRST);
/* Publish .domain_idx before .base (paired with is_pisa_iova()). */
smp_store_release(&win->base, base);
@@ -435,8 +487,10 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
/* Every leaf PTE of the page carries IOMMU_CACHE | @meta->prot. */
block_prot = IOMMU_CACHE | meta->prot;
- if (unlikely(prot & ~block_prot))
+ if (unlikely(prot & ~block_prot)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_prot);
return -EACCES;
+ }
/* Acquire pairs with the .base release in pisa_window_assign(). */
if (unlikely(!smp_load_acquire(&win->base))) {
@@ -445,8 +499,15 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
if (likely(!ret))
ret = pisa_window_assign(dev, domain, win);
- if (ret)
+ if (ret) {
+ if (ret == -EOPNOTSUPP)
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nohuge);
+ else if (ret == -EBUSY)
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_noidx);
+ else
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nowindow);
return ret;
+ }
}
addr = pisa_phys_to_iova(win, meta, phys);
@@ -454,8 +515,10 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
/* Check @dev's own DMA mask and bus limit against the window IOVA. */
limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
limit = min_not_zero(limit, dev->coherent_dma_mask);
- if (unlikely(addr + size - 1 > limit))
+ if (unlikely(addr + size - 1 > limit)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_nowindow);
return -ENOSPC;
+ }
domain_idx = win->domain_idx - PISA_IDX_FIRST;
if (likely(test_bit(domain_idx, meta->domains_mapped))) {
@@ -471,10 +534,14 @@ int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
ret = iommu_map(domain, pisa_phys_to_iova(win, meta, pisa_phys),
pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
- if (likely(!ret)) {
+ if (unlikely(ret)) {
+ pisa_stat_atomic_inc(meta->pa->stats.fallback_maperr);
+ } else {
/* Ensure leaf PTE is visible before setting the domain bit. */
smp_mb__before_atomic();
set_bit(domain_idx, meta->domains_mapped);
+ atomic_inc(&pisa_domains[domain_idx].slots);
+ pisa_stat_atomic_inc(meta->pa->stats.page_map_cnt);
}
}
spin_unlock_irqrestore(&pisa_domains[domain_idx].map_lock, flags);
diff --git a/drivers/iommu/pisa-node.c b/drivers/iommu/pisa-node.c
index 7469674133620..66280715ba504 100644
--- a/drivers/iommu/pisa-node.c
+++ b/drivers/iommu/pisa-node.c
@@ -15,6 +15,7 @@
#include <linux/nodemask.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
+#include <linux/seq_file.h>
#include <linux/spinlock.h>
#include <linux/topology.h>
#include <linux/workqueue.h>
@@ -139,8 +140,10 @@ static void pisa_reservoir_refill_node(int nid)
meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
pisa_want_decrypt(), &clean);
- if (IS_ERR(meta))
+ if (IS_ERR(meta)) {
+ pisa_stat_inc(res->stats.refill_err);
break;
+ }
/* Decrypted and init_on_alloc/init_on_free pages are already zeroed. */
if (!clean)
@@ -149,6 +152,7 @@ static void pisa_reservoir_refill_node(int nid)
spin_lock_irqsave(&res->lock, flags);
list_add(&meta->list, &res->pages);
res->nr_pages++;
+ pisa_stat_inc(res->stats.refill);
spin_unlock_irqrestore(&res->lock, flags);
cond_resched();
@@ -169,6 +173,7 @@ static void pisa_reservoir_trim_node(int nid, unsigned int max)
if (meta) {
list_del_init(&meta->list);
res->nr_pages--;
+ pisa_stat_inc(res->stats.trim);
}
spin_unlock_irqrestore(&res->lock, flags);
if (!meta)
@@ -184,8 +189,7 @@ void pisa_schedule_scrub(int nid)
{
struct pisa_node *node;
- if ((!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate)) ||
- nid == NUMA_NO_NODE)
+ if (!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate))
return;
node = &pisa_nodes[nid];
@@ -253,6 +257,7 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool
if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
list_del_init(&meta->list);
res->nr_pages--;
+ pisa_stat_inc(res->stats.hit);
refill = res->nr_pages <= max / 2;
} else {
meta = NULL;
@@ -282,6 +287,7 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *cle
*clean = true;
return meta;
}
+ pisa_stat_atomic_inc(pisa_nodes[nid].stats.miss);
/* Empty: pop_node() only schedules after a successful pop. */
pisa_schedule_reservoir_node(nid);
}
@@ -339,3 +345,34 @@ void __init pisa_node_init(void)
INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
}
}
+
+/* Print the reservoir of each node that has seen any activity. */
+void pisa_reservoir_show(struct seq_file *s)
+{
+ bool header = false;
+ int nid;
+
+ for_each_node_state(nid, N_MEMORY) {
+ struct pisa_node *res = &pisa_nodes[nid];
+ unsigned int nr = READ_ONCE(res->nr_pages);
+ s64 refill = pisa_stat_read(res->stats.refill);
+ s64 refill_err = pisa_stat_read(res->stats.refill_err);
+ s64 miss = pisa_stat_atomic_read(res->stats.miss);
+ char title[16];
+
+ /* Pages served or trimmed were refilled first. */
+ if (!nr && !refill && !refill_err && !miss)
+ continue;
+ if (!header) {
+ seq_puts(s, "reservoir\n");
+ header = true;
+ }
+ snprintf(title, sizeof(title), "node_%d", nid);
+ pisa_seq_kv(s, title, { pisa_pages_2m, nr },
+ { "hit", pisa_stat_read(res->stats.hit) },
+ { "miss", miss });
+ pisa_seq_kv(s, title, { "refill", refill },
+ { "refill_err", refill_err },
+ { "trim", pisa_stat_read(res->stats.trim) });
+ }
+}
diff --git a/drivers/iommu/pisa-priv.h b/drivers/iommu/pisa-priv.h
index 1ecfebbaeb381..8eec8996bbca7 100644
--- a/drivers/iommu/pisa-priv.h
+++ b/drivers/iommu/pisa-priv.h
@@ -8,9 +8,7 @@
#include <linux/list.h>
#include <linux/llist.h>
#include <linux/mem_encrypt.h>
-#include <linux/nodemask.h>
#include <linux/spinlock.h>
-#include <linux/topology.h>
#include <linux/types.h>
#include <linux/workqueue.h>
@@ -136,13 +134,78 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
struct pisa_pcp {
spinlock_t lock; /* by this CPU, or by a drain from any CPU */
unsigned int count;
+ u64 alloc_cnt;
+ u64 miss_cnt;
+ u64 free_cnt;
+ u64 alloc_ks;
+ u64 alloc_as;
struct page *blocks[PISA_PCP_HIGH];
};
-/* Per-allocator statistics, see pisa_allocs_show(). */
+#define pisa_stat_add(var, val) ((var) += (val))
+#define pisa_stat_inc(var) pisa_stat_add(var, 1)
+#define pisa_stat_dec(var) pisa_stat_add(var, -1)
+#define pisa_stat_read(var) READ_ONCE(var)
+
+#define pisa_stat_atomic_add(var, val) atomic64_add(val, &(var))
+#define pisa_stat_atomic_inc(var) atomic64_inc(&(var))
+#define pisa_stat_atomic_read(var) atomic64_read(&(var))
+
+/**
+ * struct pisa_alloc_stats - Per-allocator statistics, see pisa_allocs_show()
+ * @page_alloc_cnt: PISA pages allocated for this allocator
+ * @page_free_cnt: PISA pages released back to the buddy allocator
+ * @block_alloc_cnt: Block allocations satisfied
+ * @block_free_cnt: Block frees recycled into allocator
+ * @block_scrub_cnt: Dirty blocks zeroed by background scrubber
+ * @numa_mismatch_cnt: Block allocations satisfied from a non-target NUMA node
+ * @alloc_kd: Sleepable non-zeroing block allocations
+ * @alloc_kc: Sleepable zeroing block allocations served from clean_bitmap
+ * @alloc_ks: Sleepable zeroing block allocations zeroed inline
+ * @alloc_ad: Atomic non-zeroing block allocations
+ * @alloc_ac: Atomic zeroing block allocations served from clean_bitmap
+ * @alloc_as: Atomic zeroing block allocations zeroed inline
+ * @fail_backoff: PISA page acquisitions skipped due to atomic backoff
+ * @fail_budget: PISA page acquisitions refused by global pisa_max_pages
+ * @fail_nomem: PISA page acquisitions that ran out of memory
+ * @fail_split: split_page_compound() rejections
+ * @block_alloc_fail: PISA page acquisitions that failed on empty allocator
+ * @fail_gfp: pisa_alloc() calls declined due to unsupported GFP
+ * @page_map_cnt: PISA page mappings established across all domains
+ * @fallback_nowindow: Buffers mapped individually due to no usable IOVA window
+ * @fallback_noidx: Buffers mapped individually due to exhausted domain indices
+ * @fallback_nohuge: Buffers mapped individually because domain cannot use PISA
+ * @fallback_maperr: Buffers mapped individually because PISA mapping failed
+ * @fallback_prot: Buffers mapped individually due to protection mismatch
+ */
struct pisa_alloc_stats {
+ /* Updated under @pa->lock. */
u64 page_alloc_cnt;
u64 page_free_cnt;
+ u64 block_alloc_cnt;
+ u64 block_free_cnt;
+ u64 block_scrub_cnt;
+ u64 numa_mismatch_cnt;
+ u64 alloc_kd;
+ u64 alloc_kc;
+ u64 alloc_ks;
+ u64 alloc_ad;
+ u64 alloc_ac;
+ u64 alloc_as;
+
+ /* Updated without @pa->lock (e.g. pisa_alloc_split_page() and map path). */
+ atomic64_t fail_backoff;
+ atomic64_t fail_budget;
+ atomic64_t fail_nomem;
+ atomic64_t fail_split;
+ atomic64_t block_alloc_fail;
+ atomic64_t fail_gfp;
+ atomic64_t page_map_cnt;
+ atomic64_t fallback_nowindow;
+ atomic64_t fallback_noidx;
+ atomic64_t fallback_nohuge;
+ atomic64_t fallback_maperr;
+ atomic64_t fallback_prot;
};
/**
@@ -195,6 +258,18 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
READ_ONCE(pa->num_idle_pages);
}
+/* Reservoir statistics, in PISA pages, see pisa_reservoir_show(). */
+struct pisa_reservoir_stats {
+ /* Updated under @res->lock or by the per-node singleton @res->work. */
+ u64 hit; /* requests for this node it served */
+ u64 refill; /* PISA pages added by the refill */
+ u64 refill_err; /* refill allocations that failed */
+ u64 trim; /* PISA pages released by the trim */
+
+ /* Updated without @res->lock when the reservoir is empty. */
+ atomic64_t miss; /* requests for this node found it empty */
+};
+
/* Per-NUMA-node PISA state. */
struct pisa_node {
unsigned long next_alloc; /* atomic alloc backoff jiffies */
@@ -206,8 +281,28 @@ struct pisa_node {
unsigned int nr_pages;
unsigned long flags; /* PISA_NODE_* requests */
struct work_struct work; /* refill/trim/scrub, on this node */
+ struct pisa_reservoir_stats stats;
} ____cacheline_aligned_in_smp;
+/* Global reclaim and shrinker statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats {
+ /* Updated by singleton pisa_reclaim_work. */
+ u64 reclaim_run; /* pisa_reclaim_work_fn() runs */
+ u64 reclaim_pages; /* PISA pages it released */
+
+ /* Updated locklessly. */
+ atomic64_t shrink_scan; /* pisa_shrink_scan() calls */
+ atomic64_t pte_pages_res; /* 4KB pages taken from the reservoirs */
+ atomic64_t pte_pages_idle; /* 4KB pages taken from idle allocators */
+ atomic64_t unmap_fail; /* PISA page unmaps that failed */
+};
+
+extern struct pisa_reclaim_stats pisa_reclaim_stats;
+
+extern char pisa_pages_2m[16];
+extern char pisa_pages_4k[16];
+extern char pisa_no_pages_2m[16];
+
/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
extern struct pisa_node pisa_nodes[MAX_NUMNODES];
@@ -255,8 +350,6 @@ static inline void pisa_alloc_backoff(int nid, bool can_block)
WRITE_ONCE(pisa_nodes[nid].next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
}
bool pisa_want_decrypt(void);
-bool pisa_place_meta(struct pisa_meta *meta);
-void pisa_schedule_reclaim(void);
int pisa_page_prepare(struct page *page, struct pisa_meta *meta);
int pisa_page_unprepare(struct pisa_meta *meta);
@@ -267,6 +360,8 @@ __printf(2, 3)
void pisa_set_name(struct pisa_alloc *pa, const char *fmt, ...);
int pisa_enable_pcp(struct pisa_alloc *pa);
void pisa_pcp_drain(struct pisa_alloc *pa);
+bool pisa_place_meta(struct pisa_meta *meta);
+void pisa_schedule_reclaim(void);
void pisa_release_page(struct pisa_meta *meta);
unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
unsigned long nr_pages, struct page **page_array, bool frozen);
@@ -300,8 +395,28 @@ void pisa_schedule_scrub_all(void);
struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
unsigned long pisa_reservoir_count(void);
unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+
+struct seq_file;
+void pisa_reservoir_show(struct seq_file *s);
+void pisa_user_show(struct seq_file *s);
void pisa_user_boot_enable(void);
+/* One name/value pair of a debugfs statistics line. */
+struct pisa_kv {
+ const char *name;
+ s64 val;
+};
+
+void __pisa_seq_kv(struct seq_file *s, const char *title, const struct pisa_kv *st, unsigned int n);
+
+/* Print @title and fixed-width name/value pairs to debugfs seq_file @s. */
+#define pisa_seq_kv(s, title, ...) \
+do { \
+ const struct pisa_kv __st[] = { __VA_ARGS__ }; \
+ \
+ __pisa_seq_kv(s, title, __st, ARRAY_SIZE(__st)); \
+} while (0)
+
#endif /* CONFIG_PISA */
#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
@@ -316,6 +431,7 @@ void pisa_slots_init(void);
int pisa_slot_alloc(struct pisa_meta *meta);
void pisa_slot_free(unsigned int slot);
int pisa_unmap_all(struct pisa_meta *meta);
+void pisa_domains_show(struct seq_file *s);
int pisa_window_assign(struct device *dev, struct iommu_domain *domain, struct pisa_window *win);
int __pisa_map_phys(dma_addr_t *iova, struct device *dev,
struct iommu_domain *domain, phys_addr_t phys,
@@ -364,6 +480,7 @@ static inline void pisa_slots_init(void) {}
static inline int pisa_slot_alloc(struct pisa_meta *meta) { return 0; }
static inline void pisa_slot_free(unsigned int slot) {}
static inline int pisa_unmap_all(struct pisa_meta *meta) { return 0; }
+static inline void pisa_domains_show(struct seq_file *s) {}
static inline int pisa_map_phys(dma_addr_t *iova, struct device *dev,
struct iommu_domain *domain, phys_addr_t phys,
diff --git a/drivers/iommu/pisa-user.c b/drivers/iommu/pisa-user.c
index 7dca3bf066f0d..5cf199943f82c 100644
--- a/drivers/iommu/pisa-user.c
+++ b/drivers/iommu/pisa-user.c
@@ -21,6 +21,7 @@
#include <linux/pagewalk.h>
#include <linux/folio_batch.h>
#include <linux/sched/clock.h>
+#include <linux/seq_file.h>
#include <linux/swap.h>
#include "pisa-priv.h"
@@ -88,6 +89,26 @@ module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 064
static bool pisa_user_ready __read_mostly;
static DEFINE_MUTEX(pisa_user_lock);
+/* Statistics, see pisa_user_show(). */
+struct pisa_user_stats {
+ atomic64_t in; /* pages migrated into PISA blocks */
+ atomic64_t in_fail; /* ... isolated but not migrated */
+ atomic64_t out; /* pages migrated back to ordinary pages */
+ atomic64_t out_fail; /* ... isolated but not migrated */
+ atomic64_t no_budget; /* pages not moved in: cap reached */
+ atomic64_t busy; /* pages skipped: referenced elsewhere */
+ atomic64_t large; /* pages skipped: in a large anon folio */
+ atomic64_t copy; /* TCP chunks copied: page not in PISA */
+};
+
+static struct pisa_user_stats pisa_user_stats;
+
+/* Called by zerocopy_fill_skb_from_iter() when it asks TCP to copy. */
+void pisa_stats_inc_user_copy(void)
+{
+ pisa_stat_atomic_inc(pisa_user_stats.copy);
+}
+
/* Lazily create per-node user allocators on first use (process context). */
static bool pisa_user_init(void)
{
@@ -141,6 +162,9 @@ static unsigned long pisa_user_in_use(void)
unsigned long used = 0;
int nid;
+ /* Pairs with smp_store_release() in pisa_user_init(). */
+ if (!smp_load_acquire(&pisa_user_ready))
+ return 0;
for_each_node_state(nid, N_MEMORY) {
unsigned int dir;
@@ -255,7 +279,7 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
.same_alloc = same_alloc,
};
enum migrate_mode mode = MIGRATE_SYNC;
- unsigned int done = 0;
+ unsigned int nr, done = 0;
if (list_empty(list))
return 0;
@@ -264,9 +288,17 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
if (!gfpflags_allow_blocking(gfp))
mode = MIGRATE_ASYNC;
}
+ nr = list_count_nodes(list); /* all order-0 */
if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
NULL, (unsigned long)&mtc, mode, MR_SYSCALL, &done))
putback_movable_pages(list);
+ if (to_pisa) {
+ pisa_stat_atomic_add(pisa_user_stats.in, done);
+ pisa_stat_atomic_add(pisa_user_stats.in_fail, nr - done);
+ } else {
+ pisa_stat_atomic_add(pisa_user_stats.out, done);
+ pisa_stat_atomic_add(pisa_user_stats.out_fail, nr - done);
+ }
return done;
}
@@ -285,7 +317,7 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
int extra = pinned ? GUP_PIN_COUNTING_BIAS : 1;
bool drained = false;
LIST_HEAD(list);
- unsigned int i, want = 0, need_budget = 0;
+ unsigned int i, want = 0, need_budget = 0, large = 0;
if (!pisa_user_init())
return false;
@@ -293,15 +325,21 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
+ /* THP/mTHP heap pages are never migrated, see pisa_user_want() */
+ large += folio_test_large(folio) && folio_test_anon(folio);
if (pisa_user_want_dir(folio, true, dir, false)) {
want++;
need_budget += !is_pisa_pfn(folio_pfn(folio));
}
}
+ if (large)
+ pisa_stat_atomic_add(pisa_user_stats.large, large);
if (!want)
return false;
- if (need_budget && pisa_user_budget() <= 0)
+ if (need_budget && pisa_user_budget() <= 0) {
+ pisa_stat_atomic_add(pisa_user_stats.no_budget, need_budget);
return false;
+ }
for (i = 0; i < nr; i++) {
struct folio *folio = page_folio(pages[i]);
@@ -312,8 +350,10 @@ bool pisa_upgrade_user_pages(struct page **pages, unsigned int nr,
lru_add_drain();
drained = true;
}
- if (pisa_user_busy(folio, extra))
+ if (pisa_user_busy(folio, extra)) {
+ pisa_stat_atomic_inc(pisa_user_stats.busy);
continue;
+ }
isolate_folio_to_list(folio, &list);
}
if (list_empty(&list))
@@ -583,9 +623,14 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
struct folio *folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false) &&
- !pisa_user_busy(folio, 0) && isolate_folio_to_list(folio, &list))
- n++;
+ if (to_pisa && folio_test_large(folio) && folio_test_anon(folio)) {
+ pisa_stat_atomic_inc(pisa_user_stats.large);
+ } else if (pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false)) {
+ if (pisa_user_busy(folio, 0))
+ pisa_stat_atomic_inc(pisa_user_stats.busy);
+ else if (isolate_folio_to_list(folio, &list))
+ n++;
+ }
folio_walk_end(&fw, vma);
}
if (n == PISA_USER_BATCH) {
@@ -597,3 +642,21 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
pisa_user_migrate_dir(&list, to_pisa, DMA_TO_DEVICE, false, gfp);
return 0;
}
+
+/* Print user page state and migration statistics to debugfs seq_file @s. */
+void pisa_user_show(struct seq_file *s)
+{
+ seq_puts(s, "user\n");
+ pisa_seq_kv(s, pisa_pages_4k, { "in_use", pisa_user_in_use() },
+ { "max", pisa_user_max_4k_pages });
+ pisa_seq_kv(s, "migrate",
+ { "in", pisa_stat_atomic_read(pisa_user_stats.in) },
+ { "in_fail", pisa_stat_atomic_read(pisa_user_stats.in_fail) },
+ { "out", pisa_stat_atomic_read(pisa_user_stats.out) },
+ { "out_fail", pisa_stat_atomic_read(pisa_user_stats.out_fail) });
+ pisa_seq_kv(s, "skipped",
+ { "no_budget", pisa_stat_atomic_read(pisa_user_stats.no_budget) },
+ { "busy", pisa_stat_atomic_read(pisa_user_stats.busy) },
+ { "large", pisa_stat_atomic_read(pisa_user_stats.large) },
+ { "copy", pisa_stat_atomic_read(pisa_user_stats.copy) });
+}
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 1d9b07370cc75..667321b18ae47 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -147,6 +147,7 @@ bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
bool __pisa_bdev_enabled(struct block_device *bdev, enum dma_data_direction dir);
bool __pisa_mapping_enabled(const struct address_space *mapping,
enum dma_data_direction dir);
+void pisa_stats_inc_user_copy(void);
int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa);
@@ -331,6 +332,10 @@ pisa_filemap_alloc(const struct address_space *mapping, gfp_t gfp,
return NULL;
}
+static inline void pisa_stats_inc_user_copy(void)
+{
+}
+
static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
unsigned long end, bool to_pisa)
{
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 8ba9c39cb05ec..360ed445fe779 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -693,6 +693,8 @@ static bool pisa_zc_iter(int *ret, struct sk_buff *skb, struct iov_iter *from, s
if (keep)
return false;
+ if (skb->len == orig_len)
+ pisa_stats_inc_user_copy();
*ret = (skb->len != orig_len) ? 0 : -EXDEV;
return true;
}
--
2.56.0.542.g3ef89b7020-goog
^ permalink raw reply related [flat|nested] 26+ messages in thread