linux-api.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
       [not found] <20261109182533.3594984-1-lrizzo@google.com>
@ 2026-11-20  1:06 ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
                     ` (26 more replies)
  0 siblings, 27 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

This is v3 of the RFC previously posted as DMA_PMD [0], now renamed PISA
("Pre-mapped IOVA Superpage Allocator"). The text below follows
Documentation/core-api/pisa.rst (patch 25/25), which has the details.

=== OVERVIEW

PISA is an extension to the page allocator: requests with __GFP_PISA
return memory that makes I/O-related operations cheaper when using DMA-FQ,
DMA, or Confidential Computing swiotlb. I/O devices can individually opt
in to use PISA for specific buffer classes (e.g. read or write data,
headers, zerocopy, ...). Relevant kernel subsystems (mm, iommu, dma, disk
I/O, page cache, the network stack) are modified to make use of PISA
memory for opt-in devices.

The design and implementation have the following main goals:

- no changes to individual device drivers (or minimal changes for NIC
  drivers that use custom RX page allocators)
- minimize changes to base code: in most cases one or a few lines of code
- safety: PISA memory is only used for devices that opt in, and it is
  returned to the buddy allocator only after IOMMU mappings and IOTLB
  entries have been flushed
- no dependency on struct page or struct folio
- compile-time option to disable or enable PISA
- zero runtime overhead when no devices opt in (static keys and inline
  hooks)
- minimal overhead when in use (direct PFN lookup in a flat metadata
  table)
- keep memory overhead as low as possible (adaptive shrinking and
  compaction of memory)

=== MECHANISMS

PISA returns order-N blocks backed by 2MB physically contiguous pages, so
it can use 2MB IOMMU leaves and reduce the IOTLB pressure by up to 512x.
PISA maps memory in pre-allocated per-domain IOVA ranges, and recycles
blocks and mappings until the 2MB page is returned to the buddy allocator.
This makes most of dma map/unmap a no-op, also avoiding related VM exits
in overcommitted VMs. In Confidential Computing VMs, PISA pages are also
decrypted, removing the need for bounce buffers.

The kernel identifies devices and sockets that use an IOMMU or
Confidential Computing and have opted into PISA, and allocates (or
migrates) I/O buffers using PISA memory so that DMA mapping and unmapping
are mostly no-ops. All the above makes performance comparable to identity
mode, with isolation better than DMA-FQ and comparable to DMA.

Devices that do not opt in continue to use their native DMA/DMA-FQ mode.

=== PERFORMANCE

Your mileage may vary. Enabling the IOMMU will increase load on various
system components (CPU, IOTLB, bus latency, increased locking
serialization) but the impact on throughput will only show up when one or
more of these resources become fully congested.

Some anecdotal data:

NVMe: fio throughput in IOPS, 4KB random reads, O_DIRECT, on an x86 system
with 8 NVMe disks capable of 1.7M IOPS each

    drives      DMA     PISA    identity
    1           380K    1.7M    1.7M
    8           1.3M    13.5M   13.5M

Network: throughput in Gbps on a Confidential Computing VM, 200G NIC

    flows/mtu   swiotlb PISA
    1/1460      36      65
    1/4148      45      65
    16/1460     130     170
    16/4148     130     182
    256/1460    140     170
    256/4148    140     182

Network: throughput and SMMU performance counters on a large ARM64 system
with 2x200G NICs running bidirectional traffic (this shows the effect of
IOTLB thrashing)

    === DMA-FQ (total throughput ~440Gbps)
    25,757,488      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
    16,634,941      smmuv3_pmcg_*/event=0x81/   IOTLB misses

    === PISA on top of strict DMA (total throughput ~745Gbps)
    22,693,218      smmuv3_pmcg_*/event=0x80/   IOTLB lookups
        30,536      smmuv3_pmcg_*/event=0x81/   IOTLB misses

=== COMPARISON WITH STANDARD DMA MODES

PISA layered on top of strict DMA, compared with identity, DMA-FQ and
strict DMA ( > means PISA is better ):

                                      identity   DMA-FQ    DMA
  dma_map cost                          ~=         >>       >>
  dma_unmap cost                        ==         >>       >>
  IOTLB friendliness                    na         >>       >>
  Buffer shared across domains          na         <=       <=
  Memory remains accessible to device   >>         >>       <=
  Needs bounce buffers                  >>         >>       >>
  Page alloc/free cost                  >=         >=       >=

- dma_map/dma_unmap cost: once a 2MB PISA page is mapped in a domain's
  reserved IOVA window, dma_map_*() is an offset lookup and dma_unmap_*()
  is a no-op, avoiding IOVA tree allocation, page-table walks, and
  synchronous IOTLB shootdowns.
- IOTLB friendliness: 2MB IOMMU leaves reduce IOTLB pressure by up to 512x
- Buffer shared across domains: even in DMA-FQ and DMA, sub-page buffers
  from task_frag or skb head slabs can cause the underlying page to be
  mapped across multiple domains. PISA exhibits the same behavior, just at
  2MB granularity.
- Memory remains accessible to device: in identity mode all of RAM is
  always accessible. In DMA-FQ, pages returned to the buddy allocator
  remain accessible until the flush queue drains. PISA and DMA both unmap
  the page and perform a synchronous IOTLB flush before returning memory
  to the buddy allocator. In all modes, pagepool keeps receive buffers
  mapped and accessible to the device while queued in the rx socket
  buffer. PISA expands this visibility also to page cache and O_DIRECT
  pages.
- Needs bounce buffers: in Confidential Computing VMs, PISA allocators
  pre-decrypt 2MB pages in place, avoiding swiotlb bounce-buffer copies.

=== BACKGROUND

Enabling the IOMMU costs CPU to install and remove PTEs on dma_map/unmap,
CPU and latency to flush the IOTLB, and higher DMA latency from IOMMU page
walks when the working set exceeds the IOTLB size. The last one is among
the biggest bottlenecks for high speed devices: network devices have very
little IOVA locality, which makes the IOTLB almost completely ineffective.

The known mitigations are recycling buffers and mappings (common for
queues and RX buffers, but not for network TX buffers), lazy IOTLB flush
(DMA-FQ, which recycles memory without waiting for the flush), and larger
mappings (not currently pursued by the kernel). PISA combines them:

- intercept the small set of functions used to allocate, map, unmap, and
  free device-accessible memory (rings and buffers)
- return memory backed by physically contiguous 2MB pages and 2MB IOMMU
  mappings
- heavily recycle memory and mappings
- wait until the IOTLB has been flushed before returning unused PISA pages
  to the system

=== IMPLEMENTATION AND CODE LAYOUT

PISA is implemented by a few groups of commits:

- mm: infrastructure for per-page metadata (no dependency on struct page),
  hook into alloc_pages(), PISA optimizations (per-cpu caches, shrinker,
  background zeroing)
- driver core: per-device sysfs entries to opt in
- iommu/dma: manage PISA IOVA ranges and intercept map/unmap
- block, mm, net: transparently upgrade pages to PISA for disk I/O and
  network tx zerocopy
- net: identify sockets that need PISA pages, hook PISA into page frags,
  skb heads, page_pool

The code layout now matches the "allocator" nature of PISA: most of the
code is in mm/, and iommu and net only carry subsystem-specific changes.

  mm/pisa-*.c                            metadata, allocators, per-node
                                         reservoir, user and page-cache
                                         pages (~5100 lines)
  drivers/iommu/pisa-map.c               IOVA ranges, map/unmap
                                         (~500 lines)
  net/core/pisa-net.c, include/net/pisa.h
                                         socket and skb glue (~450 lines)
  include/linux/pisa*.h                  public API and inline hooks

The functions that need to be intercepted are few:

- page_pool_dev_alloc*(), socket TX page frags, __free_pages()
- dma_alloc_attrs(), dma_map_*(), dma_unmap_*()
- __filemap_get_folio_mpol(), ractl_alloc_folio(), writeback_get_folio(),
  __bio_iov_iter_get_pages(), madvise()

and most devices automatically inherit the benefits. Fast-path hooks are
guarded by static keys: pisa_key remains disabled (a single NOP) until the
first device opts in via sysfs. NIC drivers that do not use page_pool need
a few lines to pass __GFP_PISA_RW when allocating RX buffers; gve is
converted as an example.

=== INTERNAL MECHANISMS

- PISA metadata: each 2MB page used for PISA memory is identified by a
  compact index (slot). Two flat arrays in virtual address space, with
  holes mapped read-only to a zero page:

  1. pisa_pfn2slot[pfn >> PISA_ORDER] (4 bytes per 2MB physical frame)
     packs the status flags and the slot. It is populated for system RAM
     at boot or on hotplug, and grows by 4KB for each 2GB of physical RAM
     regardless of RAM layout.

  2. pisa_meta_array[slot] (256 bytes per 2MB PISA page) holds struct
     pisa_meta: base PFN, flags, free/clean/dirty subpage bitmaps, the
     bitmap of IOMMU domains with an active 2MB mapping, and list/RCU
     linkage. Backing pages are allocated on demand up to pisa.max_pages
     (default 1/8th of RAM).

- Per-domain reserved IOVA ranges: PISA reserves on each domain an IOVA
  range covering the slot table, and a page is mapped on demand at offset
  slot * PISA_SIZE in each domain that requires it. This makes PA-IOVA
  translation just an addition, uniquely identifies PISA IOVAs, and does
  not expose physical addresses.

- alloc_pages() compatible allocator: a pisa_alloc object serves
  fixed-order allocations, returning objects fully compatible with those
  of the page allocator. alloc_pages() and related requests are routed to
  PISA via __GFP_PISA, which makes it easy to select PISA pages where
  needed (two lines in page_pool, similar small changes in page frag
  refill and disk I/O buffers).

- Safe release of memory: freed blocks return to PISA with mappings still
  active. When all blocks of a 2MB page are unused the page becomes idle;
  idle pages above pisa.max_idle are released after pisa.idle_ms, or by
  the shrinker: mappings are removed, a synchronous IOTLB flush is issued,
  and after an RCU grace period (and re-encryption if needed) the page is
  returned to the buddy allocator.

- Memory compaction: partially allocated 2MB pages are classified as dense
  (preferred for new allocations) or sparse (left to drain). For user
  pages, pisa.scrub_migrate also migrates live 4KB pages out of old sparse
  2MB pages in the background.

=== MODIFIED KERNEL APIS AND CONTROLS

Opt-in is per device, through
/sys/devices/.../pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads,tx_zerocopy}.

- Streaming DMA map and unmap: a buffer belonging to a PISA page is mapped
  via the domain's reserved window with a simple addition, typically
  reusing existing mappings; unmap is a range check and a no-op. Under
  dma-direct, decrypted PISA pages are mapped without swiotlb bounce.
- Coherent DMA (pisa_coherent): sleepable allocations up to 2MB are carved
  out of the per-node order-0 allocator.
- Network receive buffers (pisa_rx_bufs): a two-line change in page_pool
  adds __GFP_PISA_RW, falling back to the buddy allocator on exhaustion.
- Socket TX page fragments (pisa_tx_bufs): fragments come from PISA pages
  mapped DMA_TO_DEVICE, only for data routed to an opt-in device with a
  DMA parent. Loopback, virtual devices, kTLS, sockmap, KCM and MPTCP use
  ordinary pages.
- Small skb heads (pisa_tx_skb_heads): served by an
  skbuff_small_head_pisa slab cache backed by PISA memory mapped
  DMA_TO_DEVICE. Never used in Confidential Computing, where it would
  expose kernel pointers; tx-copybreak avoids the swiotlb cost there.
- Zerocopy receive: RX buffers from PISA pages can be mapped into user
  space by TCP_ZEROCOPY_RECEIVE.
- Zerocopy transmit (pisa_tx_zerocopy): MSG_ZEROCOPY sends migrate
  eligible user pages into DMA_TO_DEVICE PISA blocks on a best-effort
  basis. With pisa_tx_bufs also set, TCP copies the pages that could not
  be migrated.
- User and disk I/O pages: madvise(MADV_PISA) migrates private anonymous
  pages into PISA and MADV_NOPISA migrates them back. For block devices
  with pisa_rx_bufs (reads) or pisa_tx_bufs (writes) on a parent device,
  new order-0 page-cache folios are allocated from PISA, and writeback and
  direct-I/O bio preparation migrate eligible folios.

Global controls are pisa.* parameters: max_pages, reservoir_max, max_idle,
idle_ms, pcp_batch, compact, target_usage_percent, old_ms, scrub_max,
scrub_migrate, user_max_pages. Counters are in
/sys/kernel/debug/pisa/allocs.

=== LIMITATIONS

- Not compatible with PREEMPT_RT (enforced at compile time).
- Only DMA-coherent devices can use PISA (enforced at runtime).
- IOMMU mappings follow the direction the allocator was created with:
  descriptor queues and receive buffers are DMA_BIDIRECTIONAL, disk and
  network write buffers and socket TX fragments are DMA_TO_DEVICE.

=== CHANGES SINCE v2 [1]

The changes are too many to list in detail; at a high level:

- Code layout: the allocator moved from drivers/iommu/pisa-*.c to
  mm/pisa-*.c (and CONFIG_PISA to mm/Kconfig). Only the IOVA window and
  map/unmap code is left in drivers/iommu/pisa-map.c, and the network
  hooks moved from include/net/sock.h to include/net/pisa.h and
  net/core/pisa-net.c. See CODE LAYOUT.
- Flags renamed to __GFP_PISA and __GFP_PISA_RW
- Reworked metadata: a flat pfn-indexed table plus compact per-page
  slots, with the per-domain IOVA window indexed by slot.
  Saves one load access in the hot paths, and
  removes several tuning knobs while allowing 57bit address space
  with negligible memory overhead
- Net: the opt-in flags of the leaf devices are cached on struct
  net_device, and MSG_ZEROCOPY migration has its own per-device knob,
  pisa_tx_zerocopy.
- Confidential Computing: added arm64 Realms support.
- Compaction is driven by a target usage (pisa.target_usage_percent)
  instead of pisa.wasted_pct.
- Series reorganized (mm first, then iommu/dma, block, net, drivers), with
  the three IOVA window patches merged into one.
- Fixes for the issues reported on v2 by the automated review.
- Added NVMe and Confidential Computing results.

=== CHANGES SINCE v1 [0]

v1 presented DMA_PMD as a set of pools and an arena under drivers/iommu,
called explicitly by page_pool, the socket code and four NIC drivers.

- Renamed DMA_PMD to PISA.
- Restructured as an extension to the page allocator: callers pass
  __GFP_PISA / __GFP_PISA_RW to alloc_pages() instead of calling a
  dma_pmd_pool, and the code moved from drivers/iommu/ to mm/ (see CODE
  LAYOUT).
- Per-NUMA-node allocators with per-CPU caches replace the per-CPU and
  per-queue pools; the dma_pmd_arena for dma_alloc_attrs() is merged into
  the per-node order-0 allocator.
- The per-driver TX header bounce buffers (gve, mlx5, idpf, bnxt) are gone,
  replaced by a PISA-backed slab cache for small skb heads.
- New: user pages (MADV_PISA, MSG_ZEROCOPY) and disk I/O (O_DIRECT and
  page cache), with compaction and background migration.
- Opt-in is through per-device pisa_* attributes and pisa.* parameters;
  metadata and static keys are activated on the first opt-in.


[1] v2 (PISA)
    https://lore.kernel.org/all/20261109182533.3594984-1-lrizzo@google.com/
[0] v1 (DMA_PMD):
    https://lore.kernel.org/all/20261003212241.3432303-1-lrizzo@google.com/

Luigi Rizzo (25):
  mm: Introduce CONFIG_PISA and metadata table
  mm: Add PISA allocator lifecycle and page recycle hook
  mm: Add per-NUMA-node PISA page reservoir
  mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block
    allocation
  mm: Add global cap and shrinker for PISA pages
  iommu/dma: Map PISA memory in a per-domain IOVA window
  driver core: Add per-device pisa_* sysfs attributes for PISA optin
  iommu/dma: Use PISA for coherent DMA allocations
  mm: Add per-CPU block caches to PISA allocators
  mm: Add background page scrubber for PISA allocators
  mm: Migrate user pages into PISA
  block, mm: Support PISA for disk I/O and scrub migration
  net: Cache leaf PISA TX flags on struct net_device
  net: Use per-node PISA allocators for skb_page_frag_refill()
  net: Use PISA for page_pool memory
  mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
  net: Allocate small skb heads from PISA
  net: Migrate user pages into PISA for MSG_ZEROCOPY
  mm, iommu/dma: Support decrypted PISA pages
  arm64/mm, mm: Support decrypted PISA pages in Realms
  mm: Add PISA statistics and debugfs
  net/libeth: Use GFP_KERNEL when pre-filling RX rings
  net/gve: Use GFP_KERNEL when pre-filling RX rings
  net/gve: Use PISA memory for RX buffers
  Documentation: Add PISA architecture documentation

 .../admin-guide/kernel-parameters.txt         |    7 +
 Documentation/core-api/index.rst              |    1 +
 Documentation/core-api/pisa.rst               |  485 +++++
 arch/arm64/include/asm/pgtable.h              |   28 +-
 block/bio.c                                   |    9 +
 drivers/base/core.c                           |   78 +
 drivers/iommu/Makefile                        |    3 +
 drivers/iommu/dma-iommu.c                     |   49 +-
 drivers/iommu/iommu.c                         |    7 +-
 drivers/iommu/pisa-map.c                      |  488 +++++
 .../ethernet/google/gve/gve_buffer_mgmt_dqo.c |    8 +-
 drivers/net/ethernet/google/gve/gve_rx.c      |   40 +-
 include/linux/device.h                        |   32 +
 include/linux/gfp_types.h                     |   14 +-
 include/linux/huge_mm.h                       |    7 +
 include/linux/netdevice.h                     |    4 +
 include/linux/pisa-iommu.h                    |  141 ++
 include/linux/pisa-mm.h                       |  205 ++
 include/linux/pisa.h                          |  356 ++++
 include/linux/sched.h                         |    4 +
 include/linux/slab.h                          |   10 +
 include/linux/swap.h                          |    1 +
 include/net/libeth/rx.h                       |   11 +-
 include/net/pisa.h                            |  204 ++
 include/trace/events/mmflags.h                |   11 +-
 include/uapi/asm-generic/mman-common.h        |    2 +
 kernel/dma/direct.c                           |   11 +
 kernel/exit.c                                 |    2 +
 kernel/fork.c                                 |    2 +
 mm/Kconfig                                    |   35 +
 mm/Makefile                                   |    3 +
 mm/filemap.c                                  |   12 +-
 mm/huge_memory.c                              |   36 +-
 mm/ksm.c                                      |    3 +-
 mm/madvise.c                                  |   19 +
 mm/memory.c                                   |    8 +-
 mm/migrate.c                                  |    3 +-
 mm/page-writeback.c                           |    9 +
 mm/page_alloc.c                               |   10 +
 mm/pisa-alloc.c                               | 1758 +++++++++++++++++
 mm/pisa-kunit.c                               |  778 ++++++++
 mm/pisa-meta.c                                |  834 ++++++++
 mm/pisa-node.c                                |  567 ++++++
 mm/pisa-priv.h                                |  319 +++
 mm/pisa-user.c                                |  898 +++++++++
 mm/readahead.c                                |    3 +-
 mm/slab.h                                     |    3 +-
 mm/slab_common.c                              |    2 +-
 mm/slub.c                                     |   23 +-
 mm/swapfile.c                                 |    3 +-
 mm/userfaultfd.c                              |    7 +-
 net/core/Makefile                             |    1 +
 net/core/datagram.c                           |   17 +-
 net/core/page_pool.c                          |    3 +
 net/core/pisa-net.c                           |  264 +++
 net/core/skbuff.c                             |   33 +-
 net/core/sock.c                               |   31 +-
 net/ipv4/esp4.c                               |    7 +-
 net/ipv4/ip_output.c                          |   20 +-
 net/ipv4/raw.c                                |    6 +-
 net/ipv4/tcp.c                                |   17 +-
 net/ipv4/tcp_output.c                         |   14 +-
 net/ipv6/esp6.c                               |    7 +-
 net/ipv6/ip6_output.c                         |   20 +-
 net/ipv6/raw.c                                |    6 +-
 65 files changed, 7907 insertions(+), 92 deletions(-)
 create mode 100644 Documentation/core-api/pisa.rst
 create mode 100644 drivers/iommu/pisa-map.c
 create mode 100644 include/linux/pisa-iommu.h
 create mode 100644 include/linux/pisa-mm.h
 create mode 100644 include/linux/pisa.h
 create mode 100644 include/net/pisa.h
 create mode 100644 mm/pisa-alloc.c
 create mode 100644 mm/pisa-kunit.c
 create mode 100644 mm/pisa-meta.c
 create mode 100644 mm/pisa-node.c
 create mode 100644 mm/pisa-priv.h
 create mode 100644 mm/pisa-user.c
 create mode 100644 net/core/pisa-net.c


base-commit: 20eef29b7c8465ac98d7dd6d6f2ac07a8d8056ca
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
                     ` (25 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

PISA ("Pre-mapped IOVA Superpage Allocator") is an extension to the page
allocator: requests with __GFP_PISA return memory that makes I/O-related
operations cheaper when using DMA-FQ, DMA, or Confidential Computing
swiotlb. I/O devices can individually opt in to use PISA for specific
buffer classes (e.g. read or write data, headers, zerocopy). The
relevant kernel subsystems (mm, iommu, dma, disk I/O, page cache, the
network stack) are modified to use PISA memory for devices that opt in.

PISA returns order-N blocks backed by 2MB physically contiguous pages,
so it can use 2MB IOMMU leaves and reduce IOTLB pressure by up to 512x.
It maps memory in pre-allocated per-domain IOVA ranges, and recycles
blocks and mappings until the 2MB page is returned to the buddy
allocator. This makes most of dma map/unmap a no-op, also avoiding the
related VM exits in overcommitted VMs. In Confidential Computing VMs,
PISA pages are also decrypted, removing the need for bounce buffers.
Performance is comparable to identity mode, with isolation better than
DMA-FQ and comparable to DMA.

The design has the following main goals:
- no changes to individual device drivers (or minimal changes for NIC
  drivers that use custom RX page allocators)
- minimal changes to base code: in most cases one or a few lines
- safety: PISA memory is only used for devices that opt in, and it is
  returned to the buddy allocator only after IOMMU mappings and IOTLB
  entries have been flushed
- no dependency on struct page or struct folio
- a compile-time option to disable or enable PISA
- zero runtime overhead when no devices opt in (static keys and inline
  hooks), minimal overhead when in use (direct PFN lookup in a flat
  metadata table)
- memory overhead as low as possible (adaptive shrinking and compaction)

Devices that do not opt in keep using their native DMA/DMA-FQ mode.
See Documentation/core-api/pisa.rst, added at the end of the series.

Add a CONFIG_PISA option, enabled only on supported kernels (64bit,
!PREEMPT_RT) and introduce two metadata tables, populated on demand, to
hold per-2MB buffer state:

- pisa_pfn2slot[pfn >> PISA_ORDER] to quickly access flags and
  a compact page index (slot). Covers 57 bit address space with
  only 12KB of page tables (unbacked holes are mapped to the shared
  read-only zero page), plus 4KB for each 2GB of RAM.

- pisa_meta_array[slot] (256B per slot, 4KB of RAM per 32MB of active
  PISA buffers)

Also add a built-in KUnit test suite (CONFIG_PISA_META_KUNIT_TEST)
exercising initialization idempotency, PFN/physical round-trip lookups,
out-of-bounds physical address handling, and PISA membership toggling.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa-iommu.h |  40 +++
 include/linux/pisa.h       | 114 +++++++
 mm/Kconfig                 |  34 +++
 mm/Makefile                |   3 +
 mm/pisa-kunit.c            | 112 +++++++
 mm/pisa-meta.c             | 603 +++++++++++++++++++++++++++++++++++++
 mm/pisa-priv.h             |  61 ++++
 7 files changed, 967 insertions(+)
 create mode 100644 include/linux/pisa-iommu.h
 create mode 100644 include/linux/pisa.h
 create mode 100644 mm/pisa-kunit.c
 create mode 100644 mm/pisa-meta.c
 create mode 100644 mm/pisa-priv.h

diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
new file mode 100644
index 0000000000000..7a04523b28b4c
--- /dev/null
+++ b/include/linux/pisa-iommu.h
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* PISA page metadata, shared by mm/pisa-*.c and drivers/iommu/pisa-map.c. */
+#ifndef _LINUX_PISA_IOMMU_H
+#define _LINUX_PISA_IOMMU_H
+
+#include <linux/bitmap.h>
+#include <linux/dma-direct.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/pisa.h>
+
+struct pisa_alloc;
+
+#ifdef CONFIG_PISA
+
+/* PISA page byte size and alignment mask. */
+#define PISA_SIZE		(1UL << PISA_SHIFT)
+#define PISA_MASK		(~(PISA_SIZE - 1))
+
+/* size of struct pisa_meta (256B, power of 2) and max blocks per PISA page. */
+#define _PISA_META_SHIFT	8
+#define _PISA_META_SIZE		(1U << _PISA_META_SHIFT)
+#define _PISA_MAX_BLOCKS	512U
+
+/* Metadata for a single PISA page in @pisa_meta_array (indexed by @slot). */
+struct pisa_meta {
+	unsigned long flags;	/* PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED */
+	unsigned long base_pfn;	/* base PFN of this PISA page */
+	u32 slot;		/* dense index in @pisa_meta_array and IOVA windows */
+	struct list_head list;	/* node in @pa lists or pisa_node.pages */
+} __aligned(_PISA_META_SIZE);
+
+static_assert(sizeof(struct pisa_meta) == _PISA_META_SIZE);
+
+extern struct pisa_meta *pisa_meta_array;
+
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
+
+#endif /* CONFIG_PISA */
+#endif /* _LINUX_PISA_IOMMU_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
new file mode 100644
index 0000000000000..660329fd448fe
--- /dev/null
+++ b/include/linux/pisa.h
@@ -0,0 +1,114 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* PISA core definitions and inline hooks. See Documentation/core-api/pisa.rst. */
+#ifndef _LINUX_PISA_H
+#define _LINUX_PISA_H
+
+#include <linux/types.h>
+#include <linux/bitops.h>
+#include <linux/jump_label.h>
+#include <linux/mm.h>
+#include <linux/uaccess.h>
+
+#ifdef CONFIG_PISA
+
+/* PISA page geometry (log2 byte size and base-page order). */
+#define PISA_SHIFT		21U
+#define PISA_ORDER		(PISA_SHIFT - PAGE_SHIFT)
+
+/*
+ * Physical address bits covered by @pisa_pfn2slot. KASAN_VMALLOC populates
+ * shadow for the whole table KVA up front, so cover 49 bits (1GB of KVA) there.
+ */
+#define _PISA_PHYS_BITS		(IS_ENABLED(CONFIG_KASAN_VMALLOC) ? 49U : 57U)
+#define _PISA_MAX_FRAMES	(1UL << (_PISA_PHYS_BITS - PISA_SHIFT))
+
+/*
+ * @pisa_pfn2slot is indexed by PISA frame (pfn >> PISA_ORDER). Each u32 entry
+ * packs the page's status flags in the low _PISA_SLOT_SHIFT bits and the
+ * 1-based compact metadata slot index (into @pisa_meta_array) in the upper
+ * bits. Unbacked physical ranges point to the read-only zero page.
+ */
+#define _PISA_SLOT_SHIFT	3U	/* low bits reserved for PISA_* flags */
+#define _PISA_FLAGS_MASK	((1U << _PISA_SLOT_SHIFT) - 1)
+enum {
+	_PISA_ACTIVE,
+	_PISA_QUARANTINED,
+};
+
+#define PISA_ACTIVE		BIT(_PISA_ACTIVE)	/* set up as a PISA page, not retired yet */
+#define PISA_QUARANTINED	BIT(_PISA_QUARANTINED)	/* IOMMU unmap failed: never reuse */
+
+/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
+DECLARE_STATIC_KEY_FALSE(pisa_key);
+
+extern u32 *pisa_pfn2slot;
+
+int pisa_meta_init(void);
+
+static inline bool pisa_enabled(void)
+{
+	return static_branch_unlikely(&pisa_key);
+}
+
+static inline u32 pisa_pfn_flags(unsigned long pfn)
+{
+	unsigned long frame = pfn >> PISA_ORDER;
+	u32 val;
+
+	if (!pisa_enabled() || unlikely(frame >= _PISA_MAX_FRAMES))
+		return 0;
+	/*
+	 * A fault means that memory hotplug is replacing the page tables of
+	 * this part of @pisa_pfn2slot (break-before-make): no RAM, so no PISA
+	 * page, was there. The entry is published by smp_store_release() in
+	 * pisa_pfn2slot_update(); what a reader does with it depends on its value.
+	 */
+	__get_kernel_nofault(&val, &pisa_pfn2slot[frame], u32, fault);
+	return val;
+fault:
+	return 0;
+}
+
+/*
+ * ============================================================================
+ * Kernel hooks (every hook starts inline with a static-key check)
+ * ============================================================================
+ */
+
+/* PFN / page / physical / virtual membership checks (guarded by @pisa_key). */
+static inline bool is_pisa_pfn(unsigned long pfn)
+{
+	return pisa_pfn_flags(pfn) & PISA_ACTIVE;
+}
+
+static inline bool is_pisa_page(const struct page *page)
+{
+	return is_pisa_pfn(page_to_pfn(page));
+}
+
+static inline bool is_pisa_phys(phys_addr_t phys)
+{
+	return is_pisa_pfn(phys >> PAGE_SHIFT);
+}
+
+static inline struct page *pisa_virt_to_page(const void *addr)
+{
+	struct page *page;
+
+	if (!pisa_enabled() || !addr)
+		return NULL;
+	page = virt_to_page(addr);
+	return is_pisa_page(page) ? page : NULL;
+}
+
+#else /* !CONFIG_PISA */
+
+static inline int pisa_meta_init(void) { return -EOPNOTSUPP; }
+static inline bool pisa_enabled(void) { return false; }
+static inline bool is_pisa_pfn(unsigned long pfn) { return false; }
+static inline bool is_pisa_page(const struct page *page) { return false; }
+static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
+static inline struct page *pisa_virt_to_page(const void *addr) { return NULL; }
+
+#endif /* CONFIG_PISA */
+#endif /* _LINUX_PISA_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b5438..6e7811569d33a 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1431,6 +1431,40 @@ config IOMMU_MM_DATA
 config EXECMEM
 	bool
 
+# Pre-mapped IOVA Superpage Allocator
+config PISA
+	bool "Pre-mapped IOVA Superpage Allocator"
+	depends on (X86_64 && !UML) || (ARM64 && ARM64_4K_PAGES && PGTABLE_LEVELS >= 4)
+	depends on SPARSEMEM && !PREEMPT_RT
+	default y
+	help
+	  PISA is a specialized memory allocator with an API similar to
+	  alloc_pages(). For devices that opt-in, it provides DMA-able
+	  pages backed by 2MB physically contiguous pages, and mapped in
+	  the IOMMU maps with a single leaf PTE of the same size (or with
+	  smaller ones if the IOMMU has no such page size). Mapping, unmapping
+	  and IOTLB flushing are done lazily to reduce CPU overhead.
+	  The large mappings greatly reduce IOTLB usage. As in strict iommu
+	  mode, pages are returned to the buddy allocator only when any existing
+	  mappings have been removed and IOTLB flushed.
+	  Uses 4KB of metadata for each 32MB of PISA buffers, allocated on demand.
+
+	  The config option enforces requirements (2MB large pages and !PREEMPT_RT) at compile
+	  time. Other constraints (e.g. dma-coherent devices) are verified at runtime.
+
+	  If unsure, say Y.
+
+config PISA_META_KUNIT_TEST
+	bool "KUnit test for PISA metadata table (built-in)" if !KUNIT_ALL_TESTS
+	depends on PISA && KUNIT=y
+	default KUNIT_ALL_TESTS
+	help
+	  Builds KUnit unit tests for the sparse, per-PISA-frame metadata table,
+	  and PISA page allocators in mm/pisa-*.c, verifying metadata
+	  lookup, allocator recycling, and coherent block allocation.
+
+	  If unsure, say N.
+
 config NUMA_MEMBLKS
 	bool
 
diff --git a/mm/Makefile b/mm/Makefile
index 2a3ec53d62eef..55dff84df1555 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -146,6 +146,9 @@ obj-$(CONFIG_PAGE_REPORTING) += page_reporting.o
 obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
 obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
+obj-$(CONFIG_PISA) += pisa.o
+pisa-y := pisa-meta.o
+obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
 obj-$(CONFIG_MEM_ALLOC_PROFILING) += alloc_tag.o
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
new file mode 100644
index 0000000000000..0b0dda51e8e60
--- /dev/null
+++ b/mm/pisa-kunit.c
@@ -0,0 +1,112 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * KUnit tests for the opaque struct pisa_meta table API.
+ */
+#include <kunit/test.h>
+#include <linux/gfp.h>
+#include <linux/pisa.h>
+#include <linux/mm.h>
+
+#include "pisa-priv.h"
+
+static void test_meta_init_and_roundtrip(struct kunit *test)
+{
+	unsigned long pfn, base_pfn;
+	struct pisa_meta *m_pfn;
+	struct page *page;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	/* Second call must be idempotent. */
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+	/* Hold the whole PISA frame so that its slot cannot belong to anyone else. */
+	page = alloc_pages(GFP_KERNEL, PISA_ORDER);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+
+	base_pfn = page_to_pfn(page);
+	pfn = base_pfn + 1;
+
+	if (pisa_meta_ensure_pfn(pfn, true)) {
+		__free_pages(page, PISA_ORDER);
+		KUNIT_FAIL(test, "pisa_meta_ensure_pfn(%lu) failed", pfn);
+		return;
+	}
+	m_pfn = pisa_meta_of_pfn(pfn);
+	KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
+	pisa_slot_free(m_pfn);
+	__free_pages(page, PISA_ORDER);
+}
+
+static void test_meta_invalid_phys(struct kunit *test)
+{
+	struct pisa_meta *meta;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+	meta = pisa_meta_of_pfn(ULONG_MAX >> PAGE_SHIFT);
+	KUNIT_ASSERT_NOT_NULL(test, meta);
+	KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(meta), ULONG_MAX);
+	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(ULONG_MAX >> PAGE_SHIFT));
+	KUNIT_EXPECT_NE(test, pisa_meta_ensure_pfn(ULONG_MAX >> PAGE_SHIFT, false), 0);
+}
+
+static void pisa_meta_set_active(struct pisa_meta *meta, bool active)
+{
+	if (active)
+		pisa_meta_set_flags(meta, PISA_ACTIVE);
+	else
+		pisa_meta_clear_flags(meta, PISA_ACTIVE);
+}
+
+static void test_meta_active_toggle(struct kunit *test)
+{
+	unsigned long pfn, base_pfn;
+	struct pisa_meta *meta;
+	struct page *page;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+	page = alloc_pages(GFP_KERNEL, PISA_ORDER);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+
+	pfn = page_to_pfn(page);
+	base_pfn = ALIGN_DOWN(pfn, 1UL << PISA_ORDER);
+	/* The buddy allocator returns naturally aligned blocks. */
+	KUNIT_EXPECT_EQ(test, pfn, base_pfn);
+	if (pisa_meta_ensure_pfn(pfn, true)) {
+		__free_pages(page, PISA_ORDER);
+		KUNIT_FAIL(test, "pisa_meta_ensure_pfn(%lu) failed", pfn);
+		return;
+	}
+	KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
+	meta = pisa_meta_of_pfn(pfn);
+
+	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+	pisa_meta_set_active(meta, true);
+	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
+	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
+	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+
+	pisa_meta_set_active(meta, false);
+	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+
+	pisa_slot_free(meta);
+	__free_pages(page, PISA_ORDER);
+}
+
+static struct kunit_case pisa_meta_test_cases[] = {
+	KUNIT_CASE(test_meta_init_and_roundtrip),
+	KUNIT_CASE(test_meta_invalid_phys),
+	KUNIT_CASE(test_meta_active_toggle),
+	{}
+};
+
+static struct kunit_suite pisa_meta_test_suite = {
+	.name = "pisa_meta",
+	.test_cases = pisa_meta_test_cases,
+};
+
+kunit_test_suite(pisa_meta_test_suite);
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_LICENSE("Dual BSD/GPL");
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
new file mode 100644
index 0000000000000..734f49320047a
--- /dev/null
+++ b/mm/pisa-meta.c
@@ -0,0 +1,603 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA two-level metadata tables: @pisa_pfn2slot and @pisa_meta_array.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ *
+ * Metadata is split into two tables to keep fast-path PFN membership checks
+ * compact and cache-friendly while allocating full 256-byte per-page metadata
+ * only for active and reservoir PISA pages:
+ *
+ * 1. @pisa_pfn2slot[pfn >> PISA_ORDER] (4 bytes per PISA_SIZE physical frame):
+ *    Each u32 entry packs status flags (PISA_ACTIVE, PISA_QUARANTINED,
+ *    PISA_DECRYPTED) in the low _PISA_SLOT_SHIFT (3) bits and a 1-based compact
+ *    slot index (1 .. @pisa_nr_slots) in bits [31:3] (0 = not a PISA frame).
+ *
+ *    One PTE page of @pisa_pfn2slot holds 1,024 entries covering 2GB of
+ *    physical address space:
+ *    - At pisa_meta_init(), leaf PTEs for the entire 256GB KVA range (covering
+ *      57-bit physical space, @_PISA_MAX_FRAMES; 1GB and 49-bit with
+ *      KASAN_VMALLOC) are mapped to the shared
+ *      read-only zero page (PAGE_KERNEL_RO) using a single shared PTE table
+ *      (@pisa_zero_ptes) across PMD holes (1TB physical) and a single
+ *      shared PMD table (@pisa_zero_pmds) across 1GB PUD holes (512TB
+ *      physical). Every 2GB chunk overlapping present system RAM is backed with
+ *      a writable PTE page before @pisa_key is enabled (and on MEM_GOING_ONLINE
+ *      for hotplugged RAM).
+ *    - Any PFN < @_PISA_MAX_FRAMES (including MMIO/sparse holes) can therefore be
+ *      read directly without a chunk bitmap or runtime TLB shootdowns, and
+ *      is_pisa_pfn() reduces to a single bounds-checked u32 load.
+ *
+ * 2. @pisa_meta_array[slot] (256 bytes per slot, 16 slots per PTE page):
+ *    Indexed by the 0-based compact @slot in [0, @pisa_nr_slots), which also
+ *    serves as the page's offset, in PISA_SIZE units, in every domain's PISA IOVA window.
+ *    Its PTE page tables are preallocated at pisa_meta_init(), and backing
+ *    pages are installed lazily on first allocation of a slot in each 16-slot
+ *    chunk.
+ *
+ * Memory footprint for 57-bit physical address space and 128K PISA pages:
+ *   - @pisa_pfn2slot KVA reservation: 256GB (PUD-aligned)
+ *   - @pisa_pfn2slot page tables: 2 PTE pages shared + 1 per 1TB slice with RAM
+ *   - @pisa_pfn2slot RAM backing pages: 2KB per 1GB of present RAM (2MB / 1TB)
+ *   - @pisa_meta_array PTE tables (128K slots = 32MB KVA): 64KB
+ *   - @pisa_meta_array backing pages (on demand): 1 PTE page per 16 slots (32MB at 128K)
+ */
+
+#include <linux/bitmap.h>
+#include <linux/cache.h>
+#include <linux/cacheflush.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/memblock.h>
+#include <linux/memory.h>
+#include <linux/moduleparam.h>
+#include <linux/mm.h>
+#include <linux/mutex.h>
+#include <linux/pgalloc.h>
+#include <linux/pgtable.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/vmalloc.h>
+#include <asm/tlbflush.h>
+
+#include "pisa-priv.h"
+
+static_assert(_PISA_META_SIZE <= PAGE_SIZE);
+static_assert(PISA_SIZE <= SZ_2M);
+static_assert(PISA_BLOCKS(0) <= _PISA_MAX_BLOCKS);
+static_assert(PISA_ORDER <= MAX_PAGE_ORDER);
+static_assert(PISA_SHIFT <= SUBSECTION_SHIFT);
+
+/* log2(sizeof(u32)): 4 bytes per PISA frame in @pisa_pfn2slot. */
+#define PISA_PFN2SLOT_SHIFT		2
+/* Base-page order covered by one PTE page of @pisa_pfn2slot (1024 PISA frames, 2GB). */
+#define PISA_PFN2SLOT_CHUNK_ORDER	(PISA_ORDER + PAGE_SHIFT - PISA_PFN2SLOT_SHIFT)
+/* Total KVA size of @pisa_pfn2slot (2^36 frames * 4B = 256GB). */
+#define PISA_PFN2SLOT_SIZE		(_PISA_MAX_FRAMES << PISA_PFN2SLOT_SHIFT)
+/* Number of 256B struct pisa_meta entries per PTE page (16). */
+#define PISA_META_PER_PAGE	(PAGE_SIZE >> _PISA_META_SHIFT)
+/* Max 1-based slot index representable in the upper (32 - _PISA_SLOT_SHIFT) bits. */
+#define PISA_MAX_SLOTS		((1U << (32 - _PISA_SLOT_SHIFT)) - 1)
+
+/* Enabled by pisa_meta_init() to activate fast-path hooks. */
+DEFINE_STATIC_KEY_FALSE(pisa_key);
+EXPORT_SYMBOL(pisa_key);
+
+/*
+ * Number of compact metadata / PISA-page IOVA slots (half of physical RAM). Each
+ * PISA page holds one slot in [0, pisa_nr_slots), indexing @pisa_meta_array[slot]
+ * and mapping at slot * PISA_SIZE in each domain's PISA window.
+ */
+unsigned int pisa_nr_slots __read_mostly;
+/* One bit per slot, set while a PISA page holds the slot. */
+unsigned long *pisa_slot_bitmap __read_mostly;
+
+/* Indirection table from PISA frame (pfn >> PISA_ORDER) to (slot + 1) | flags. */
+u32 *pisa_pfn2slot __read_mostly;
+EXPORT_SYMBOL(pisa_pfn2slot);
+
+/* Compact metadata table, indexed by @slot in [0, @pisa_nr_slots). */
+struct pisa_meta *pisa_meta_array __read_mostly;
+
+/* Shared sink entry returned for unbacked or out-of-range PFNs. */
+static struct pisa_meta pisa_meta_nil __read_mostly = {
+	.base_pfn = ULONG_MAX,
+};
+
+/* Shared read-only page tables mapping unbacked @pisa_pfn2slot holes to ZERO_PAGE. */
+static pte_t *pisa_zero_ptes __read_mostly;
+static pmd_t *pisa_zero_pmds __read_mostly;
+
+/* Protects init and hotplug page-table population. */
+static DEFINE_MUTEX(pisa_meta_mutex);
+/* Protects slot allocation/release and on-demand @pisa_meta_array PTE install. */
+static DEFINE_SPINLOCK(pisa_slot_lock);
+static unsigned int pisa_slot_hint;	/* no free slot below this */
+
+/*
+ * Size the global PISA page budget (@pisa_max_pages, default 12.5% of RAM) and
+ * the compact slot / per-domain IOVA window (@pisa_nr_slots, half of RAM).
+ */
+void pisa_slots_configure(void)
+{
+	pisa_nr_slots = min_t(unsigned long, totalram_pages() >> (PISA_ORDER + 1),
+			      PISA_MAX_SLOTS);
+}
+
+/* Return metadata for @pfn, or shared sink @pisa_meta_nil (flags == 0) if unbacked. */
+struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
+{
+	u32 slot = pisa_pfn_flags(pfn) >> _PISA_SLOT_SHIFT;
+
+	if (unlikely(!slot || slot > pisa_nr_slots))
+		return &pisa_meta_nil;
+
+	return &pisa_meta_array[slot - 1];
+}
+
+/* PTE for @addr in a vmalloc metadata table, or NULL if not allocated. */
+static pte_t *pisa_meta_pte(unsigned long addr)
+{
+	pgd_t *pgd = pgd_offset_k(addr);
+	p4d_t *p4d;
+	pud_t *pud;
+	pmd_t *pmd;
+
+	if (pgd_none(pgdp_get(pgd)))
+		return NULL;
+	p4d = p4d_offset(pgd, addr);
+	if (p4d_none(p4dp_get(p4d)))
+		return NULL;
+	pud = pud_offset(p4d, addr);
+	if (pud_none(pudp_get(pud)))
+		return NULL;
+	pmd = pmd_offset(pud, addr);
+	if (pmd_none(pmdp_get(pmd)))
+		return NULL;
+	return pte_offset_kernel(pmd, addr);
+}
+
+static void pisa_zero_tables_free(void)
+{
+	if (pisa_zero_pmds)
+		pmd_free(&init_mm, pisa_zero_pmds);
+	if (pisa_zero_ptes)
+		pte_free_kernel(&init_mm, pisa_zero_ptes);
+	pisa_zero_pmds = NULL;
+	pisa_zero_ptes = NULL;
+}
+
+/*
+ * Allocate two shared page-table pages for unbacked @pisa_pfn2slot ranges:
+ * - @pisa_zero_ptes (covers one PMD of KVA / 1TB physical): 512 read-only ZERO_PAGE PTEs.
+ * - @pisa_zero_pmds (covers 1GB KVA / 512TB physical): 512 PMDs -> @pisa_zero_ptes.
+ */
+static int pisa_zero_tables_init(void)
+{
+	pisa_zero_ptes = pte_alloc_one_kernel(&init_mm);
+	pisa_zero_pmds = pmd_alloc_one(&init_mm, 0);
+	if (!pisa_zero_ptes || !pisa_zero_pmds) {
+		pisa_zero_tables_free();
+		return -ENOMEM;
+	}
+
+	for (unsigned int i = 0; i < PTRS_PER_PTE; i++)
+		set_pte(&pisa_zero_ptes[i], mk_pte(ZERO_PAGE(0), PAGE_KERNEL_RO));
+
+	for (unsigned int i = 0; i < PTRS_PER_PMD; i++)
+		pmd_populate_kernel(&init_mm, &pisa_zero_pmds[i], pisa_zero_ptes);
+	return 0;
+}
+
+/*
+ * Map the PUD-aligned 256GB KVA range of @pisa_pfn2slot to ZERO_PAGE
+ * by pointing every 1GB PUD entry at @pisa_zero_pmds.
+ */
+static int pisa_pfn2slot_init_zero(unsigned long start)
+{
+	unsigned long end = start + PISA_PFN2SLOT_SIZE, addr;
+	int ret = -ENOMEM;
+
+	for (addr = start; addr < end; addr += PUD_SIZE) {
+		pgd_t *pgd = pgd_offset_k(addr);
+		p4d_t *p4d = p4d_alloc(&init_mm, pgd, addr);
+		pud_t *pud = p4d ? pud_alloc(&init_mm, p4d, addr) : NULL;
+
+		if (!pud)
+			goto undo;
+		/* An earlier user of this vmalloc range may have left page tables. */
+		if (!pud_none(pudp_get(pud)) && !pud_free_pmd_page(pud, addr)) {
+			ret = -EBUSY;
+			goto undo;
+		}
+		pud_populate(&init_mm, pud, pisa_zero_pmds);
+	}
+	return 0;
+
+undo:
+	/* Only the shared zero tables are mapped so far: unhook them. */
+	while (addr > start) {
+		addr -= PUD_SIZE;
+		pud_clear(pud_offset(p4d_offset(pgd_offset_k(addr), addr), addr));
+	}
+	return ret;
+}
+
+/*
+ * Lookups run only with @pisa_key on. Test the key itself, not the patched
+ * branch, and only after clearing an entry: a lookup that could have cached
+ * the old entry started with the key already on.
+ */
+static bool pisa_pfn2slot_has_readers(void)
+{
+	return static_key_enabled(&pisa_key);
+}
+
+/*
+ * Unshare @pisa_zero_pmds and @pisa_zero_ptes along the page-table path to
+ * @addr so the caller can install a writable PTE page at the leaf PTE while
+ * sibling entries remain mapped to ZERO_PAGE.
+ *
+ * A valid entry is cleared before it is replaced (break-before-make, which
+ * arm64 requires) and flushed if lookups may be running. A lookup that runs
+ * in between faults, and pisa_pfn_flags() reads 0. That is right: an entry
+ * that still points to a shared zero table covers no RAM, so only lookups of
+ * addresses without RAM can hit the window. The table-level flushes are
+ * flush_tlb_all(): a range flush may leave intermediate entries cached.
+ */
+static pte_t *pisa_pfn2slot_unshare_pte(unsigned long addr)
+{
+	pgd_t *pgd = pgd_offset_k(addr);
+	p4d_t *p4d = p4d_offset(pgd, addr);
+	pud_t *pud = pud_offset(p4d, addr);
+	pmd_t *pmd;
+
+	/* Replace shared 1GB @pisa_zero_pmds with a private PMD table. */
+	if (pud_pgtable(pudp_get(pud)) == pisa_zero_pmds) {
+		pmd = pmd_alloc_one(&init_mm, addr);
+		if (!pmd)
+			return NULL;
+		for (unsigned int i = 0; i < PTRS_PER_PMD; i++)
+			pmd_populate_kernel(&init_mm, &pmd[i], pisa_zero_ptes);
+		pud_clear(pud);
+		if (pisa_pfn2slot_has_readers())
+			flush_tlb_all();
+		pud_populate(&init_mm, pud, pmd);
+	}
+
+	/* Replace the shared @pisa_zero_ptes with a private PTE table. */
+	pmd = pmd_offset(pud, addr);
+	if ((pte_t *)pmd_page_vaddr(pmdp_get(pmd)) == pisa_zero_ptes) {
+		pte_t *new_ptes = pte_alloc_one_kernel(&init_mm);
+
+		if (!new_ptes)
+			return NULL;
+		for (unsigned int i = 0; i < PTRS_PER_PTE; i++)
+			set_pte(&new_ptes[i], mk_pte(ZERO_PAGE(0), PAGE_KERNEL_RO));
+		pmd_clear(pmd);
+		if (pisa_pfn2slot_has_readers())
+			flush_tlb_all();
+		pmd_populate_kernel(&init_mm, pmd, new_ptes);
+	}
+
+	return pte_offset_kernel(pmd, addr);
+}
+
+/*
+ * Back each PTE page of @pfn2slot (covering 2GB of physical space, or
+ * 1 << PISA_PFN2SLOT_CHUNK_ORDER base pages) that overlaps [@start_pfn, @end_pfn)
+ * with a writable zeroed page on the corresponding NUMA node.
+ */
+static int pisa_pfn2slot_populate(u32 *pfn2slot, unsigned long start_pfn,
+				  unsigned long end_pfn)
+{
+	unsigned long idx, end_idx;
+
+	end_pfn = min(end_pfn, _PISA_MAX_FRAMES << PISA_ORDER);
+	if (start_pfn >= end_pfn)
+		return 0;
+
+	idx = start_pfn >> PISA_PFN2SLOT_CHUNK_ORDER;
+	end_idx = ((end_pfn - 1) >> PISA_PFN2SLOT_CHUNK_ORDER) + 1;
+	for (; idx < end_idx; idx++) {
+		unsigned long addr = (unsigned long)pfn2slot + (idx << PAGE_SHIFT);
+		unsigned long pfn = max(start_pfn, idx << PISA_PFN2SLOT_CHUNK_ORDER);
+		/* Not pfn_to_online_page(): a range that is going online is not online yet. */
+		int nid = pfn_valid(pfn) ? pfn_to_nid(pfn) : NUMA_NO_NODE;
+		pte_t *ptep = pisa_meta_pte(addr);
+		struct page *page;
+
+		if (WARN_ON_ONCE(!ptep))
+			return -EINVAL;
+		/* Already backed by a writable page from an earlier RAM range. */
+		if (pte_write(ptep_get(ptep)))
+			continue;
+
+		ptep = pisa_pfn2slot_unshare_pte(addr);
+		if (!ptep)
+			return -ENOMEM;
+
+		page = alloc_pages_node(nid, GFP_KERNEL | __GFP_ZERO, 0);
+		if (!page)
+			return -ENOMEM;
+
+		/* Break the zero-page PTE before making the new one, as above. */
+		pte_clear(&init_mm, addr, ptep);
+		if (pisa_pfn2slot_has_readers())
+			flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
+		set_pte_at(&init_mm, addr, ptep, pfn_pte(page_to_pfn(page), PAGE_KERNEL));
+		flush_cache_vmap(addr, addr + PAGE_SIZE);
+	}
+	return 0;
+}
+
+static int pisa_pfn2slot_populate_ram(unsigned long start_pfn, unsigned long nr_pages, void *arg)
+{
+	return pisa_pfn2slot_populate(arg, start_pfn, start_pfn + nr_pages);
+}
+
+/*
+ * Tear down a partially populated @pisa_pfn2slot VM area on pisa_meta_init()
+ * failure. All PUD entries are cleared before free_vm_area() so
+ * vunmap_range_noflush() does not clear PTEs inside the shared zero tables;
+ * private page tables and their writable pages are freed. The range was never
+ * accessed, so no TLB entry refers to the freed tables.
+ */
+static void pisa_pfn2slot_free(struct vm_struct *vm)
+{
+	unsigned long start = ALIGN((unsigned long)vm->addr, PUD_SIZE);
+	unsigned long end = start + PISA_PFN2SLOT_SIZE;
+
+	for (unsigned long addr = start; addr < end; addr += PUD_SIZE) {
+		pgd_t *pgd = pgd_offset_k(addr);
+		p4d_t *p4d = pgd_none(pgdp_get(pgd)) ? NULL : p4d_offset(pgd, addr);
+		pud_t *pud = (!p4d || p4d_none(p4dp_get(p4d))) ? NULL : pud_offset(p4d, addr);
+		pmd_t *pmds;
+
+		if (!pud || pud_none(pudp_get(pud)))
+			continue;
+		pmds = pud_pgtable(pudp_get(pud));
+		pud_clear(pud);
+		if (pmds == pisa_zero_pmds)
+			continue;
+
+		for (unsigned int i = 0; i < PTRS_PER_PMD; i++) {
+			pte_t *ptes = (pte_t *)pmd_page_vaddr(pmdp_get(&pmds[i]));
+
+			if (ptes == pisa_zero_ptes)
+				continue;
+			for (unsigned int j = 0; j < PTRS_PER_PTE; j++) {
+				pte_t pte = ptep_get(&ptes[j]);
+
+				if (pte_write(pte))
+					__free_page(pte_page(pte));
+			}
+			pte_free_kernel(&init_mm, ptes);
+		}
+		pmd_free(&init_mm, pmds);
+	}
+	free_vm_area(vm);
+}
+
+/* Populate @pisa_pfn2slot chunks for memory about to go online. */
+static int pisa_meta_memory_cb(struct notifier_block *nb, unsigned long action, void *arg)
+{
+	struct memory_notify *mn = arg;
+	int ret = 0;
+
+	if (action != MEM_GOING_ONLINE)
+		return NOTIFY_OK;
+
+	guard(mutex)(&pisa_meta_mutex);
+	if (pisa_pfn2slot)
+		ret = pisa_pfn2slot_populate(pisa_pfn2slot, mn->start_pfn,
+					     mn->start_pfn + mn->nr_pages);
+	return notifier_from_errno(ret);
+}
+
+static struct notifier_block pisa_meta_memory_nb = {
+	.notifier_call = pisa_meta_memory_cb,
+};
+
+/*
+ * Registered at boot so that no MEM_GOING_ONLINE event is missed around
+ * pisa_meta_init(): the callback does nothing until @pisa_pfn2slot is set, and
+ * RAM added before that is in the resource tree that pisa_meta_init() walks.
+ */
+static int __init pisa_meta_notifier_init(void)
+{
+	return register_memory_notifier(&pisa_meta_memory_nb);
+}
+core_initcall(pisa_meta_notifier_init);
+
+/*
+ * Bind @pfn's PISA frame to a compact slot in @pisa_meta_array and record
+ * (slot + 1) << _PISA_SLOT_SHIFT in @pisa_pfn2slot[pfn >> PISA_ORDER]. Backing
+ * PTE pages for @pisa_meta_array (16 slots per page) are allocated lazily on
+ * first use of a slot within each PTE page.
+ */
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block)
+{
+	gfp_t gfp = (can_block ? GFP_KERNEL : GFP_ATOMIC) | __GFP_ZERO | __GFP_NOWARN;
+	unsigned long frame = pfn >> PISA_ORDER, flags;
+	struct page *new_page = NULL;
+	struct pisa_meta *meta;
+	unsigned int slot;
+	pte_t *ptep;
+	int ret = 0;
+
+	if (!pisa_enabled() || frame >= _PISA_MAX_FRAMES)
+		return -ERANGE;
+	if (READ_ONCE(pisa_pfn2slot[frame]) >> _PISA_SLOT_SHIFT)
+		return 0;
+	/* Reject PFNs outside present RAM (chunk still mapped to ZERO_PAGE) or offline. */
+	ptep = pisa_meta_pte((unsigned long)&pisa_pfn2slot[frame]);
+	if (unlikely(!ptep || !pte_write(ptep_get(ptep)) || !pfn_to_online_page(pfn)))
+		return -ERANGE;
+
+retry:
+	spin_lock_irqsave(&pisa_slot_lock, flags);
+	if (unlikely(READ_ONCE(pisa_pfn2slot[frame]) >> _PISA_SLOT_SHIFT))
+		goto out_unlock;
+
+	slot = find_next_zero_bit(pisa_slot_bitmap, pisa_nr_slots, pisa_slot_hint);
+	if (unlikely(slot >= pisa_nr_slots)) {
+		ret = -ENOSPC;
+		goto out_unlock;
+	}
+
+	/* Lazily back the PTE page covering this 16-slot chunk of @pisa_meta_array. */
+	meta = &pisa_meta_array[slot];
+	ptep = pisa_meta_pte((unsigned long)meta);
+	if (pte_none(ptep_get(ptep))) {
+		unsigned long addr = (unsigned long)meta & PAGE_MASK;
+
+		if (!new_page) {
+			/* Allocate outside the lock, this opens a race so retry. */
+			spin_unlock_irqrestore(&pisa_slot_lock, flags);
+			new_page = alloc_pages_node(pfn_to_nid(pfn), gfp, 0);
+			if (new_page)
+				goto retry;
+			return -ENOMEM;
+		}
+		set_pte_at(&init_mm, addr, ptep, pfn_pte(page_to_pfn(new_page), PAGE_KERNEL));
+		flush_cache_vmap(addr, addr + PAGE_SIZE);
+		new_page = NULL;
+	}
+
+	memset(meta, 0, sizeof(*meta));
+	meta->base_pfn = frame << PISA_ORDER;
+	meta->slot = slot;
+	INIT_LIST_HEAD(&meta->list);
+	/* Ensure @meta is initialized before @slot is visible in @pisa_slot_bitmap. */
+	smp_wmb();
+	__set_bit(slot, pisa_slot_bitmap);
+	pisa_slot_hint = slot + 1;
+	pisa_pfn2slot_update(meta);
+out_unlock:
+	spin_unlock_irqrestore(&pisa_slot_lock, flags);
+	if (new_page)
+		__free_page(new_page);
+	return ret;
+}
+
+/* Return @meta's slot once no domain or RCU reader can reference it. */
+void pisa_slot_free(struct pisa_meta *meta)
+{
+	unsigned long frame = meta->base_pfn >> PISA_ORDER;
+	unsigned int slot = meta->slot;
+
+	if (unlikely(meta == &pisa_meta_nil || frame >= _PISA_MAX_FRAMES ||
+		     (meta->flags & PISA_QUARANTINED)))
+		return;
+
+	guard(spinlock_irqsave)(&pisa_slot_lock);
+	WRITE_ONCE(pisa_pfn2slot[frame], 0);
+	WRITE_ONCE(meta->base_pfn, ULONG_MAX);
+	__clear_bit(slot, pisa_slot_bitmap);
+	if (slot < pisa_slot_hint)
+		pisa_slot_hint = slot;
+}
+
+/* Reserve @pisa_pfn2slot and @pisa_meta_array and populate page tables. */
+int pisa_meta_init(void)
+{
+	struct vm_struct *pfn2slot_vm, *meta_vm;
+	unsigned long meta_size;
+	unsigned long *bitmap;
+	struct pisa_meta *meta;
+	u32 *pfn2slot;
+	static int pisa_init_err;
+	int ret = 0;
+
+	if (pisa_enabled())
+		return 0;
+
+	mutex_lock(&pisa_meta_mutex);
+	if (pisa_init_err) {
+		/* A previous init attempt failed. */
+		mutex_unlock(&pisa_meta_mutex);
+		return pisa_init_err;
+	}
+	if (pisa_pfn2slot) {
+		/* The previous caller completed init while we waited on the mutex. */
+		mutex_unlock(&pisa_meta_mutex);
+		goto out_enable;
+	}
+
+	pisa_slots_configure();
+	meta_size = ALIGN((unsigned long)pisa_nr_slots << _PISA_META_SHIFT, PMD_SIZE);
+
+	ret = pisa_zero_tables_init();
+	if (ret)
+		goto out_unlock;
+
+	bitmap = kvcalloc(BITS_TO_LONGS(pisa_nr_slots), sizeof(unsigned long), GFP_KERNEL);
+	if (!bitmap) {
+		ret = -ENOMEM;
+		goto out_free_bitmap;
+	}
+
+	/*
+	 * Over-allocate by PUD_SIZE / PMD_SIZE so the aligned base pointers
+	 * span an exact number of PUDs / PMDs with no unaligned fringes.
+	 */
+	pfn2slot_vm = get_vm_area(PISA_PFN2SLOT_SIZE + PUD_SIZE, VM_MAP);
+	if (!pfn2slot_vm) {
+		ret = -ENOMEM;
+		goto out_free_bitmap;
+	}
+
+	meta_vm = get_vm_area(meta_size + PMD_SIZE, VM_MAP);
+	if (!meta_vm) {
+		free_vm_area(pfn2slot_vm);
+		ret = -ENOMEM;
+		goto out_free_bitmap;
+	}
+
+	/* Map all 256GB of @pfn2slot to ZERO_PAGE, then back present RAM chunks. */
+	pfn2slot = PTR_ALIGN(pfn2slot_vm->addr, PUD_SIZE);
+	ret = pisa_pfn2slot_init_zero((unsigned long)pfn2slot);
+	if (ret) {
+		/* Nothing is mapped in either area. */
+		free_vm_area(meta_vm);
+		free_vm_area(pfn2slot_vm);
+		goto out_free_bitmap;
+	}
+
+	ret = walk_system_ram_range(0, _PISA_MAX_FRAMES << PISA_ORDER, pfn2slot,
+				    pisa_pfn2slot_populate_ram);
+	if (ret)
+		goto out_free_vms;
+
+	/* Pre-allocate PTE page tables for @pisa_meta_array (leaf PTEs stay pte_none). */
+	meta = PTR_ALIGN(meta_vm->addr, PMD_SIZE);
+	ret = apply_to_page_range(&init_mm, (unsigned long)meta, meta_size,
+				  NULL, NULL);
+	if (ret)
+		goto out_free_vms;
+
+	pisa_slot_bitmap = bitmap;
+	pisa_meta_array = meta;
+	pisa_pfn2slot = pfn2slot;
+
+	pr_info("pisa: %lu GB pfn2slot KVA, %u slots (%lu KB meta KVA)\n",
+		PISA_PFN2SLOT_SIZE >> 30, pisa_nr_slots, meta_size >> 10);
+	mutex_unlock(&pisa_meta_mutex);
+out_enable:
+	static_branch_enable(&pisa_key);
+	return 0;
+
+out_free_vms:
+	free_vm_area(meta_vm);
+	pisa_pfn2slot_free(pfn2slot_vm);
+out_free_bitmap:
+	kvfree(bitmap);
+	pisa_zero_tables_free();
+out_unlock:
+	pr_err("pisa: metadata init failed (%d)\n", ret);
+	pisa_init_err = ret;
+	mutex_unlock(&pisa_meta_mutex);
+	return ret;
+}
+EXPORT_SYMBOL(pisa_meta_init);
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
new file mode 100644
index 0000000000000..b52522cb28fcc
--- /dev/null
+++ b/mm/pisa-priv.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* PISA internals shared by mm/pisa-*.c. */
+#ifndef _MM_PISA_PRIV_H
+#define _MM_PISA_PRIV_H
+
+#include <linux/moduleparam.h>
+#include <linux/pisa-iommu.h>
+
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "pisa."
+#include <linux/spinlock.h>
+
+#ifdef CONFIG_PISA
+
+/* Number of order-@order blocks in one PISA page. */
+#define PISA_BLOCKS(order)	(1U << (PISA_ORDER - (order)))
+
+/* Publish meta->flags into the pfn2slot array. */
+static inline void pisa_pfn2slot_update(const struct pisa_meta *meta)
+{
+	u32 entry = ((meta->slot + 1) << _PISA_SLOT_SHIFT) |
+		    (READ_ONCE(meta->flags) & _PISA_FLAGS_MASK);
+
+	/* Pairs with smp_load_acquire() in pisa_pfn_flags(). */
+	smp_store_release(&pisa_pfn2slot[meta->base_pfn >> PISA_ORDER], entry);
+}
+
+/*
+ * Atomic, because pisa_quarantine_page() sets PISA_QUARANTINED concurrently.
+ * PISA_ACTIVE and PISA_DECRYPTED, the only flags read from @pisa_pfn2slot,
+ * have one writer at a time, so the copy published below cannot go backwards.
+ */
+static inline void pisa_meta_set_flags(struct pisa_meta *meta, unsigned long mask)
+{
+	set_mask_bits(&meta->flags, 0, mask);
+	pisa_pfn2slot_update(meta);
+}
+
+static inline void pisa_meta_clear_flags(struct pisa_meta *meta, unsigned long mask)
+{
+	set_mask_bits(&meta->flags, mask, 0);
+	pisa_pfn2slot_update(meta);
+}
+
+int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
+void pisa_slots_configure(void);
+
+static inline unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
+{
+	return meta->base_pfn;
+}
+
+static inline struct page *pisa_meta_to_page(const struct pisa_meta *meta)
+{
+	return pfn_to_page(pisa_meta_to_pfn(meta));
+}
+
+void pisa_slot_free(struct pisa_meta *meta);
+
+#endif /* CONFIG_PISA */
+#endif /* _MM_PISA_PRIV_H */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir Luigi Rizzo
                     ` (24 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

PISA allocators use 2MB physically contiguous memory pages from which IO
buffers are allocated. Recycling memory to the PISA allocator amortizes the
cost of mapping, unmapping and IOTLB flush. Memory is returned to the buddy
allocator only when all active IOMMU/IOTLB mappings have been flushed,
so sensitive data or code will not ever be accessible to IO devices.

Add the allocator creation (struct pisa_alloc, pisa_alloc_create()) and the
page release side, hooking into __free_pages_prepare() so that PISA
pages are returned to the allocator.

Also implement asynchronous PISA page retirement and release to the
buddy allocator via a WQ_MEM_RECLAIM worker. Retirement removes PISA
pages from allocators and destroys the mappings flushing the IOTLB, but
defers the buddy release by an RCU grace period.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa-iommu.h |  24 +++
 include/linux/pisa-mm.h    |  35 ++++
 include/linux/pisa.h       |   2 +
 mm/Makefile                |   2 +-
 mm/page_alloc.c            |   4 +
 mm/pisa-alloc.c            | 338 +++++++++++++++++++++++++++++++++++++
 mm/pisa-kunit.c            | 101 ++++++++++-
 mm/pisa-meta.c             |  71 ++++++++
 mm/pisa-node.c             |  61 +++++++
 mm/pisa-priv.h             |  88 ++++++++++
 10 files changed, 724 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/pisa-mm.h
 create mode 100644 mm/pisa-alloc.c
 create mode 100644 mm/pisa-node.c

diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index 7a04523b28b4c..098430c5f864a 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -26,8 +26,20 @@ struct pisa_alloc;
 struct pisa_meta {
 	unsigned long flags;	/* PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED */
 	unsigned long base_pfn;	/* base PFN of this PISA page */
+	u16 nr_clean;		/* zeroed available blocks in @clean_bitmap */
+	u16 nr_dirty;		/* dirty available blocks in @dirty_bitmap */
+	u8 on_list;		/* current placement in @pa (PISA_LIST_*) */
 	u32 slot;		/* dense index in @pisa_meta_array and IOVA windows */
+	struct pisa_alloc *pa;	/* NULL for reservoir pages */
 	struct list_head list;	/* node in @pa lists or pisa_node.pages */
+
+	union {
+		struct llist_node llnode; /* node in lockless pisa_free_list */
+		struct rcu_head rcu;
+	};
+
+	DECLARE_BITMAP(clean_bitmap, _PISA_MAX_BLOCKS) ____cacheline_aligned;
+	DECLARE_BITMAP(dirty_bitmap, _PISA_MAX_BLOCKS) ____cacheline_aligned;
 } __aligned(_PISA_META_SIZE);
 
 static_assert(sizeof(struct pisa_meta) == _PISA_META_SIZE);
@@ -35,6 +47,18 @@ static_assert(sizeof(struct pisa_meta) == _PISA_META_SIZE);
 extern struct pisa_meta *pisa_meta_array;
 
 struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
+bool pisa_quarantine_page(struct pisa_meta *meta);
+void pisa_schedule_reclaim(void);
 
 #endif /* CONFIG_PISA */
+
+#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
+
+void pisa_unmap_all_domains(struct pisa_meta *meta);
+
+#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
+
+static inline void pisa_unmap_all_domains(struct pisa_meta *meta) {}
+
+#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
 #endif /* _LINUX_PISA_IOMMU_H */
diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
new file mode 100644
index 0000000000000..60dae9ab6116d
--- /dev/null
+++ b/include/linux/pisa-mm.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* PISA hooks for mm and the block layer. */
+#ifndef _LINUX_PISA_MM_H
+#define _LINUX_PISA_MM_H
+
+#include <linux/pisa.h>
+
+#ifdef CONFIG_PISA
+
+bool __pisa_free_page(struct page *page, unsigned int order);
+
+/*
+ * ============================================================================
+ * MM hooks (every hook starts inline with a static-key check)
+ * ============================================================================
+ */
+
+/* Hook for __free_pages_prepare() (guarded by @pisa_key). */
+static inline bool pisa_free_page(struct page *page, unsigned int order)
+{
+	if (!pisa_enabled())
+		return false;
+	guard(rcu)();
+	return is_pisa_page(page) && __pisa_free_page(page, order);
+}
+
+#else /* !CONFIG_PISA */
+
+static inline bool pisa_free_page(struct page *page, unsigned int order)
+{
+	return false;
+}
+
+#endif /* CONFIG_PISA */
+#endif /* _LINUX_PISA_MM_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 660329fd448fe..f07432d2e6809 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -5,9 +5,11 @@
 
 #include <linux/types.h>
 #include <linux/bitops.h>
+#include <linux/dma-mapping.h>
 #include <linux/jump_label.h>
 #include <linux/mm.h>
 #include <linux/uaccess.h>
+#include <linux/rcupdate.h>
 
 #ifdef CONFIG_PISA
 
diff --git a/mm/Makefile b/mm/Makefile
index 55dff84df1555..7d624e4bad456 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,7 +147,7 @@ obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
 obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_PISA) += pisa.o
-pisa-y := pisa-meta.o
+pisa-y := pisa-meta.o pisa-alloc.o pisa-node.o
 obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2fef4092c913c..e053cc7298eab 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -17,6 +17,7 @@
 #include <linux/stddef.h>
 #include <linux/mm.h>
 #include <linux/highmem.h>
+#include <linux/pisa-mm.h>
 #include <linux/interrupt.h>
 #include <linux/jiffies.h>
 #include <linux/compiler.h>
@@ -1324,6 +1325,9 @@ static __always_inline bool __free_pages_prepare(struct page *page,
 
 	VM_BUG_ON_PAGE(PageTail(page), page);
 
+	if (unlikely(pisa_free_page(page, order)))
+		return false;
+
 	trace_mm_page_free(page, order);
 	kmsan_free_page(page, order);
 
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
new file mode 100644
index 0000000000000..557ec1ba88c78
--- /dev/null
+++ b/mm/pisa-alloc.c
@@ -0,0 +1,338 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA streaming page allocator, async page reclaim, shrinker, and debugfs.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-mapping.h>
+#include <linux/pisa.h>
+#include <linux/export.h>
+#include <linux/gfp.h>
+#include <linux/kasan.h>
+#include <linux/list.h>
+#include <linux/llist.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/mutex.h>
+#include <linux/page_table_check.h>
+#include <linux/rcupdate.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/vmstat.h>
+#include <linux/workqueue.h>
+
+#include "pisa-priv.h"
+
+/* Idle PISA pages over the allocator watermark queued locklessly for release. */
+LLIST_HEAD(pisa_free_list);
+
+/* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
+static struct workqueue_struct *pisa_wq __ro_after_init;
+
+/* Max completely idle PISA pages kept per allocator before async release (at least 1). */
+static unsigned int pisa_max_idle __read_mostly = 1;
+
+/* Grace period in ms before releasing idle PISA pages above @pisa_max_idle (0 = immediate). */
+unsigned int pisa_idle_ms __read_mostly = 200;
+module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
+
+/* All live allocators, protected by pisa_allocs_lock. */
+static LIST_HEAD(pisa_allocs);
+static DEFINE_RWLOCK(pisa_allocs_lock);
+
+/* Allocators copy pisa_max_idle at creation: update the existing ones too. */
+static int pisa_max_idle_set(const char *val, const struct kernel_param *kp)
+{
+	int ret = param_set_uint_minmax(val, kp, 1, UINT_MAX);
+	struct pisa_alloc *pa;
+
+	if (ret)
+		return ret;
+	scoped_guard(read_lock, &pisa_allocs_lock) {
+		list_for_each_entry(pa, &pisa_allocs, node) {
+			guard(spinlock_irqsave)(&pa->lock);
+			WRITE_ONCE(pa->max_idle_pages, pisa_max_idle);
+			/* Pages that the new cap makes excess start their grace period now. */
+			if (pa->num_idle_pages > pisa_max_idle)
+				WRITE_ONCE(pa->idle_jiffies, jiffies);
+		}
+	}
+	/* Release pages that a lower cap made excess (once initialized). */
+	if (pisa_enabled())
+		pisa_schedule_reclaim();
+	return 0;
+}
+
+static const struct kernel_param_ops pisa_max_idle_ops = {
+	.set = pisa_max_idle_set,
+	.get = param_get_uint,
+};
+module_param_cb(max_idle, &pisa_max_idle_ops, &pisa_max_idle, 0644);
+
+static void pisa_reclaim_work_fn(struct work_struct *work);
+
+/* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
+static DECLARE_DELAYED_WORK(pisa_reclaim_work, pisa_reclaim_work_fn);
+
+static void pisa_reclaim_work_fn(struct work_struct *work)
+{
+	unsigned long grace = msecs_to_jiffies(READ_ONCE(pisa_idle_ms));
+	unsigned long next_delay = MAX_JIFFY_OFFSET;
+	struct pisa_meta *meta, *tmp;
+	struct llist_node *node;
+	struct pisa_alloc *pa;
+
+	read_lock(&pisa_allocs_lock);
+	list_for_each_entry(pa, &pisa_allocs, node) {
+		unsigned long expire = READ_ONCE(pa->idle_jiffies) + grace;
+		unsigned long now = jiffies, flags;
+
+		if (READ_ONCE(pa->num_idle_pages) <= READ_ONCE(pa->max_idle_pages))
+			continue;
+		if (grace && time_before(now, expire)) {
+			next_delay = min(next_delay, expire - now);
+			continue;
+		}
+		spin_lock_irqsave(&pa->lock, flags);
+		while ((!grace || time_after_eq(now, pa->idle_jiffies + grace)) &&
+		       pa->num_idle_pages > pa->max_idle_pages && !list_empty(&pa->idle)) {
+			meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+			list_del_init(&meta->list);
+			WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+			llist_add(&meta->llnode, &pisa_free_list);
+			pa->num_idle_pages--;
+		}
+		spin_unlock_irqrestore(&pa->lock, flags);
+	}
+	read_unlock(&pisa_allocs_lock);
+
+	node = llist_del_all(&pisa_free_list);
+	llist_for_each_entry_safe(meta, tmp, node, llnode)
+		pisa_release_page(meta);
+
+	if (next_delay != MAX_JIFFY_OFFSET)
+		queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
+}
+
+void pisa_schedule_reclaim(void)
+{
+	struct workqueue_struct *wq = pisa_wq ?: system_wq;
+	unsigned int idle_ms = READ_ONCE(pisa_idle_ms);
+
+	if (!llist_empty(&pisa_free_list) || !idle_ms)
+		mod_delayed_work(wq, &pisa_reclaim_work, 0);
+	else
+		queue_delayed_work(wq, &pisa_reclaim_work, msecs_to_jiffies(idle_ms));
+}
+
+/* Mark @meta quarantined after an IOMMU unmap failure; return true if queued for release. */
+bool pisa_quarantine_page(struct pisa_meta *meta)
+{
+	bool released = false;
+	struct pisa_alloc *pa;
+	unsigned long flags;
+
+	set_bit(_PISA_QUARANTINED, &meta->flags);
+	pa = READ_ONCE(meta->pa);
+	if (!pa || READ_ONCE(meta->on_list) == PISA_LIST_NONE)
+		return false;
+	spin_lock_irqsave(&pa->lock, flags);
+	if (meta->pa == pa && meta->on_list != PISA_LIST_NONE) {
+		if (meta->on_list & PISA_LIST_IDLE)
+			pa->num_idle_pages--;
+		released = pisa_place_meta(meta);
+	}
+	spin_unlock_irqrestore(&pa->lock, flags);
+	return released;
+}
+
+/*
+ * pisa_free_page() in __free_pages_prepare() recycles a PISA block into
+ * its allocator before the rest of that function runs. __pisa_free_page() calls
+ * this to apply the buddy allocator's checks, and to clear the order-0 state of
+ * blocks that were user pages (see mm/pisa-user.c).
+ *
+ * Return: false if the block is in a bad state (already reported): leak it.
+ */
+static bool pisa_free_page_prepare(struct page *page, unsigned int order)
+{
+	struct folio *folio = page_folio(page);
+
+	if (!order) {
+		if (unlikely(folio_test_mlocked(folio))) {
+			__folio_clear_mlocked(folio);
+			zone_stat_mod_folio(folio, NR_MLOCK, -1);
+			count_vm_events(UNEVICTABLE_PGCLEARED, 1);
+		}
+		if (folio_test_anon(folio))
+			folio->mapping = NULL;
+	}
+	if (WARN_ON_ONCE(atomic_read(&page->_mapcount) != -1 ||
+			 page->mapping || page_ref_count(page) ||
+			 (page->flags.f & PAGE_FLAGS_CHECK_AT_FREE)))
+		return false;
+	/* Compound blocks: the checks free_tail_page_prepare() does on the first tail page. */
+	if (order && WARN_ON_ONCE(folio_large_mapcount(folio) ||
+				  atomic_read(&folio->_entire_mapcount) != -1 ||
+				  atomic_read(&folio->_pincount)))
+		return false;
+	page_table_check_free(page, order);
+	for (unsigned int i = 0; i < (1U << order); i++)
+		page_kasan_tag_reset(page + i);
+	kasan_unpoison_range(page_address(page), PAGE_SIZE << order);
+	if (!order) {
+		page_cpupid_reset_last(page);
+		page->flags.f &= ~PAGE_FLAGS_CHECK_AT_PREP;
+		page->private = 0;
+	}
+	return true;
+}
+
+/* Check for hwpoisoned block; if poisoned, warn once and leak it to keep @pa intact. */
+static bool is_pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
+{
+	for (unsigned int i = 0; i < (1U << pa->order); i++) {
+		if (unlikely(PageHWPoison(block + i))) {
+			pr_err_once("pisa: poisoned block at pfn %lu, leaking it to keep allocator %p intact\n",
+				    page_to_pfn(block), pa);
+			return true;
+		}
+	}
+	return false;
+}
+
+/**
+ * pisa_alloc_create - Create a PISA page allocator
+ * @nid: Home NUMA node for this allocator
+ * @order: Block order to dispense (<= PISA_ORDER)
+ * @dir: Supported DMA direction (DMA_BIDIRECTIONAL, DMA_TO_DEVICE, or
+ *	DMA_FROM_DEVICE); other mappings fall back to per-buffer IOMMU mappings
+ *
+ * Return: Pointer to the created allocator, or NULL on failure.
+ */
+struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir)
+{
+	struct pisa_alloc *pa;
+
+	if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY) ||
+	    order > PISA_ORDER || !valid_dma_direction(dir))
+		return NULL;
+
+	pa = kzalloc(sizeof(*pa), GFP_KERNEL);
+	if (!pa)
+		return NULL;
+
+	spin_lock_init(&pa->lock);
+	pa->order = order;
+	pa->nid = nid;
+	pisa_alloc_set_name(pa, "%ps", (void *)_RET_IP_);
+	INIT_LIST_HEAD(&pa->partial);
+	INIT_LIST_HEAD(&pa->partial_dirty);
+	INIT_LIST_HEAD(&pa->idle);
+	INIT_LIST_HEAD(&pa->full);
+	pa->idle_jiffies = jiffies;
+
+	guard(write_lock)(&pisa_allocs_lock);
+	/* Under the lock, so pisa_max_idle_set() either sees @pa or was seen here. */
+	pa->max_idle_pages = READ_ONCE(pisa_max_idle);
+	list_add_tail(&pa->node, &pisa_allocs);
+
+	return pa;
+}
+
+/**
+ * pisa_alloc_set_name - Name a PISA allocator for debugfs
+ * @pa: Allocator to name (may be NULL)
+ * @fmt: printf-style format, truncated to fit @pa->name
+ */
+void pisa_alloc_set_name(struct pisa_alloc *pa, const char *fmt, ...)
+{
+	va_list args;
+
+	if (!pa)
+		return;
+	va_start(args, fmt);
+	vsnprintf(pa->name, sizeof(pa->name), fmt, args);
+	va_end(args);
+}
+
+/* Return freed @block (at refcount 0) to @pa's bitmaps under @pa->lock. */
+static bool pisa_return_block_locked(struct pisa_alloc *pa, struct page *block)
+{
+	unsigned long pfn = page_to_pfn(block);
+	struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+	unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+	lockdep_assert_held(&pa->lock);
+
+	if (WARN_ON_ONCE(test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+		return false;
+
+	/* __pisa_free_page() zeroed it if init_on_free is set. */
+	if (want_init_on_free()) {
+		__set_bit(idx, meta->clean_bitmap);
+		meta->nr_clean++;
+	} else {
+		__set_bit(idx, meta->dirty_bitmap);
+		meta->nr_dirty++;
+	}
+	return pisa_place_meta(meta);
+}
+
+/**
+ * __pisa_free_page - Recycle a freed block back into its PISA allocator
+ * @page: Block being freed (called from __free_pages_prepare() under RCU)
+ * @order: Order the caller is freeing @page with
+ *
+ * Return: true if the page belonged to a PISA allocator and was recycled.
+ */
+bool __pisa_free_page(struct page *page, unsigned int order)
+{
+	bool should_release;
+	unsigned long pfn = page_to_pfn(page), flags;
+	struct pisa_meta *meta = pisa_meta_of_pfn(pfn);
+	struct pisa_alloc *pa = meta->pa;
+	unsigned int idx = (pfn & (PISA_BLOCKS(0) - 1)) >> pa->order;
+
+	/* Reject double frees, mismatched orders, or unaligned blocks. */
+	if (WARN_ON_ONCE(!IS_ALIGNED(pfn, 1UL << pa->order) ||
+			 order != pa->order || compound_order(page) != pa->order ||
+			 test_bit(idx, meta->clean_bitmap) || test_bit(idx, meta->dirty_bitmap)))
+		return true;
+
+	/* Never reuse or return a hwpoisoned block to the buddy allocator. */
+	if (is_pisa_block_poisoned(pa, page))
+		return true;
+
+	/* Checks skipped by the early return in __free_pages_prepare(). */
+	if (unlikely(!pisa_free_page_prepare(page, pa->order)))
+		return true;
+
+	/* KASAN/KMSAN/pgalloc_tag_sub run when pisa_release_page_rcu() frees to buddy. */
+	if (want_init_on_free())
+		memset(page_address(page), 0, PAGE_SIZE << pa->order);
+
+	spin_lock_irqsave(&pa->lock, flags);
+	should_release = pisa_return_block_locked(pa, page);
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (should_release)
+		pisa_schedule_reclaim();
+
+	return true;
+}
+
+static int __init pisa_init(void)
+{
+	/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
+	pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
+	if (!pisa_wq)
+		pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
+
+	return 0;
+}
+subsys_initcall(pisa_init);
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 0b0dda51e8e60..7436d1c9b86a6 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -3,6 +3,7 @@
  * KUnit tests for the opaque struct pisa_meta table API.
  */
 #include <kunit/test.h>
+#include <linux/delay.h>
 #include <linux/gfp.h>
 #include <linux/pisa.h>
 #include <linux/mm.h>
@@ -95,10 +96,108 @@ static void test_meta_active_toggle(struct kunit *test)
 	__free_pages(page, PISA_ORDER);
 }
 
+static void test_alloc_and_recycle(struct kunit *test)
+{
+	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, 0, DMA_NONE));
+	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, PISA_ORDER + 1,
+						  DMA_BIDIRECTIONAL));
+	pisa_alloc_set_name(NULL, "ignored");
+}
+
+/* Wait for the reclaim worker and its RCU callback to retire @meta. */
+static bool pisa_test_wait_retired(struct pisa_meta *meta)
+{
+	for (int i = 0; i < 250 && READ_ONCE(meta->pa); i++)
+		msleep(20);
+	rcu_barrier();
+	return !READ_ONCE(meta->pa);
+}
+
+/*
+ * A quarantined page serves no allocation, and once its blocks are back it is
+ * retired without giving its slot or its memory back. The state of the page is
+ * built by hand, in a private allocator that nothing else allocates from.
+ */
+static void test_quarantine(struct kunit *test)
+{
+	const unsigned int bpp = PISA_BLOCKS(0);
+	unsigned long pfn, flags;
+	struct pisa_alloc *pa;
+	struct pisa_meta *meta;
+	struct page *page;
+	bool released;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	pa = pisa_alloc_create(first_memory_node, 0, DMA_BIDIRECTIONAL);
+	KUNIT_ASSERT_NOT_NULL(test, pa);
+	pisa_alloc_set_name(pa, "kunit_quarantine");
+
+	page = alloc_pages(GFP_KERNEL, PISA_ORDER);
+	KUNIT_ASSERT_NOT_NULL(test, page);
+	pfn = page_to_pfn(page);
+	if (pisa_meta_ensure_pfn(pfn, true)) {
+		__free_pages(page, PISA_ORDER);
+		KUNIT_FAIL(test, "pisa_meta_ensure_pfn(%lu) failed", pfn);
+		return;
+	}
+	meta = pisa_meta_of_pfn(pfn);
+
+	/* A page that no allocator owns is only flagged, and keeps its slot. */
+	KUNIT_EXPECT_FALSE(test, pisa_quarantine_page(meta));
+	KUNIT_EXPECT_TRUE(test, meta->flags & PISA_QUARANTINED);
+	pisa_slot_free(meta);
+	KUNIT_EXPECT_PTR_EQ(test, pisa_meta_of_pfn(pfn), meta);
+	pisa_meta_clear_flags(meta, PISA_QUARANTINED);
+
+	/* Give the page to @pa with half of its blocks in use. */
+	spin_lock_irqsave(&pa->lock, flags);
+	meta->pa = pa;
+	meta->nr_clean = bpp / 2;
+	pisa_meta_set_flags(meta, PISA_ACTIVE);
+	pisa_place_meta(meta);
+	spin_unlock_irqrestore(&pa->lock, flags);
+	KUNIT_EXPECT_EQ(test, (unsigned int)meta->on_list, (unsigned int)PISA_LIST_CLEAN);
+	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
+
+	/* Quarantined with blocks in use: parked on @full, where nothing allocates from it. */
+	KUNIT_EXPECT_FALSE(test, pisa_quarantine_page(meta));
+	KUNIT_EXPECT_EQ(test, (unsigned int)meta->on_list, (unsigned int)PISA_LIST_FULL);
+	KUNIT_EXPECT_TRUE(test, list_empty(&pa->partial));
+	KUNIT_EXPECT_TRUE(test, list_is_singular(&pa->full));
+
+	/* Last block returned: queued for release, not kept on @idle for reuse. */
+	spin_lock_irqsave(&pa->lock, flags);
+	meta->nr_clean = bpp;
+	released = pisa_place_meta(meta);
+	spin_unlock_irqrestore(&pa->lock, flags);
+	KUNIT_ASSERT_TRUE(test, released);
+	KUNIT_EXPECT_EQ(test, (unsigned int)meta->on_list, (unsigned int)PISA_LIST_NONE);
+	KUNIT_EXPECT_TRUE(test, list_empty(&pa->full));
+	KUNIT_EXPECT_TRUE(test, list_empty(&pa->idle));
+	KUNIT_EXPECT_EQ(test, pa->num_idle_pages, 0U);
+
+	pisa_schedule_reclaim();
+	KUNIT_ASSERT_TRUE(test, pisa_test_wait_retired(meta));
+
+	/* Retired and leaked: not a PISA page any more, slot and memory still held. */
+	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+	KUNIT_EXPECT_TRUE(test, meta->flags & PISA_QUARANTINED);
+	KUNIT_EXPECT_PTR_EQ(test, pisa_meta_of_pfn(pfn), meta);
+	KUNIT_EXPECT_EQ(test, page_count(page), 1);
+
+	/* Undo the quarantine by hand, so that the test itself leaks nothing. */
+	pisa_meta_clear_flags(meta, PISA_QUARANTINED);
+	pisa_slot_free(meta);
+	KUNIT_EXPECT_PTR_NE(test, pisa_meta_of_pfn(pfn), meta);
+	__free_pages(page, PISA_ORDER);
+}
+
 static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_meta_init_and_roundtrip),
 	KUNIT_CASE(test_meta_invalid_phys),
 	KUNIT_CASE(test_meta_active_toggle),
+	KUNIT_CASE(test_alloc_and_recycle),
+	KUNIT_CASE(test_quarantine),
 	{}
 };
 
@@ -108,5 +207,5 @@ static struct kunit_suite pisa_meta_test_suite = {
 };
 
 kunit_test_suite(pisa_meta_test_suite);
-MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table");
+MODULE_DESCRIPTION("KUnit tests for struct pisa_meta table and PISA allocator");
 MODULE_LICENSE("Dual BSD/GPL");
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 734f49320047a..41ea277c60f00 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -48,6 +48,7 @@
 #include <linux/pisa.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/jiffies.h>
 #include <linux/memblock.h>
 #include <linux/memory.h>
 #include <linux/moduleparam.h>
@@ -135,6 +136,76 @@ struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
 	return &pisa_meta_array[slot - 1];
 }
 
+/*
+ * Update @meta's position on @pa's lists under @pa->lock. When all blocks
+ * become available (avail == bpp), either queues @meta to @pisa_free_list (and
+ * returns true so the caller schedules reclaim) or moves it to @pa->idle.
+ */
+bool pisa_place_meta(struct pisa_meta *meta)
+{
+	struct pisa_alloc *pa = meta->pa;
+	unsigned int avail = pisa_meta_avail(meta), bpp = PISA_BLOCKS(pa->order);
+	u8 to_list;
+
+	lockdep_assert_held(&pa->lock);
+
+	if (avail == bpp) {
+		/* All blocks free: retire immediately if quarantined or over idle cap with idle_ms == 0. */
+		if (pa->num_idle_pages >= pa->max_idle_pages)
+			WRITE_ONCE(pa->idle_jiffies, jiffies);
+		if (unlikely(meta->flags & PISA_QUARANTINED) ||
+		    (pa->num_idle_pages >= pa->max_idle_pages && !READ_ONCE(pisa_idle_ms))) {
+			WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+			list_del_init(&meta->list);
+			llist_add(&meta->llnode, &pisa_free_list);
+			return true;
+		}
+		pa->num_idle_pages++;
+		/* Clean pages go to the head of @idle, dirty pages to the tail. */
+		to_list = PISA_LIST_IDLE | (meta->nr_dirty ? PISA_LIST_SPARSE : 0);
+	} else if (!avail || unlikely(meta->flags & PISA_QUARANTINED)) {
+		/* Quarantined pages stay on pa->full until all blocks drain. */
+		to_list = PISA_LIST_FULL;
+	} else {
+		/*
+		 * Classify partial pages with hysteresis: >= 75% free is sparse
+		 * (tail of partial list, eligible to drain), <= 50% free is dense
+		 * (head of partial list, preferred for allocation). In between,
+		 * keep the current dense/sparse state.
+		 */
+		to_list = meta->nr_clean ? PISA_LIST_CLEAN : PISA_LIST_DIRTY;
+		if (avail >= bpp - (bpp >> 2))
+			to_list |= PISA_LIST_SPARSE;
+		else if (avail > (bpp >> 1))
+			to_list |= meta->on_list & PISA_LIST_SPARSE;
+	}
+
+	if (to_list != meta->on_list) {
+		struct list_head *head;
+
+		WRITE_ONCE(meta->on_list, to_list);
+		switch (to_list & PISA_LIST_MASK) {
+		case PISA_LIST_FULL:
+			head = &pa->full;
+			break;
+		case PISA_LIST_IDLE:
+			head = &pa->idle;
+			break;
+		case PISA_LIST_CLEAN:
+			head = &pa->partial;
+			break;
+		default:
+			head = &pa->partial_dirty;
+			break;
+		}
+		if (to_list & PISA_LIST_SPARSE)
+			list_move_tail(&meta->list, head);
+		else
+			list_move(&meta->list, head);
+	}
+	return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
+}
+
 /* PTE for @addr in a vmalloc metadata table, or NULL if not allocated. */
 static pte_t *pisa_meta_pte(unsigned long addr)
 {
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
new file mode 100644
index 0000000000000..fe9f37deac755
--- /dev/null
+++ b/mm/pisa-node.c
@@ -0,0 +1,61 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-NUMA-node state: PISA page reservoir and background work.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/pisa.h>
+#include <linux/mm.h>
+#include <linux/nodemask.h>
+#include <linux/rcupdate.h>
+
+#include "pisa-priv.h"
+
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+struct pisa_node pisa_nodes[MAX_NUMNODES];
+
+/*
+ * Idle PISA page release proceeds in three stages:
+ * 1. Detach onto lockless pisa_free_list and schedule pisa_reclaim_work.
+ * 2. In process context (pisa_release_page()), unmap from all IOMMU domains,
+ *    clear PISA_ACTIVE, and queue pisa_release_page_rcu() via call_rcu().
+ * 3. After an RCU grace period, pisa_release_page_rcu() unfreezes all blocks
+ *    and returns them to the buddy allocator.
+ */
+
+/* RCU callback: unfreeze and return a retired PISA page's blocks to the buddy allocator. */
+static void pisa_release_page_rcu(struct rcu_head *head)
+{
+	/* Read @meta before freeing slot/blocks; both can be reused immediately. */
+	struct pisa_meta *meta = container_of(head, struct pisa_meta, rcu);
+	struct page *page = pisa_meta_to_page(meta);
+	unsigned long flags = READ_ONCE(meta->flags);
+	unsigned int order = meta->pa->order;
+
+	WRITE_ONCE(meta->pa, NULL);
+
+	/* The page has been unmapped from every domain by pisa_release_page() */
+	pisa_slot_free(meta);
+
+	/* Possibly still mapped by a device: leak it. */
+	if (flags & PISA_QUARANTINED)
+		return;
+
+	for (unsigned int i = 0; i < PISA_BLOCKS(order); i++) {
+		struct page *block = page + (i << order);
+
+		page_ref_unfreeze(block, 1);
+		__free_pages(block, order);
+	}
+}
+
+/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
+void pisa_release_page(struct pisa_meta *meta)
+{
+	/* pisa_unmap_all_domains(meta) will be called here once IOMMU mappings are added. */
+
+	/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
+	pisa_meta_clear_flags(meta, PISA_ACTIVE);
+	call_rcu(&meta->rcu, pisa_release_page_rcu);
+}
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index b52522cb28fcc..776462a2f1ee2 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -5,16 +5,45 @@
 
 #include <linux/moduleparam.h>
 #include <linux/pisa-iommu.h>
+#include <linux/pisa-mm.h>
 
 #undef MODULE_PARAM_PREFIX
 #define MODULE_PARAM_PREFIX "pisa."
+#include <linux/nodemask.h>
 #include <linux/spinlock.h>
+#include <linux/topology.h>
+
 
 #ifdef CONFIG_PISA
 
 /* Number of order-@order blocks in one PISA page. */
 #define PISA_BLOCKS(order)	(1U << (PISA_ORDER - (order)))
 
+/*
+ * Locking (disjoint groups; locks in different groups do not nest):
+ *
+ * 1. Allocators (mm/pisa-alloc.c, mm/pisa-node.c):
+ *    pisa_allocs_lock (rwlock) -> pa->lock (spinlock)
+ *
+ * 2. Metadata tables (mm/pisa-meta.c):
+ *    pisa_meta_mutex (mutex): init and hotplug page-table population.
+ *    pisa_slot_lock (spinlock): slot allocation and @pisa_meta_array PTE install.
+ */
+
+/*
+ * Value of @meta->on_list: low bits select the target list in @pa, and
+ * PISA_LIST_SPARSE selects tail (list_move_tail) vs head (list_move).
+ */
+enum {
+	PISA_LIST_NONE		= 0,		/* not on an allocator list */
+	PISA_LIST_FULL		= BIT(0),	/* on @pa->full (0 free blocks) */
+	PISA_LIST_IDLE		= BIT(1),	/* on @pa->idle (all blocks free) */
+	PISA_LIST_CLEAN		= BIT(2),	/* on @pa->partial */
+	PISA_LIST_DIRTY		= BIT(3),	/* on @pa->partial_dirty */
+	PISA_LIST_MASK		= 0xf,
+	PISA_LIST_SPARSE	= BIT(4),	/* tail: list_move_tail() */
+};
+
 /* Publish meta->flags into the pfn2slot array. */
 static inline void pisa_pfn2slot_update(const struct pisa_meta *meta)
 {
@@ -42,6 +71,40 @@ static inline void pisa_meta_clear_flags(struct pisa_meta *meta, unsigned long m
 	pisa_pfn2slot_update(meta);
 }
 
+static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
+{
+	return meta->nr_clean + meta->nr_dirty;
+}
+
+/* Allocator managing blocks of a fixed order, carved from PISA pages. */
+struct pisa_alloc {
+	/* First cacheline: hot fields touched on block alloc and free. */
+	spinlock_t		lock;
+	u8			order;		/* block order (<= PISA_ORDER) */
+	struct list_head	partial;	/* partial pages with nr_clean > 0 */
+	struct list_head	partial_dirty;	/* partial pages with nr_clean == 0 */
+	struct list_head	idle;		/* completely free PISA pages */
+	struct list_head	full;		/* PISA pages with 0 free blocks */
+	unsigned int		num_idle_pages;
+	unsigned int		max_idle_pages;
+	unsigned long		idle_jiffies;	/* last change of @idle at/above @max_idle_pages */
+	/* Cold. */
+	int			nid;		/* home NUMA node */
+	struct list_head	node;		/* node in pisa_allocs */
+	char			name[48];
+} ____cacheline_aligned;
+
+/* Max block order for predefined per-node allocators in pisa_node.pa[][]. */
+#define PISA_MAX_ALLOC_ORDER	3U
+
+/* Per-NUMA-node PISA state. */
+struct pisa_node {
+	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
+} ____cacheline_aligned_in_smp;
+
+/* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
+extern struct pisa_node pisa_nodes[MAX_NUMNODES];
+
 int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
 void pisa_slots_configure(void);
 
@@ -57,5 +120,30 @@ static inline struct page *pisa_meta_to_page(const struct pisa_meta *meta)
 
 void pisa_slot_free(struct pisa_meta *meta);
 
+static inline int pisa_sanitize_nid(int nid)
+{
+	if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
+		nid = numa_mem_id();
+	if (!node_state(nid, N_MEMORY))
+		nid = first_memory_node;
+	return nid;
+}
+
+/* Return the predefined per-node allocator for (@nid, @order, @read_only), or NULL. */
+static inline struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool read_only)
+{
+	return order > PISA_MAX_ALLOC_ORDER ? NULL :
+	       pisa_nodes[pisa_sanitize_nid(nid)].pa[read_only][order];
+}
+
+/* pisa-alloc.c, also used by the per-node code in pisa-node.c */
+struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir);
+__printf(2, 3)
+void pisa_alloc_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+bool pisa_place_meta(struct pisa_meta *meta);
+extern struct llist_head pisa_free_list;
+extern unsigned int pisa_idle_ms;
+void pisa_release_page(struct pisa_meta *meta);
+
 #endif /* CONFIG_PISA */
 #endif /* _MM_PISA_PRIV_H */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation Luigi Rizzo
                     ` (23 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Allocating a large page may involve compacting and blocking, so it cannot
be issued in GFP_ATOMIC context.

Add a per-NUMA-node reservoir of pre-allocated and set up PISA pages,
that can serve allocations without blocking. The reservoir is refilled in
process context by a per-node work item, queued with queue_work_node()
on system_dfl_wq so each node is refilled from its own CPUs, not bound
to the CPU that caused the miss.

pisa.reservoir_max, writable at runtime, sets the number of PISA pages
per node (default 16 = 32 MB per node; 0 = disabled).

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 mm/pisa-alloc.c |   2 +
 mm/pisa-kunit.c |   7 ++
 mm/pisa-meta.c  |   1 +
 mm/pisa-node.c  | 273 +++++++++++++++++++++++++++++++++++++++++++++++-
 mm/pisa-priv.h  |  22 ++++
 5 files changed, 304 insertions(+), 1 deletion(-)

diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 557ec1ba88c78..7ba93746f6a89 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -328,6 +328,8 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 
 static int __init pisa_init(void)
 {
+	pisa_node_init();
+
 	/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
 	pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
 	if (!pisa_wq)
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 7436d1c9b86a6..bdf5184bbd52a 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -192,12 +192,19 @@ static void test_quarantine(struct kunit *test)
 	__free_pages(page, PISA_ORDER);
 }
 
+static void test_reservoir_and_scrubber(struct kunit *test)
+{
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+}
+
 static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_meta_init_and_roundtrip),
 	KUNIT_CASE(test_meta_invalid_phys),
 	KUNIT_CASE(test_meta_active_toggle),
 	KUNIT_CASE(test_alloc_and_recycle),
 	KUNIT_CASE(test_quarantine),
+	KUNIT_CASE(test_reservoir_and_scrubber),
 	{}
 };
 
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 41ea277c60f00..7cdb1c45d65c8 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -657,6 +657,7 @@ int pisa_meta_init(void)
 	mutex_unlock(&pisa_meta_mutex);
 out_enable:
 	static_branch_enable(&pisa_key);
+	pisa_schedule_reservoir();
 	return 0;
 
 out_free_vms:
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index fe9f37deac755..5b9b44186b967 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -6,14 +6,114 @@
 
 #include <linux/atomic.h>
 #include <linux/pisa.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
 #include <linux/mm.h>
+#include <linux/moduleparam.h>
 #include <linux/nodemask.h>
 #include <linux/rcupdate.h>
+#include <linux/sched.h>
+#include <linux/sched/mm.h>
+#include <linux/spinlock.h>
+#include <linux/topology.h>
+#include <linux/workqueue.h>
 
 #include "pisa-priv.h"
 
+/*
+ * Per-NUMA-node reservoir of pre-allocated, pre-zeroed, and pre-decrypted
+ * PISA pages (pisa.reservoir_max, default 16, 0 = disabled). Refilled and
+ * trimmed by each node's work item on system_dfl_wq, which also scrubs
+ * allocators on that node.
+ */
+unsigned int pisa_reservoir_max __read_mostly = 16;
+
 /* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
-struct pisa_node pisa_nodes[MAX_NUMNODES];
+struct pisa_node pisa_nodes[MAX_NUMNODES] = {
+	[0 ... MAX_NUMNODES - 1] = {
+		.next_alloc = INITIAL_JIFFIES,
+		.lock = __SPIN_LOCK_UNLOCKED(pisa_nodes.lock),
+	},
+};
+
+static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
+{
+	int ret = param_set_uint(val, kp);
+
+	/* Schedule per-node workers to refill or trim once pisa_meta_init() has run. */
+	if (pisa_enabled() && !ret)
+		pisa_schedule_reservoir();
+	return ret;
+}
+
+static const struct kernel_param_ops pisa_reservoir_max_ops = {
+	.set = pisa_reservoir_max_set,
+	.get = param_get_uint,
+};
+module_param_cb(reservoir_max, &pisa_reservoir_max_ops, &pisa_reservoir_max, 0644);
+
+/* Apply GFP_NOFS/GFP_NOIO scope around internal GFP_KERNEL allocations. */
+static unsigned int pisa_memalloc_save(gfp_t gfp)
+{
+	unsigned int flags = 0;
+
+	if (!(gfp & __GFP_FS))
+		flags |= PF_MEMALLOC_NOFS;
+	if (!(gfp & __GFP_IO))
+		flags |= PF_MEMALLOC_NOIO;
+	return memalloc_flags_save(flags);
+}
+
+/* Allocate a PISA page from the buddy allocator and initialize its metadata. */
+static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
+					   bool *clean)
+{
+	struct pisa_meta *meta;
+	unsigned int memflags;
+	bool can_reclaim_fs;
+	struct page *page;
+	int err;
+
+	gfp = current_gfp_context(gfp & gfp_allowed_mask);
+	can_reclaim_fs = can_block && (gfp & (__GFP_FS | __GFP_IO)) == (__GFP_FS | __GFP_IO);
+	if (unlikely(!pisa_enabled())) {
+		if (!can_reclaim_fs || pisa_meta_init())
+			return ERR_PTR(-EAGAIN);
+	}
+
+	/*
+	 * Long-lived active pages must be unmovable, lowmem, non-compound, and
+	 * must not dip into emergency reserves or trigger OOM/warnings.
+	 */
+	gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+	gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
+
+	if (can_block)
+		memflags = pisa_memalloc_save(gfp);
+	page = alloc_pages_node(nid, gfp, PISA_ORDER);
+	if (!page) {
+		if (can_block)
+			memalloc_flags_restore(memflags);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	/* Bind this PISA frame to a compact metadata slot. */
+	err = pisa_meta_ensure_pfn(page_to_pfn(page), can_block);
+	if (unlikely(err)) {
+		if (can_block)
+			memalloc_flags_restore(memflags);
+		__free_pages(page, PISA_ORDER);
+		return ERR_PTR(err);
+	}
+	if (can_block)
+		memalloc_flags_restore(memflags);
+
+	meta = pisa_meta_of_pfn(page_to_pfn(page));
+	*clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+
+	return meta;
+}
 
 /*
  * Idle PISA page release proceeds in three stages:
@@ -53,9 +153,180 @@ static void pisa_release_page_rcu(struct rcu_head *head)
 /* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
 void pisa_release_page(struct pisa_meta *meta)
 {
+	if (!meta->pa) {
+		struct page *page = pisa_meta_to_page(meta);
+
+		pisa_slot_free(meta);
+		__free_pages(page, PISA_ORDER);
+		return;
+	}
+
 	/* pisa_unmap_all_domains(meta) will be called here once IOMMU mappings are added. */
 
 	/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
 	pisa_meta_clear_flags(meta, PISA_ACTIVE);
 	call_rcu(&meta->rcu, pisa_release_page_rcu);
 }
+
+/* Refill node @nid's reservoir up to pisa_reservoir_max pre-zeroed PISA pages. */
+static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
+{
+	while (READ_ONCE(node->nr_pages) < READ_ONCE(pisa_reservoir_max)) {
+		struct pisa_meta *meta;
+		unsigned long flags;
+		bool clean, added;
+
+		meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+		if (IS_ERR(meta)) {
+			pisa_alloc_backoff(node, false);
+			break;
+		}
+
+		/* init_on_alloc/init_on_free pages are already zeroed. */
+		if (!clean)
+			memset(page_address(pisa_meta_to_page(meta)), 0, PISA_SIZE);
+
+		spin_lock_irqsave(&node->lock, flags);
+		/* pisa_reservoir_max may have been lowered while allocating and zeroing. */
+		added = node->nr_pages < READ_ONCE(pisa_reservoir_max);
+		if (added) {
+			list_add(&meta->list, &node->pages);
+			WRITE_ONCE(node->nr_pages, node->nr_pages + 1);
+		}
+		spin_unlock_irqrestore(&node->lock, flags);
+		if (!added) {
+			pisa_release_page(meta);
+			break;
+		}
+
+		cond_resched();
+	}
+}
+
+/* Release PISA pages in @node's reservoir above @max. */
+static void pisa_reservoir_trim_node(struct pisa_node *node, unsigned int max)
+{
+	while (READ_ONCE(node->nr_pages) > max &&
+	       READ_ONCE(node->nr_pages) > READ_ONCE(pisa_reservoir_max)) {
+		struct pisa_meta *meta;
+		unsigned long flags;
+
+		spin_lock_irqsave(&node->lock, flags);
+		meta = list_first_entry_or_null(&node->pages, struct pisa_meta, list);
+		if (meta) {
+			list_del_init(&meta->list);
+			WRITE_ONCE(node->nr_pages, node->nr_pages - 1);
+		}
+		spin_unlock_irqrestore(&node->lock, flags);
+		if (!meta)
+			break;
+
+		pisa_release_page(meta);
+		cond_resched();
+	}
+}
+
+/* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
+static void pisa_node_work_fn(struct work_struct *work)
+{
+	struct pisa_node *node = container_of(work, struct pisa_node, work);
+	int nid = node - pisa_nodes;
+
+	pisa_reservoir_trim_node(node, READ_ONCE(pisa_reservoir_max));
+	if (pisa_enabled() && READ_ONCE(pisa_reservoir_max))
+		pisa_reservoir_refill_node(node, nid);
+}
+
+/* Queue node @nid's work if its reservoir is below or above the target. */
+static void pisa_schedule_reservoir_node(int nid)
+{
+	struct pisa_node *node = &pisa_nodes[nid];
+
+	if (READ_ONCE(node->nr_pages) != READ_ONCE(pisa_reservoir_max) &&
+	    !work_pending(&node->work))
+		queue_work_node(nid, system_dfl_wq, &node->work);
+}
+
+void pisa_schedule_reservoir(void)
+{
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY)
+		pisa_schedule_reservoir_node(nid);
+}
+
+/* Pop one pre-prepared PISA page from node @nid's reservoir under @node->lock. */
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+{
+	struct pisa_node *node = &pisa_nodes[nid];
+	struct pisa_meta *meta;
+	unsigned long flags;
+	bool refill = false;
+
+	if (!READ_ONCE(node->nr_pages))
+		return NULL;
+
+	spin_lock_irqsave(&node->lock, flags);
+	meta = list_first_entry_or_null(&node->pages, struct pisa_meta, list);
+	if (meta) {
+		list_del_init(&meta->list);
+		WRITE_ONCE(node->nr_pages, node->nr_pages - 1);
+		refill = node->nr_pages <= max / 2;
+	}
+	spin_unlock_irqrestore(&node->lock, flags);
+
+	if (refill)
+		pisa_schedule_reservoir_node(nid);
+
+	return meta;
+}
+
+/*
+ * Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
+ * refill when <= half full or empty), or allocate one inline on miss.
+ */
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+{
+	bool can_block = gfpflags_allow_blocking(gfp);
+	unsigned int max = READ_ONCE(pisa_reservoir_max);
+	struct pisa_meta *meta;
+
+	if (max) {
+		meta = pisa_reservoir_pop_node(nid, max);
+		if (meta) {
+			*clean = true;
+			return meta;
+		}
+	}
+
+	/* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
+	if (!can_block && time_before(jiffies, READ_ONCE(pisa_nodes[nid].next_alloc)))
+		return ERR_PTR(-EAGAIN);
+	if (max)
+		pisa_schedule_reservoir_node(nid);
+
+	meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+	if (IS_ERR(meta))
+		pisa_alloc_backoff(&pisa_nodes[nid], can_block);
+	return meta;
+}
+
+/* Number of PISA pages held in all the reservoirs. */
+unsigned long pisa_reservoir_count(void)
+{
+	unsigned long nr = 0;
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY)
+		nr += READ_ONCE(pisa_nodes[nid].nr_pages);
+	return nr;
+}
+
+void __init pisa_node_init(void)
+{
+	for (int nid = 0; nid < MAX_NUMNODES; nid++) {
+		spin_lock_init(&pisa_nodes[nid].lock);
+		INIT_LIST_HEAD(&pisa_nodes[nid].pages);
+		INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
+	}
+}
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 776462a2f1ee2..bca35f463c3b7 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -9,9 +9,11 @@
 
 #undef MODULE_PARAM_PREFIX
 #define MODULE_PARAM_PREFIX "pisa."
+#include <linux/jiffies.h>
 #include <linux/nodemask.h>
 #include <linux/spinlock.h>
 #include <linux/topology.h>
+#include <linux/workqueue.h>
 
 
 #ifdef CONFIG_PISA
@@ -24,6 +26,7 @@
  *
  * 1. Allocators (mm/pisa-alloc.c, mm/pisa-node.c):
  *    pisa_allocs_lock (rwlock) -> pa->lock (spinlock)
+ *    pisa_nodes[].lock (spinlock): leaf lock over a node's reservoir.
  *
  * 2. Metadata tables (mm/pisa-meta.c):
  *    pisa_meta_mutex (mutex): init and hotplug page-table population.
@@ -99,7 +102,13 @@ struct pisa_alloc {
 
 /* Per-NUMA-node PISA state. */
 struct pisa_node {
+	unsigned long		next_alloc;	/* atomic alloc backoff jiffies */
 	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
+
+	spinlock_t		lock;		/* protects @pages and @nr_pages */
+	unsigned int		nr_pages;	/* entries in @pages */
+	struct list_head	pages;		/* reservoir */
+	struct work_struct	work;		/* refill/trim, on this node */
 } ____cacheline_aligned_in_smp;
 
 /* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
@@ -136,6 +145,12 @@ static inline struct pisa_alloc *pisa_node_pa(int nid, unsigned int order, bool
 	       pisa_nodes[pisa_sanitize_nid(nid)].pa[read_only][order];
 }
 
+static inline void pisa_alloc_backoff(struct pisa_node *node, bool can_block)
+{
+	if (!can_block)
+		WRITE_ONCE(node->next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
+}
+
 /* pisa-alloc.c, also used by the per-node code in pisa-node.c */
 struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir);
 __printf(2, 3)
@@ -145,5 +160,12 @@ extern struct llist_head pisa_free_list;
 extern unsigned int pisa_idle_ms;
 void pisa_release_page(struct pisa_meta *meta);
 
+/* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
+extern unsigned int pisa_reservoir_max;
+void pisa_node_init(void);
+void pisa_schedule_reservoir(void);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+unsigned long pisa_reservoir_count(void);
+
 #endif /* CONFIG_PISA */
 #endif /* _MM_PISA_PRIV_H */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (2 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages Luigi Rizzo
                     ` (22 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Add block allocation (pisa_alloc_bulk(), pisa_alloc())
and split_page_compound(), which splits a newly acquired 2MB PISA page
into blocks of the allocator's order, frozen at refcount 0.

Introduce __GFP_PISA and __GFP_PISA_RW GFP flags and hook
__alloc_frozen_pages_noprof() and alloc_pages_bulk_noprof() so callers
can allocate from per-node PISA allocators (pisa_nodes[nid].pa[ro][order])
using standard page allocation APIs.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/gfp_types.h      |  14 +-
 include/linux/pisa-mm.h        |  46 +++++
 include/trace/events/mmflags.h |  11 +-
 mm/page_alloc.c                |   6 +
 mm/pisa-alloc.c                | 342 +++++++++++++++++++++++++++++++++
 mm/pisa-kunit.c                |  69 +++++++
 mm/pisa-priv.h                 |  15 ++
 7 files changed, 499 insertions(+), 4 deletions(-)

diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h
index c1331bf0b2398..80d1371212375 100644
--- a/include/linux/gfp_types.h
+++ b/include/linux/gfp_types.h
@@ -33,7 +33,7 @@ enum {
 	___GFP_IO_BIT,
 	___GFP_FS_BIT,
 	___GFP_ZERO_BIT,
-	___GFP_UNUSED_BIT,	/* 0x200u unused */
+	___GFP_PISA_BIT,
 	___GFP_DIRECT_RECLAIM_BIT,
 	___GFP_KSWAPD_RECLAIM_BIT,
 	___GFP_WRITE_BIT,
@@ -68,7 +68,13 @@ enum {
 #define ___GFP_IO		BIT(___GFP_IO_BIT)
 #define ___GFP_FS		BIT(___GFP_FS_BIT)
 #define ___GFP_ZERO		BIT(___GFP_ZERO_BIT)
-/* 0x200u unused */
+#ifdef CONFIG_PISA
+#define ___GFP_PISA		BIT(___GFP_PISA_BIT)
+#define ___GFP_PISA_RW		(___GFP_PISA | ___GFP_WRITE)
+#else
+#define ___GFP_PISA		0
+#define ___GFP_PISA_RW		0
+#endif
 #define ___GFP_DIRECT_RECLAIM	BIT(___GFP_DIRECT_RECLAIM_BIT)
 #define ___GFP_KSWAPD_RECLAIM	BIT(___GFP_KSWAPD_RECLAIM_BIT)
 #define ___GFP_WRITE		BIT(___GFP_WRITE_BIT)
@@ -289,6 +295,10 @@ enum {
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
+/* Allocate from per-node PISA allocators (fall back to buddy on failure) */
+#define __GFP_PISA			((__force gfp_t)___GFP_PISA)
+#define __GFP_PISA_RW			((__force gfp_t)___GFP_PISA_RW)
+
 /* Room for N __GFP_FOO bits */
 #define __GFP_BITS_SHIFT ___GFP_LAST_BIT
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
index 60dae9ab6116d..d55a42397c4b3 100644
--- a/include/linux/pisa-mm.h
+++ b/include/linux/pisa-mm.h
@@ -7,6 +7,9 @@
 
 #ifdef CONFIG_PISA
 
+unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
+				      int preferred_nid, int nr_pages,
+				      struct page **page_array, bool frozen);
 bool __pisa_free_page(struct page *page, unsigned int order);
 
 /*
@@ -15,6 +18,34 @@ bool __pisa_free_page(struct page *page, unsigned int order);
  * ============================================================================
  */
 
+/* Hook for alloc_pages_bulk_noprof() (guarded by @pisa_key). */
+static inline unsigned long
+pisa_alloc_pages_bulk(gfp_t *gfp, unsigned int order, int preferred_nid,
+		      nodemask_t *nodemask, int nr_pages,
+		      struct page **page_array, bool frozen)
+{
+	unsigned long nr = 0;
+
+	if (likely(!(*gfp & __GFP_PISA)))
+		return 0;
+	if (pisa_enabled() && !nodemask)
+		nr = __pisa_alloc_pages_bulk(*gfp, order, preferred_nid,
+					     nr_pages, page_array, frozen);
+	*gfp &= ~__GFP_PISA_RW;
+	return nr;
+}
+
+/* Hook for __alloc_frozen_pages_noprof() (guarded by @pisa_key). */
+static inline struct page *
+pisa_alloc_pages(gfp_t *gfp, unsigned int order, int preferred_nid,
+		 nodemask_t *nodemask)
+{
+	struct page *page = NULL;
+
+	pisa_alloc_pages_bulk(gfp, order, preferred_nid, nodemask, 1, &page, true);
+	return page;
+}
+
 /* Hook for __free_pages_prepare() (guarded by @pisa_key). */
 static inline bool pisa_free_page(struct page *page, unsigned int order)
 {
@@ -26,6 +57,21 @@ static inline bool pisa_free_page(struct page *page, unsigned int order)
 
 #else /* !CONFIG_PISA */
 
+static inline unsigned long
+pisa_alloc_pages_bulk(gfp_t *gfp, unsigned int order, int preferred_nid,
+		      nodemask_t *nodemask, int nr_pages,
+		      struct page **page_array, bool frozen)
+{
+	return 0;
+}
+
+static inline struct page *
+pisa_alloc_pages(gfp_t *gfp, unsigned int order, int preferred_nid,
+		 nodemask_t *nodemask)
+{
+	return NULL;
+}
+
 static inline bool pisa_free_page(struct page *page, unsigned int order)
 {
 	return false;
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 3c153b3ad8450..c2b2ec99ab63c 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -54,10 +54,18 @@
 # define TRACE_GFP_FLAGS_LOCKDEP
 #endif
 
+#ifdef CONFIG_PISA
+# define TRACE_GFP_FLAGS_PISA			\
+	TRACE_GFP_EM(PISA)
+#else
+# define TRACE_GFP_FLAGS_PISA
+#endif
+
 #define TRACE_GFP_FLAGS				\
 	TRACE_GFP_FLAGS_GENERAL			\
 	TRACE_GFP_FLAGS_KASAN			\
-	TRACE_GFP_FLAGS_LOCKDEP
+	TRACE_GFP_FLAGS_LOCKDEP			\
+	TRACE_GFP_FLAGS_PISA
 
 #undef TRACE_GFP_EM
 #define TRACE_GFP_EM(a) TRACE_DEFINE_ENUM(___GFP_##a##_BIT);
@@ -65,7 +73,6 @@
 TRACE_GFP_FLAGS
 
 /* Just in case these are ever used */
-TRACE_DEFINE_ENUM(___GFP_UNUSED_BIT);
 TRACE_DEFINE_ENUM(___GFP_LAST_BIT);
 
 #define gfpflag_string(flag) {(__force unsigned long)flag, #flag}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e053cc7298eab..af1399659a163 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5228,6 +5228,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 	unsigned int zonelist_iter_cookie;
 	int nr_populated = 0, nr_account = 0;
 
+	nr_populated = pisa_alloc_pages_bulk(&gfp, 0, preferred_nid, nodemask,
+					     nr_pages, page_array, false);
 	/*
 	 * Skip populated array elements to determine if any pages need
 	 * to be allocated before disabling IRQs.
@@ -5480,6 +5482,10 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 		fastpath_alloc_flags |= ALLOC_WMARK_LOW;
 	}
 
+	page = pisa_alloc_pages(&gfp, order, preferred_nid, nodemask);
+	if (page)
+		return page;
+
 	gfp &= gfp_allowed_mask;
 	/*
 	 * Apply scoped allocation constraints. This is mainly about GFP_NOFS
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 7ba93746f6a89..e7f7ed5dca067 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -15,15 +15,21 @@
 #include <linux/kasan.h>
 #include <linux/list.h>
 #include <linux/llist.h>
+#include <linux/memcontrol.h>
 #include <linux/mm.h>
 #include <linux/moduleparam.h>
 #include <linux/mutex.h>
+#include <linux/page_owner.h>
+#include <linux/pgalloc_tag.h>
 #include <linux/page_table_check.h>
 #include <linux/rcupdate.h>
+#include <linux/sched/mm.h>
+#include <linux/skbuff.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/vmstat.h>
 #include <linux/workqueue.h>
+#include <net/sock.h>
 
 #include "pisa-priv.h"
 
@@ -283,6 +289,331 @@ static bool pisa_return_block_locked(struct pisa_alloc *pa, struct page *block)
 	return pisa_place_meta(meta);
 }
 
+static DEFINE_MUTEX(pisa_init_mutex);
+
+/* Return *@pap, creating and publishing it under @pisa_init_mutex if NULL. */
+__printf(5, 6)
+struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
+				     unsigned int order, enum dma_data_direction dir,
+				     const char *fmt, ...)
+{
+	/* Pairs with smp_store_release() below. */
+	struct pisa_alloc *pa = smp_load_acquire(pap);
+
+	if (likely(pa))
+		return pa;
+
+	guard(mutex)(&pisa_init_mutex);
+	pa = *pap;
+	if (!pa) {
+		va_list args;
+
+		pa = pisa_alloc_create(nid, order, dir);
+		if (!pa)
+			return NULL;
+		va_start(args, fmt);
+		vsnprintf(pa->name, sizeof(pa->name), fmt, args);
+		va_end(args);
+		/* Pairs with smp_load_acquire() in lockless readers. */
+		smp_store_release(pap, pa);
+	}
+	return pa;
+}
+
+/**
+ * __pisa_alloc_pages_bulk - Allocate pages from a per-node __GFP_PISA* allocator
+ * @gfp: GFP flags (must have __GFP_PISA or __GFP_PISA_RW set)
+ * @order: Page order
+ * @preferred_nid: Preferred NUMA node (or NUMA_NO_NODE)
+ * @nr_pages: Total entries in @page_array
+ * @page_array: Output array of page pointers (populated entries are skipped)
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
+				      int preferred_nid, int nr_pages,
+				      struct page **page_array, bool frozen)
+{
+	struct pisa_alloc *pa = NULL;
+
+	if (unlikely(nr_pages <= 0))
+		return 0;
+	if (!order || (gfp & __GFP_COMP))
+		pa = pisa_node_pa(preferred_nid, order, !(gfp & __GFP_WRITE));
+	return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
+}
+
+/*
+ * Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
+ * pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
+ * Fails with -EBUSY if a speculative PFN walker holds a reference to @page.
+ */
+static int split_page_compound(struct page *page, unsigned int old_order, unsigned int new_order)
+{
+	if (WARN_ON_ONCE(PageCompound(page) || !page_count(page) ||
+			 new_order > old_order ||
+			 folio_memcg_charged(page_folio(page))))
+		return -EINVAL;
+
+	/* Freeze @page[0] (1 -> 0) so all resulting pieces start at refcount 0. */
+	if (!page_ref_freeze(page, 1))
+		return -EBUSY;
+
+	if (new_order < old_order) {
+		split_page_owner(page, old_order, new_order);
+		pgalloc_tag_split(page_folio(page), old_order, new_order);
+	}
+
+	/* Subpages of non-compound pages are already initialized as order 0. */
+	if (new_order == 0)
+		return 0;
+
+	for (unsigned int i = 0; i < (1U << old_order); i += (1U << new_order))
+		prep_compound_page(page + i, new_order);
+
+	return 0;
+}
+
+/* Obtain and split a new PISA page on @nid into blocks of @order. */
+static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, int nid)
+{
+	bool can_block = gfpflags_allow_blocking(gfp), clean;
+	unsigned int bpp = PISA_BLOCKS(order);
+	struct pisa_meta *meta;
+	struct page *page;
+	int err;
+
+	/* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
+	meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+	if (IS_ERR(meta))
+		return meta;
+
+	page = pisa_meta_to_page(meta);
+
+	/* Fails with -EBUSY if a concurrent PFN walker holds a speculative ref. */
+	err = split_page_compound(page, PISA_ORDER, order);
+	if (err) {
+		pisa_release_page(meta);
+		if (err == -EBUSY)
+			pisa_alloc_backoff(&pisa_nodes[nid], can_block);
+		return ERR_PTR(err);
+	}
+
+	if (clean) {
+		bitmap_set(meta->clean_bitmap, 0, bpp);
+		meta->nr_clean = bpp;
+	} else {
+		bitmap_set(meta->dirty_bitmap, 0, bpp);
+		meta->nr_dirty = bpp;
+	}
+
+	return meta;
+}
+
+/* Reset fields normally cleared by prep_new_page()/post_alloc_hook(). */
+static void pisa_block_prep(struct page *block)
+{
+	set_page_private(block, 0);
+	clear_page_pfmemalloc(block);
+}
+
+/*
+ * Collect up to @want blocks from @meta's dirty or clean bitmap. With
+ * @unfreeze they get refcount 1; otherwise they stay at refcount 0.
+ */
+static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
+				     unsigned int want, struct page **out, bool unfreeze)
+{
+	unsigned long *bitmap = dirty ? meta->dirty_bitmap : meta->clean_bitmap;
+	u16 *countp = dirty ? &meta->nr_dirty : &meta->nr_clean;
+	unsigned int idx, got = 0, used = 0;
+	struct pisa_alloc *pa = meta->pa;
+	struct page *page, *block;
+
+	if (!*countp || !want)
+		return 0;
+
+	page = pisa_meta_to_page(meta);
+	for_each_set_bit(idx, bitmap, PISA_BLOCKS(pa->order)) {
+		block = page + (idx << pa->order);
+		__clear_bit(idx, bitmap);
+		used++;
+		if (!is_pisa_block_poisoned(pa, block)) {
+			pisa_block_prep(block);
+			if (unfreeze)
+				page_ref_unfreeze(block, 1);
+			out[got++] = block;
+		}
+
+		if (used == *countp || got == want)
+			break;
+	}
+	*countp -= used;
+	return got;
+}
+
+/* @zero: prefer PISA pages with blocks in @meta->clean_bitmap. */
+static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
+{
+	struct pisa_meta *m_dirty, *m_clean, *meta;
+
+	if (zero) {
+		/* Clean blocks first: the head of @partial, then of @idle. */
+		if (!list_empty(&pa->partial))
+			return list_first_entry(&pa->partial, struct pisa_meta, list);
+		meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
+		if (meta && meta->nr_clean) {
+			pa->num_idle_pages--;
+			WRITE_ONCE(pa->idle_jiffies, jiffies);
+			return meta;
+		}
+	}
+
+	m_dirty = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+	m_clean = list_first_entry_or_null(&pa->partial, struct pisa_meta, list);
+	if (m_dirty && m_clean) {
+		bool dirty_dense = !(m_dirty->on_list & PISA_LIST_SPARSE);
+		bool clean_dense = !(m_clean->on_list & PISA_LIST_SPARSE);
+
+		if (dirty_dense != clean_dense)
+			return dirty_dense ? m_dirty : m_clean;
+		return pisa_meta_avail(m_dirty) <= pisa_meta_avail(m_clean) ?
+		       m_dirty : m_clean;
+	}
+	if (m_dirty)
+		return m_dirty;
+	if (m_clean)
+		return m_clean;
+	if (!list_empty(&pa->idle)) {
+		meta = list_last_entry(&pa->idle, struct pisa_meta, list);
+		pa->num_idle_pages--;
+		WRITE_ONCE(pa->idle_jiffies, jiffies);
+		return meta;
+	}
+	return NULL;
+}
+
+/* Extract up to @want blocks from @meta under @pa->lock. */
+static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
+				struct page **out, bool zero, bool frozen)
+{
+	unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
+	struct pisa_alloc *pa = meta->pa;
+
+	lockdep_assert_held(&pa->lock);
+
+	if (zero) {
+		unsigned int dirty;
+
+		/* First choice are clean blocks, followed by dirty */
+		got += pisa_scan_bitmap(meta, false, want, out, !frozen);
+		dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
+		for (unsigned int i = got; i < got + dirty; i++)
+			set_page_private(out[i], 1);
+		got += dirty;
+	} else {
+		/* First choice are dirty blocks, followed by clean */
+		got += pisa_scan_bitmap(meta, true, want, out, !frozen);
+		got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+	}
+
+	used = avail_before - pisa_meta_avail(meta);
+	/* If counts disagreed with bitmaps, zero them so @meta moves to @pa->full. */
+	if (WARN_ON_ONCE(!used)) {
+		meta->nr_clean = 0;
+		meta->nr_dirty = 0;
+	}
+	pisa_place_meta(meta);
+
+	return got;
+}
+
+/* Allocate @nr_pages blocks from @pa. */
+static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+				       struct page **pages, bool zero, bool frozen)
+{
+	unsigned long n, flags;
+	struct pisa_meta *meta;
+	unsigned int want;
+
+	/* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
+	spin_lock_irqsave(&pa->lock, flags);
+	for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+		want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
+		meta = pisa_pick_meta(pa, zero);
+		if (meta)
+			continue;
+
+		/* Fallback to a new allocation */
+		spin_unlock_irqrestore(&pa->lock, flags);
+		/* With @zero, fresh dirty blocks are zeroed below. */
+		meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
+		if (IS_ERR(meta))
+			goto out;
+		spin_lock_irqsave(&pa->lock, flags);
+		meta->pa = pa;
+		/* Set PISA_ACTIVE after @meta is fully set up. */
+		pisa_meta_set_flags(meta, PISA_ACTIVE);
+	}
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+out:
+	for (unsigned long i = 0; zero && i < n; i++) {
+		/* Dirty blocks for a zeroing caller were marked with page_private = 1. */
+		if (!page_private(pages[i]))
+			continue;
+		set_page_private(pages[i], 0);
+		memset(page_address(pages[i]), 0, PAGE_SIZE << pa->order);
+		if (frozen) /* Order zeroing before caller publishes the page. */
+			smp_wmb();
+		else
+			page_ref_unfreeze(pages[i], 1);
+	}
+	return n;
+}
+
+/**
+ * pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
+ * @pa: Allocator to allocate from
+ * @gfp: GFP flags (requests with __GFP_THISNODE, __GFP_DMA, __GFP_DMA32, or
+ *	__GFP_ACCOUNT are declined)
+ * @nr_pages: Number of entries in @page_array (populated entries are skipped)
+ * @page_array: Output array of page pointers
+ * @frozen: Leave allocated pages frozen at refcount 0 if true
+ *
+ * Return: Number of populated entries at the start of @page_array.
+ */
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+			      unsigned long nr_pages, struct page **page_array, bool frozen)
+{
+	unsigned long i = 0, n, got;
+
+	/* Skip any already populated prefix, as alloc_pages_bulk_node() does. */
+	while (i < nr_pages && page_array[i])
+		i++;
+	if (unlikely(!pa || i == nr_pages ||
+		     (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+		return i;
+	gfp = current_gfp_context(gfp);
+
+	while (i < nr_pages) {
+		if (page_array[i]) {
+			i++;
+			continue;
+		}
+		/* Fill the run of @n NULL entries at @i. */
+		for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
+			;
+		got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+					want_init_on_alloc(gfp), frozen);
+		i += got;
+		if (got < n)
+			break;
+	}
+	return i;
+}
+
 /**
  * __pisa_free_page - Recycle a freed block back into its PISA allocator
  * @page: Block being freed (called from __free_pages_prepare() under RCU)
@@ -328,8 +659,19 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 
 static int __init pisa_init(void)
 {
+	int nid;
+
 	pisa_node_init();
 
+	/* Preallocate the supported per-node allocators. */
+	for_each_node_state(nid, N_MEMORY) {
+		pisa_alloc_ensure(&pisa_nodes[nid].pa[0][0], nid, 0,
+				  DMA_BIDIRECTIONAL, "pisa_o0_n%d", nid);
+		pisa_alloc_ensure(&pisa_nodes[nid].pa[1][SKB_FRAG_PAGE_ORDER],
+				  nid, SKB_FRAG_PAGE_ORDER, DMA_TO_DEVICE,
+				  "pisa_ro_o%u_n%d", SKB_FRAG_PAGE_ORDER, nid);
+	}
+
 	/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
 	pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
 	if (!pisa_wq)
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index bdf5184bbd52a..ea605cb6afe5b 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -7,6 +7,8 @@
 #include <linux/gfp.h>
 #include <linux/pisa.h>
 #include <linux/mm.h>
+#include <linux/skbuff.h>
+#include <net/sock.h>
 
 #include "pisa-priv.h"
 
@@ -33,6 +35,7 @@ static void test_meta_init_and_roundtrip(struct kunit *test)
 		return;
 	}
 	m_pfn = pisa_meta_of_pfn(pfn);
+	KUNIT_EXPECT_PTR_EQ(test, pisa_meta_of_pfn(base_pfn), m_pfn);
 	KUNIT_EXPECT_EQ(test, pisa_meta_to_pfn(m_pfn), base_pfn);
 	pisa_slot_free(m_pfn);
 	__free_pages(page, PISA_ORDER);
@@ -98,10 +101,76 @@ static void test_meta_active_toggle(struct kunit *test)
 
 static void test_alloc_and_recycle(struct kunit *test)
 {
+	const struct {
+		unsigned int order;
+		bool ro;
+	} cases[] = {
+		{ 0, false },
+		{ SKB_FRAG_PAGE_ORDER, true },
+	};
+	struct page *p1, *p2, *bulk[3];
+	struct pisa_alloc *pa;
+
 	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, 0, DMA_NONE));
 	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, PISA_ORDER + 1,
 						  DMA_BIDIRECTIONAL));
+	KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 1, false));
+	KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, 4, true));
+	KUNIT_EXPECT_NULL(test, pisa_node_pa(first_memory_node, PISA_ORDER + 1, false));
 	pisa_alloc_set_name(NULL, "ignored");
+
+	for (unsigned int i = 0; i < ARRAY_SIZE(cases); i++) {
+		unsigned int order = cases[i].order;
+
+		pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
+		KUNIT_ASSERT_NOT_NULL(test, pa);
+		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
+		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
+		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
+
+		p1 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+		KUNIT_ASSERT_NOT_NULL(test, p1);
+		KUNIT_EXPECT_TRUE(test, is_pisa_page(p1));
+
+		/* Freeing p1 returns it to the allocator; next allocation reuses index 0. */
+		__free_pages(p1, order);
+		p2 = pisa_alloc(pa, GFP_KERNEL);
+		KUNIT_ASSERT_NOT_NULL(test, p2);
+		if (!want_init_on_alloc(GFP_KERNEL) || want_init_on_free() || order == PISA_ORDER)
+			KUNIT_EXPECT_PTR_EQ(test, p1, p2);
+
+		if (order == 0) {
+			/* Bulk alloc skips already populated entries. */
+			bulk[0] = NULL;
+			bulk[1] = p2;
+			bulk[2] = NULL;
+			KUNIT_EXPECT_EQ(test,
+					pisa_alloc_bulk(pa, GFP_KERNEL, 3, bulk, false),
+					3UL);
+			KUNIT_EXPECT_PTR_EQ(test, bulk[1], p2);
+			KUNIT_EXPECT_NOT_NULL(test, bulk[0]);
+			KUNIT_EXPECT_NOT_NULL(test, bulk[2]);
+			/* Declined GFP reports populated prefix length. */
+			KUNIT_EXPECT_EQ(test,
+					pisa_alloc_bulk(pa,
+							GFP_KERNEL | __GFP_ACCOUNT,
+							3, bulk, false),
+					3UL);
+			if (bulk[0])
+				__free_pages(bulk[0], 0);
+			if (bulk[2])
+				__free_pages(bulk[2], 0);
+		}
+
+		if (p2)
+			__free_pages(p2, order);
+	}
+
+	/* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
+	p1 = alloc_page(GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, p1);
+	KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
+	__free_pages(p1, 0);
 }
 
 /* Wait for the reclaim worker and its RCU callback to retire @meta. */
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index bca35f463c3b7..46170da8a0f0d 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -155,10 +155,25 @@ static inline void pisa_alloc_backoff(struct pisa_node *node, bool can_block)
 struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir);
 __printf(2, 3)
 void pisa_alloc_set_name(struct pisa_alloc *pa, const char *fmt, ...);
+__printf(5, 6)
+struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
+				     unsigned int order, enum dma_data_direction dir,
+				     const char *fmt, ...);
 bool pisa_place_meta(struct pisa_meta *meta);
 extern struct llist_head pisa_free_list;
 extern unsigned int pisa_idle_ms;
 void pisa_release_page(struct pisa_meta *meta);
+unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
+			      unsigned long nr_pages, struct page **page_array, bool frozen);
+
+static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
+{
+	struct page *page = NULL;
+
+	if (pa)
+		pisa_alloc_bulk(pa, gfp, 1, &page, false);
+	return page;
+}
 
 /* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
 extern unsigned int pisa_reservoir_max;
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (3 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window Luigi Rizzo
                     ` (21 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Add mechanisms to bound the number of unused PISA pages:

 - a global ceiling, pisa.max_pages, on the number of PISA pages pinned
   across all allocators and reservoirs, defaulting to one eighth of RAM,
   with a floor for small machines.

 - a shrinker that, under reclaim, drains the per-node reservoirs first
   and then completely idle pages across all allocators.
   Idle pages are detached under the allocator lock but retired outside
   it, to avoid holding the lock during the expensive unmaps/iotlb flush.

 - recycle gating for caller pools (pisa_page_can_recycle()): blocks that
   belong to old, sparse partial pages are returned to their allocator
   instead of being recycled in place, so those pages can drain to idle.
   Tuned by pisa.compact, pisa.target_usage_percent and pisa.old_ms.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa-iommu.h |   3 +
 include/linux/pisa.h       |  20 +++++-
 mm/pisa-alloc.c            | 116 ++++++++++++++++++++++++++++++++++
 mm/pisa-kunit.c            |  16 +++++
 mm/pisa-meta.c             | 123 ++++++++++++++++++++++++++++++++++++-
 mm/pisa-node.c             |  33 +++++++++-
 mm/pisa-priv.h             |  38 ++++++++++++
 7 files changed, 344 insertions(+), 5 deletions(-)

diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index 098430c5f864a..b29b405804193 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -29,6 +29,7 @@ struct pisa_meta {
 	u16 nr_clean;		/* zeroed available blocks in @clean_bitmap */
 	u16 nr_dirty;		/* dirty available blocks in @dirty_bitmap */
 	u8 on_list;		/* current placement in @pa (PISA_LIST_*) */
+	bool drain;		/* sparse old page draining back to @idle */
 	u32 slot;		/* dense index in @pisa_meta_array and IOVA windows */
 	struct pisa_alloc *pa;	/* NULL for reservoir pages */
 	struct list_head list;	/* node in @pa lists or pisa_node.pages */
@@ -37,6 +38,7 @@ struct pisa_meta {
 		struct llist_node llnode; /* node in lockless pisa_free_list */
 		struct rcu_head rcu;
 	};
+	unsigned long alloc_jiffies;	/* jiffies when page entered/re-entered use */
 
 	DECLARE_BITMAP(clean_bitmap, _PISA_MAX_BLOCKS) ____cacheline_aligned;
 	DECLARE_BITMAP(dirty_bitmap, _PISA_MAX_BLOCKS) ____cacheline_aligned;
@@ -45,6 +47,7 @@ struct pisa_meta {
 static_assert(sizeof(struct pisa_meta) == _PISA_META_SIZE);
 
 extern struct pisa_meta *pisa_meta_array;
+extern unsigned int pisa_nr_slots;
 
 struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
 bool pisa_quarantine_page(struct pisa_meta *meta);
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index f07432d2e6809..3307fae1dbd66 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -40,12 +40,18 @@ enum {
 #define PISA_ACTIVE		BIT(_PISA_ACTIVE)	/* set up as a PISA page, not retired yet */
 #define PISA_QUARANTINED	BIT(_PISA_QUARANTINED)	/* IOMMU unmap failed: never reuse */
 
-/* Enabled on first pisa_meta_init() to activate fast-path hooks. */
+/*
+ * Static keys guarding all kernel hooks:
+ * - @pisa_key: enabled on first device opt-in (pisa_meta_init()).
+ * - @pisa_recycle_check_key: enabled when pisa.compact or pisa.target_usage_percent > 0.
+ */
 DECLARE_STATIC_KEY_FALSE(pisa_key);
+DECLARE_STATIC_KEY_FALSE(pisa_recycle_check_key);
 
 extern u32 *pisa_pfn2slot;
 
 int pisa_meta_init(void);
+bool __pisa_page_can_recycle(const struct page *page);
 
 static inline bool pisa_enabled(void)
 {
@@ -103,6 +109,13 @@ static inline struct page *pisa_virt_to_page(const void *addr)
 	return is_pisa_page(page) ? page : NULL;
 }
 
+/* Hook for skb_page_frag_refill() and __pisa_free_page() (@pisa_recycle_check_key). */
+static inline bool pisa_page_can_recycle(const struct page *page)
+{
+	return !static_branch_unlikely(&pisa_recycle_check_key) ||
+	       __pisa_page_can_recycle(page);
+}
+
 #else /* !CONFIG_PISA */
 
 static inline int pisa_meta_init(void) { return -EOPNOTSUPP; }
@@ -112,5 +125,10 @@ static inline bool is_pisa_page(const struct page *page) { return false; }
 static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
 static inline struct page *pisa_virt_to_page(const void *addr) { return NULL; }
 
+static inline bool pisa_page_can_recycle(const struct page *page)
+{
+	return true;
+}
+
 #endif /* CONFIG_PISA */
 #endif /* _LINUX_PISA_H */
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index e7f7ed5dca067..c09f321b07eeb 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -4,6 +4,7 @@
  * See Documentation/core-api/pisa.rst for the architecture overview.
  */
 
+#include <kunit/visibility.h>
 #include <linux/atomic.h>
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
@@ -24,6 +25,7 @@
 #include <linux/page_table_check.h>
 #include <linux/rcupdate.h>
 #include <linux/sched/mm.h>
+#include <linux/shrinker.h>
 #include <linux/skbuff.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
@@ -39,6 +41,30 @@ LLIST_HEAD(pisa_free_list);
 /* Dedicated WQ_MEM_RECLAIM workqueue, can run without allocations. */
 static struct workqueue_struct *pisa_wq __ro_after_init;
 
+/* Global count and ceiling on PISA pages across all allocators and reservoirs (0 = 1/8 RAM). */
+atomic_long_t pisa_nr_pages __cacheline_aligned_in_smp;
+unsigned long pisa_max_pages __read_mostly;
+
+static int pisa_max_pages_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned long v;
+	int ret = kstrtoul(val, 0, &v);
+
+	if (ret)
+		return ret;
+	/* Once sized at boot, 0 selects the default and the slot count caps the value. */
+	if (pisa_nr_slots)
+		v = min_t(unsigned long, v ?: pisa_default_max_pages(), pisa_nr_slots);
+	WRITE_ONCE(pisa_max_pages, v);
+	return 0;
+}
+
+static const struct kernel_param_ops pisa_max_pages_ops = {
+	.set = pisa_max_pages_set,
+	.get = param_get_ulong,
+};
+module_param_cb(max_pages, &pisa_max_pages_ops, &pisa_max_pages, 0644);
+
 /* Max completely idle PISA pages kept per allocator before async release (at least 1). */
 static unsigned int pisa_max_idle __read_mostly = 1;
 
@@ -286,6 +312,7 @@ static bool pisa_return_block_locked(struct pisa_alloc *pa, struct page *block)
 		__set_bit(idx, meta->dirty_bitmap);
 		meta->nr_dirty++;
 	}
+	pa->blocks_in_use--;
 	return pisa_place_meta(meta);
 }
 
@@ -450,6 +477,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
 			break;
 	}
 	*countp -= used;
+	pa->blocks_in_use += used;
 	return got;
 }
 
@@ -465,6 +493,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
 		meta = list_first_entry_or_null(&pa->idle, struct pisa_meta, list);
 		if (meta && meta->nr_clean) {
 			pa->num_idle_pages--;
+			WRITE_ONCE(meta->alloc_jiffies, jiffies);
 			WRITE_ONCE(pa->idle_jiffies, jiffies);
 			return meta;
 		}
@@ -488,6 +517,7 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
 	if (!list_empty(&pa->idle)) {
 		meta = list_last_entry(&pa->idle, struct pisa_meta, list);
 		pa->num_idle_pages--;
+		WRITE_ONCE(meta->alloc_jiffies, jiffies);
 		WRITE_ONCE(pa->idle_jiffies, jiffies);
 		return meta;
 	}
@@ -524,6 +554,8 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
 		meta->nr_clean = 0;
 		meta->nr_dirty = 0;
 	}
+	if (READ_ONCE(meta->drain))
+		WRITE_ONCE(meta->drain, false);
 	pisa_place_meta(meta);
 
 	return got;
@@ -553,8 +585,10 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 			goto out;
 		spin_lock_irqsave(&pa->lock, flags);
 		meta->pa = pa;
+		WRITE_ONCE(meta->alloc_jiffies, jiffies);
 		/* Set PISA_ACTIVE after @meta is fully set up. */
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
+		pa->stats.page_alloc++;
 	}
 	spin_unlock_irqrestore(&pa->lock, flags);
 
@@ -657,12 +691,83 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 	return true;
 }
 
+/* Shrinker: releases reservoir pages first, then idle allocator pages. */
+static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
+{
+	unsigned long nr = pisa_reservoir_count();
+	struct pisa_alloc *pa;
+
+	guard(read_lock)(&pisa_allocs_lock);
+	list_for_each_entry(pa, &pisa_allocs, node)
+		nr += READ_ONCE(pa->num_idle_pages);
+
+	return nr << PISA_ORDER;
+}
+
+/* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+			       unsigned int *nr, struct list_head *release_list,
+			       unsigned long nr_to_scan)
+{
+	unsigned long freed = 0;
+
+	guard(spinlock_irqsave)(lock);
+	while (freed < nr_to_scan && !list_empty(list)) {
+		/* Dirty idle pages are at the tail: keep the pre-zeroed ones at the head. */
+		struct pisa_meta *meta = list_last_entry(list, struct pisa_meta, list);
+
+		WRITE_ONCE(meta->on_list, PISA_LIST_NONE);
+		list_move(&meta->list, release_list);
+		WRITE_ONCE(*nr, *nr - 1);
+		freed += 1UL << PISA_ORDER;
+	}
+
+	return freed;
+}
+
+VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
+{
+	struct pisa_meta *meta, *tmp;
+	struct pisa_alloc *pa;
+	unsigned long freed;
+	LIST_HEAD(release_list);
+
+	freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+
+	if (freed < sc->nr_to_scan) {
+		read_lock(&pisa_allocs_lock);
+		list_for_each_entry(pa, &pisa_allocs, node) {
+			freed += pisa_shrink_list(&pa->lock, &pa->idle,
+						  &pa->num_idle_pages, &release_list,
+						  sc->nr_to_scan - freed);
+
+			if (freed >= sc->nr_to_scan)
+				break;
+		}
+		read_unlock(&pisa_allocs_lock);
+	}
+
+	/* Retiring must not run under @pa->lock. */
+	list_for_each_entry_safe(meta, tmp, &release_list, list) {
+		list_del_init(&meta->list);
+		pisa_release_page(meta);
+	}
+
+	/* Charge do_shrink_slab() for the full PISA pages reclaimed. */
+	sc->nr_scanned = freed;
+
+	return freed ?: SHRINK_STOP;
+}
+
 static int __init pisa_init(void)
 {
+	struct shrinker *shrink;
 	int nid;
 
 	pisa_node_init();
 
+	pisa_slots_configure();
+
 	/* Preallocate the supported per-node allocators. */
 	for_each_node_state(nid, N_MEMORY) {
 		pisa_alloc_ensure(&pisa_nodes[nid].pa[0][0], nid, 0,
@@ -677,6 +782,17 @@ static int __init pisa_init(void)
 	if (!pisa_wq)
 		pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
 
+	shrink = shrinker_alloc(0, "pisa");
+	if (!shrink) {
+		pr_warn("pisa: shrinker registration failed\n");
+		return 0;
+	}
+
+	shrink->count_objects = pisa_shrink_count;
+	shrink->scan_objects = pisa_shrink_scan;
+	shrink->seeks = DEFAULT_SEEKS;
+	shrinker_register(shrink);
+
 	return 0;
 }
 subsys_initcall(pisa_init);
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index ea605cb6afe5b..cd2bfbfb0f1a7 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -7,6 +7,7 @@
 #include <linux/gfp.h>
 #include <linux/pisa.h>
 #include <linux/mm.h>
+#include <linux/shrinker.h>
 #include <linux/skbuff.h>
 #include <net/sock.h>
 
@@ -108,8 +109,10 @@ static void test_alloc_and_recycle(struct kunit *test)
 		{ 0, false },
 		{ SKB_FRAG_PAGE_ORDER, true },
 	};
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
 	struct page *p1, *p2, *bulk[3];
 	struct pisa_alloc *pa;
+	unsigned long old_max;
 
 	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, 0, DMA_NONE));
 	KUNIT_EXPECT_NULL(test, pisa_alloc_create(first_memory_node, PISA_ORDER + 1,
@@ -124,6 +127,7 @@ static void test_alloc_and_recycle(struct kunit *test)
 
 		pa = pisa_node_pa(first_memory_node, order, cases[i].ro);
 		KUNIT_ASSERT_NOT_NULL(test, pa);
+		pisa_shrink_scan(NULL, &sc);
 		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_ACCOUNT));
 		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_DMA32));
 		KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL | __GFP_THISNODE));
@@ -164,6 +168,7 @@ static void test_alloc_and_recycle(struct kunit *test)
 
 		if (p2)
 			__free_pages(p2, order);
+		pisa_shrink_scan(NULL, &sc);
 	}
 
 	/* Plain alloc_page() without __GFP_PISA_* comes from the buddy allocator. */
@@ -171,6 +176,16 @@ static void test_alloc_and_recycle(struct kunit *test)
 	KUNIT_ASSERT_NOT_NULL(test, p1);
 	KUNIT_EXPECT_FALSE(test, is_pisa_page(p1));
 	__free_pages(p1, 0);
+
+	/* Global budget cap (pisa_max_pages = 0) refuses new PISA pages. */
+	pa = pisa_node_pa(first_memory_node, SKB_FRAG_PAGE_ORDER, true);
+	KUNIT_ASSERT_NOT_NULL(test, pa);
+	old_max = READ_ONCE(pisa_max_pages);
+	WRITE_ONCE(pisa_max_pages, 0);
+	WRITE_ONCE(pisa_reservoir_max, 0);
+	KUNIT_EXPECT_NULL(test, pisa_alloc(pa, GFP_KERNEL));
+	WRITE_ONCE(pisa_reservoir_max, 16);
+	WRITE_ONCE(pisa_max_pages, old_max);
 }
 
 /* Wait for the reclaim worker and its RCU callback to retire @meta. */
@@ -221,6 +236,7 @@ static void test_quarantine(struct kunit *test)
 	/* Give the page to @pa with half of its blocks in use. */
 	spin_lock_irqsave(&pa->lock, flags);
 	meta->pa = pa;
+	pa->stats.page_alloc++;	/* balanced by page_free when the page is retired */
 	meta->nr_clean = bpp / 2;
 	pisa_meta_set_flags(meta, PISA_ACTIVE);
 	pisa_place_meta(meta);
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 7cdb1c45d65c8..30f0f2f6ef628 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -84,6 +84,41 @@ static_assert(PISA_SHIFT <= SUBSECTION_SHIFT);
 DEFINE_STATIC_KEY_FALSE(pisa_key);
 EXPORT_SYMBOL(pisa_key);
 
+static unsigned int pisa_compact __read_mostly;
+static unsigned int pisa_target_usage_percent __read_mostly;
+static unsigned int pisa_old_ms __read_mostly;
+
+/* Enabled when pisa.compact > 0 or pisa.target_usage_percent > 0. */
+DEFINE_STATIC_KEY_FALSE(pisa_recycle_check_key);
+EXPORT_SYMBOL(pisa_recycle_check_key);
+
+static int pisa_recycle_param_set(const char *val, const struct kernel_param *kp)
+{
+	int ret;
+
+	ret = (kp->arg == &pisa_target_usage_percent) ?
+	      param_set_uint_minmax(val, kp, 0, 100) : param_set_uint(val, kp);
+
+	if (!ret) {
+		if (pisa_compact || pisa_target_usage_percent)
+			static_branch_enable(&pisa_recycle_check_key);
+		else
+			static_branch_disable(&pisa_recycle_check_key);
+	}
+	return ret;
+}
+
+static const struct kernel_param_ops pisa_recycle_param_ops = {
+	.set = pisa_recycle_param_set,
+	.get = param_get_uint,
+};
+
+/* Stop recycling partial-page blocks when free blocks exceed @pisa_compact PISA pages. */
+module_param_cb(compact, &pisa_recycle_param_ops, &pisa_compact, 0644);
+/* Stop recycling partial pages older than @pisa_old_ms when usage < @pisa_target_usage_percent%. */
+module_param_cb(target_usage_percent, &pisa_recycle_param_ops, &pisa_target_usage_percent, 0644);
+module_param_cb(old_ms, &pisa_recycle_param_ops, &pisa_old_ms, 0644);
+
 /*
  * Number of compact metadata / PISA-page IOVA slots (half of physical RAM). Each
  * PISA page holds one slot in [0, pisa_nr_slots), indexing @pisa_meta_array[slot]
@@ -121,8 +156,13 @@ static unsigned int pisa_slot_hint;	/* no free slot below this */
  */
 void pisa_slots_configure(void)
 {
-	pisa_nr_slots = min_t(unsigned long, totalram_pages() >> (PISA_ORDER + 1),
-			      PISA_MAX_SLOTS);
+	unsigned long max_slots = min_t(unsigned long, totalram_pages() >> (PISA_ORDER + 1),
+					PISA_MAX_SLOTS);
+
+	if (!pisa_max_pages)
+		pisa_max_pages = pisa_default_max_pages();
+	pisa_max_pages = min(pisa_max_pages, max_slots);
+	pisa_nr_slots = max_slots;
 }
 
 /* Return metadata for @pfn, or shared sink @pisa_meta_nil (flags == 0) if unbacked. */
@@ -180,6 +220,10 @@ bool pisa_place_meta(struct pisa_meta *meta)
 			to_list |= meta->on_list & PISA_LIST_SPARSE;
 	}
 
+	/* Done draining once idle or dense (on @idle, PISA_LIST_SPARSE only means dirty). */
+	if ((avail == bpp || !(to_list & PISA_LIST_SPARSE)) && READ_ONCE(meta->drain))
+		WRITE_ONCE(meta->drain, false);
+
 	if (to_list != meta->on_list) {
 		struct list_head *head;
 
@@ -206,6 +250,81 @@ bool pisa_place_meta(struct pisa_meta *meta)
 	return avail == bpp && pa->num_idle_pages > pa->max_idle_pages;
 }
 
+/*
+ * __pisa_page_can_recycle - Check if a block can stay cached in a caller pool
+ * @page: Block head page
+ *
+ * Called on the recycle fast path (per-CPU cache, TX page_frag) while the
+ * caller still owns @page (so its PISA page cannot be retired).
+ *
+ * Allow recycling when:
+ * - @meta is dense (not on a sparse partial list),
+ * - @meta is not already marked @drain and was recently allocated (within
+ *   @pisa_old_ms), or
+ * - unused blocks in @pa are within @pisa_compact * bpp or used capacity is
+ *   at least @pisa_target_usage_percent%.
+ * Otherwise mark @meta->drain and refuse recycling until @meta drains to idle
+ * or becomes dense again on allocation.
+ *
+ * Return: true if @page can be recycled in place, false if it should be
+ * released back to its allocator so sparse PISA pages can drain.
+ */
+static bool pisa_meta_can_recycle(struct pisa_meta *meta)
+{
+	unsigned int compact, target_pct, old_ms;
+	unsigned long bpp, in_use, free_blocks, max_age;
+	const struct pisa_alloc *pa;
+	long held;
+
+	/* Let a quarantined page drain: never recycle its blocks in place. */
+	if (unlikely(READ_ONCE(meta->flags) & PISA_QUARANTINED))
+		return false;
+
+	if (!(READ_ONCE(meta->on_list) & PISA_LIST_SPARSE))
+		return true;
+
+	if (READ_ONCE(meta->drain))
+		return false;
+
+	old_ms = READ_ONCE(pisa_old_ms);
+	if (old_ms) {
+		max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+		if (!time_after(jiffies, READ_ONCE(meta->alloc_jiffies) + max_age))
+			return true;
+	}
+
+	pa = READ_ONCE(meta->pa);
+	if (!pa || READ_ONCE(meta->list.prev) == &pa->partial ||
+	    READ_ONCE(meta->list.prev) == &pa->partial_dirty)
+		return true;
+
+	bpp = PISA_BLOCKS(pa->order);
+	held = pisa_alloc_held_pages(pa);
+	in_use = READ_ONCE(pa->blocks_in_use);
+	free_blocks = held > 0 && (unsigned long)held * bpp > in_use ?
+		      (unsigned long)held * bpp - in_use : 0;
+
+	compact = READ_ONCE(pisa_compact);
+	target_pct = READ_ONCE(pisa_target_usage_percent);
+	if ((!compact && !target_pct) ||
+	    (compact && free_blocks <= (unsigned long)compact * bpp) ||
+	    (target_pct && in_use * 100 >= (unsigned long)held * bpp * target_pct)) {
+		WRITE_ONCE(meta->alloc_jiffies, jiffies);
+		return true;
+	}
+
+	WRITE_ONCE(meta->drain, true);
+	return false;
+}
+
+bool __pisa_page_can_recycle(const struct page *page)
+{
+	unsigned long pfn = page_to_pfn(page);
+
+	return !is_pisa_pfn(pfn) || pisa_meta_can_recycle(pisa_meta_of_pfn(pfn));
+}
+EXPORT_SYMBOL(__pisa_page_can_recycle);
+
 /* PTE for @addr in a vmalloc metadata table, or NULL if not allocated. */
 static pte_t *pisa_meta_pte(unsigned long addr)
 {
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index 5b9b44186b967..29f6ba866b633 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -82,6 +82,10 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
 			return ERR_PTR(-EAGAIN);
 	}
 
+	/* Racy check against global ceiling across allocators and reservoirs. */
+	if (atomic_long_read(&pisa_nr_pages) >= READ_ONCE(pisa_max_pages))
+		return ERR_PTR(-ENOSPC);
+
 	/*
 	 * Long-lived active pages must be unmovable, lowmem, non-compound, and
 	 * must not dip into emergency reserves or trigger OOM/warnings.
@@ -110,6 +114,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
 		memalloc_flags_restore(memflags);
 
 	meta = pisa_meta_of_pfn(page_to_pfn(page));
+
+	atomic_long_inc(&pisa_nr_pages);
 	*clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
 
 	return meta;
@@ -138,10 +144,10 @@ static void pisa_release_page_rcu(struct rcu_head *head)
 	/* The page has been unmapped from every domain by pisa_release_page() */
 	pisa_slot_free(meta);
 
-	/* Possibly still mapped by a device: leak it. */
+	/* Possibly still mapped by a device: leak it, still charged. */
 	if (flags & PISA_QUARANTINED)
 		return;
-
+	atomic_long_dec(&pisa_nr_pages);
 	for (unsigned int i = 0; i < PISA_BLOCKS(order); i++) {
 		struct page *block = page + (i << order);
 
@@ -158,9 +164,13 @@ void pisa_release_page(struct pisa_meta *meta)
 
 		pisa_slot_free(meta);
 		__free_pages(page, PISA_ORDER);
+		atomic_long_dec(&pisa_nr_pages);
 		return;
 	}
 
+	/* Count it now: pisa_alloc_held_pages() must not see a retiring page as held. */
+	atomic64_inc(&meta->pa->stats.page_free);
+
 	/* pisa_unmap_all_domains(meta) will be called here once IOMMU mappings are added. */
 
 	/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
@@ -322,6 +332,25 @@ unsigned long pisa_reservoir_count(void)
 	return nr;
 }
 
+/* Detach up to @nr_to_scan base pages from reservoirs onto @release_list. */
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan)
+{
+	unsigned long freed = 0;
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY) {
+		struct pisa_node *node = &pisa_nodes[nid];
+
+		if (!READ_ONCE(node->nr_pages))
+			continue;
+		freed += pisa_shrink_list(&node->lock, &node->pages, &node->nr_pages,
+					  release_list, nr_to_scan - freed);
+		if (freed >= nr_to_scan)
+			break;
+	}
+	return freed;
+}
+
 void __init pisa_node_init(void)
 {
 	for (int nid = 0; nid < MAX_NUMNODES; nid++) {
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 46170da8a0f0d..a191ede9bf3ce 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -79,6 +79,12 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
 	return meta->nr_clean + meta->nr_dirty;
 }
 
+/* Per-allocator statistics, see pisa_allocs_show(). */
+struct pisa_alloc_stats {
+	u64		page_alloc;
+	atomic64_t	page_free;
+};
+
 /* Allocator managing blocks of a fixed order, carved from PISA pages. */
 struct pisa_alloc {
 	/* First cacheline: hot fields touched on block alloc and free. */
@@ -91,12 +97,22 @@ struct pisa_alloc {
 	unsigned int		num_idle_pages;
 	unsigned int		max_idle_pages;
 	unsigned long		idle_jiffies;	/* last change of @idle at/above @max_idle_pages */
+	unsigned long		blocks_in_use;	/* blocks handed out and not yet freed */
 	/* Cold. */
 	int			nid;		/* home NUMA node */
 	struct list_head	node;		/* node in pisa_allocs */
 	char			name[48];
+	struct pisa_alloc_stats	stats;
 } ____cacheline_aligned;
 
+/* Active non-idle PISA pages held by @pa, for soft compaction checks. */
+static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
+{
+	return (long)(READ_ONCE(pa->stats.page_alloc) -
+		      atomic64_read(&pa->stats.page_free)) -
+	       READ_ONCE(pa->num_idle_pages);
+}
+
 /* Max block order for predefined per-node allocators in pisa_node.pa[][]. */
 #define PISA_MAX_ALLOC_ORDER	3U
 
@@ -117,6 +133,12 @@ extern struct pisa_node pisa_nodes[MAX_NUMNODES];
 int pisa_meta_ensure_pfn(unsigned long pfn, bool can_block);
 void pisa_slots_configure(void);
 
+/* Default global cap: 1/8 of RAM, at least 16 PISA pages. */
+static inline unsigned long pisa_default_max_pages(void)
+{
+	return max(totalram_pages() >> (PISA_ORDER + 3), 16UL);
+}
+
 static inline unsigned long pisa_meta_to_pfn(const struct pisa_meta *meta)
 {
 	return meta->base_pfn;
@@ -129,6 +151,16 @@ static inline struct page *pisa_meta_to_page(const struct pisa_meta *meta)
 
 void pisa_slot_free(struct pisa_meta *meta);
 
+struct shrink_control;
+struct shrinker;
+#if IS_ENABLED(CONFIG_KUNIT)
+unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+#endif
+
+/* Global PISA page budget across all allocators and per-node reservoirs. */
+extern atomic_long_t pisa_nr_pages;
+extern unsigned long pisa_max_pages;
+
 static inline int pisa_sanitize_nid(int nid)
 {
 	if ((unsigned int)nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
@@ -175,12 +207,18 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
 	return page;
 }
 
+#define PISA_OLD_MS_MAX		100000U
+unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
+			       unsigned int *nr, struct list_head *release_list,
+			       unsigned long nr_to_scan);
+
 /* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
 extern unsigned int pisa_reservoir_max;
 void pisa_node_init(void);
 void pisa_schedule_reservoir(void);
 struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
 unsigned long pisa_reservoir_count(void);
+unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
 
 #endif /* CONFIG_PISA */
 #endif /* _MM_PISA_PRIV_H */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (4 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
                     ` (20 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Each PISA page may have to be mapped in multiple IOMMU domains, and PISA
buffers must be recognized quickly from their IOVA. Both problems are
solved by reserving, in each domain, an IOVA window that only holds PISA
pages. With this:
- a PISA page is mapped at offset "slot * 2MB" in each domain, so there
  is no need to track individual offsets and one bit per domain suffices
  to tell whether the page is mapped.
- since the reserved IOVA window is only used for PISA memory,
  a simple address comparison suffices to identify a PISA-backed buffer.

Reserve an IOVA window in each domain, and use it to map PISA pages as
described above. Unmapping becomes a no-op, after a quick identification
of the address. The actual unmapping happens when the backing PISA page
is retired, or when the domain has no more users.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/iommu/Makefile     |   3 +
 drivers/iommu/dma-iommu.c  |  39 +++-
 drivers/iommu/iommu.c      |   7 +-
 drivers/iommu/pisa-map.c   | 385 +++++++++++++++++++++++++++++++++++++
 include/linux/pisa-iommu.h |  42 +++-
 include/linux/pisa.h       |  95 +++++++++
 mm/Kconfig                 |   1 +
 mm/pisa-alloc.c            |   4 +
 mm/pisa-kunit.c            | 170 ++++++++++++++++
 mm/pisa-node.c             |   3 +-
 mm/pisa-priv.h             |  10 +
 11 files changed, 755 insertions(+), 4 deletions(-)
 create mode 100644 drivers/iommu/pisa-map.c

diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index 29a26d2a3af12..a04b4458efaaa 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -11,6 +11,9 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_IOMMU_DEBUGFS) += iommu-debugfs.o
 obj-$(CONFIG_IOMMU_DMA) += dma-iommu.o
+ifdef CONFIG_IOMMU_DMA
+obj-$(CONFIG_PISA) += pisa-map.o
+endif
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 78742aeba932d..9b26d0aee9a97 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -36,6 +36,7 @@
 #include <trace/events/swiotlb.h>
 
 #include "dma-iommu.h"
+#include <linux/pisa.h>
 #include "iommu-pages.h"
 #include "iommu-priv.h"
 
@@ -76,6 +77,8 @@ struct iommu_dma_cookie {
 	struct iommu_domain *fq_domain;
 	/* Options for dma-iommu use */
 	struct iommu_dma_options options;
+	/* IOVA window reserved for PISA pages. Nothing else can go here. */
+	struct pisa_iova pisa_iova;
 };
 
 struct iommu_dma_msi_cookie {
@@ -379,6 +382,7 @@ int iommu_get_dma_cookie(struct iommu_domain *domain)
 		return -ENOMEM;
 
 	INIT_LIST_HEAD(&cookie->msi_page_list);
+	cookie->pisa_iova.iovad = &cookie->iovad;
 	domain->cookie_type = IOMMU_COOKIE_DMA_IOVA;
 	domain->iova_cookie = cookie;
 	return 0;
@@ -427,6 +431,9 @@ void iommu_put_dma_cookie(struct iommu_domain *domain)
 	struct iommu_dma_cookie *cookie = domain->iova_cookie;
 	struct iommu_dma_msi_page *msi, *tmp;
 
+	/* Drop cached PISA mappings before the IOVA domain goes away. */
+	pisa_domain_release(domain);
+
 	if (cookie->iovad.granule) {
 		iommu_dma_free_fq(cookie);
 		put_iova_domain(&cookie->iovad);
@@ -831,6 +838,8 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr,
 	struct iommu_iotlb_gather iotlb_gather;
 	size_t unmapped;
 
+	if (is_pisa_iova(&cookie->pisa_iova, dma_addr))
+		return;
 	dma_addr -= iova_off;
 	size = iova_align(iovad, size + iova_off);
 	iommu_iotlb_gather_init(&iotlb_gather);
@@ -857,6 +866,8 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys,
 	    iommu_deferred_attach(dev, domain))
 		return DMA_MAPPING_ERROR;
 
+	if (pisa_map_phys(&iova, dev, domain, phys, size, prot))
+		return iova;
 	/* If anyone ever wants this we'd need support in the IOVA allocator */
 	if (dev_WARN_ONCE(dev, dma_get_min_align_mask(dev) > iova_mask(iovad),
 	    "Unsupported alignment constraint\n"))
@@ -1215,6 +1226,16 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,
 	return iova_offset(iovad, phys | size);
 }
 
+/* Return @domain's PISA window, or NULL if @domain cannot use PISA. */
+struct pisa_iova *pisa_dma_iova(struct iommu_domain *domain)
+{
+	if (static_branch_unlikely(&iommu_deferred_attach_enabled) || !domain ||
+	    domain->cookie_type != IOMMU_COOKIE_DMA_IOVA)
+		return NULL;
+
+	return &domain->iova_cookie->pisa_iova;
+}
+
 dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
 		enum dma_data_direction dir, unsigned long attrs)
 {
@@ -1256,6 +1277,10 @@ void iommu_dma_unmap_phys(struct device *dev, dma_addr_t dma_handle,
 {
 	phys_addr_t phys;
 
+	/* PISA mapped buffer: nothing to unmap or release. */
+	if (is_pisa_iova(&iommu_get_dma_domain(dev)->iova_cookie->pisa_iova, dma_handle))
+		return;
+
 	if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) {
 		__iommu_dma_unmap(dev, dma_handle, size);
 		return;
@@ -1457,6 +1482,13 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
 			 */
 			break;
 		case PCI_P2PDMA_MAP_NONE:
+			/* Map PISA pages and mark as bus addresses to skip IOVA alloc and map. */
+			if (pisa_map_phys(&s->dma_address, dev, domain,
+					  sg_phys(s), s_length, prot)) {
+				sg_dma_len(s) = s_length;
+				sg_dma_mark_bus_address(s);
+				continue;
+			}
 			break;
 		case PCI_P2PDMA_MAP_BUS_ADDR:
 			/*
@@ -1822,7 +1854,12 @@ bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state,
 	dma_addr_t addr;
 
 	memset(state, 0, sizeof(*state));
-	if (!use_dma_iommu(dev))
+	/*
+	 * PISA memory is mapped in its own IOVA window. A PISA page whose
+	 * protection does not cover the request is declined here too, and
+	 * __iommu_dma_map() then maps it as ordinary memory, page by page.
+	 */
+	if (!use_dma_iommu(dev) || is_pisa_phys(phys))
 		return false;
 
 	domain = iommu_get_dma_domain(dev);
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index b486b8bbd1fcd..ac49b3b33ced6 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -37,6 +37,7 @@
 #include <linux/generic_pt/iommu.h>
 
 #include "dma-iommu.h"
+#include <linux/pisa.h>
 #include "iommu-priv.h"
 
 static struct kset *iommu_group_kset;
@@ -2867,7 +2868,7 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
 	while (i <= nents) {
 		phys_addr_t s_phys = sg_phys(sg);
 
-		if (len && s_phys != start + len) {
+		if (len && (sg_dma_is_bus_address(sg) || s_phys != start + len)) {
 			ret = iommu_map_nosync(domain, iova + mapped, start,
 					len, prot, gfp);
 			if (ret)
@@ -3363,6 +3364,10 @@ void iommu_device_unuse_default_domain(struct device *dev)
 	if (!WARN_ON(!group->owner_cnt || !xa_empty(&group->pasid_array)))
 		group->owner_cnt--;
 
+	/* Unmap persistent PISA windows when the last driver detaches. */
+	if (!group->owner_cnt)
+		pisa_domain_quiesce(group->default_domain);
+
 	mutex_unlock(&group->mutex);
 }
 
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
new file mode 100644
index 0000000000000..3ff9f1a6c5b85
--- /dev/null
+++ b/drivers/iommu/pisa-map.c
@@ -0,0 +1,385 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA per-domain IOVA window management and IOMMU mapping.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/atomic.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/dma-map-ops.h>
+#include <linux/dma-mapping.h>
+#include <linux/iommu.h>
+#include <linux/iova.h>
+#include <linux/log2.h>
+#include <linux/mm.h>
+#include <linux/pci.h>
+#include <linux/pisa-iommu.h>
+#include <linux/spinlock.h>
+#include <linux/srcu.h>
+
+
+/*
+ * Registry of PISA IOMMU domains. Each slot is FREE (!domain && !base),
+ * ACTIVE (domain), or DRAINING (!domain && base).
+ * Protected by pisa_domains_lock.
+ */
+static struct {
+	struct iommu_domain *domain;
+	dma_addr_t base;
+	spinlock_t map_lock;	/* serialises iommu_map() in __pisa_map_phys() */
+} pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
+static DEFINE_SPINLOCK(pisa_domains_lock);
+
+/* Protects pisa_unmap_all_domains() readers against concurrent pisa_domain_release(). */
+DEFINE_STATIC_SRCU(pisa_srcu);
+
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+unsigned int pisa_domains_generation __read_mostly;
+
+/* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
+static bool pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
+				 int idx, dma_addr_t iova, size_t unmapped)
+{
+	pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
+			   &iova, domain, idx, unmapped);
+	return pisa_quarantine_page(meta);
+}
+
+/* Offset of @meta from the start of a domain's window. */
+static inline dma_addr_t pisa_meta2iova_offset(const struct pisa_meta *meta)
+{
+	return (dma_addr_t)meta->slot << PISA_SHIFT;
+}
+
+/* Unmap @meta from every active domain that maps it. */
+void pisa_unmap_all_domains(struct pisa_meta *meta)
+{
+	dma_addr_t offset = pisa_meta2iova_offset(meta);
+	int idx;
+
+	guard(srcu)(&pisa_srcu);
+	for_each_set_bit(idx, meta->domains_mapped, PISA_MAX_DOMAINS) {
+		struct iommu_domain *domain;
+		unsigned long flags;
+		size_t unmapped;
+		dma_addr_t base;
+
+		if (!test_and_clear_bit(idx, meta->domains_mapped))
+			continue;
+
+		spin_lock_irqsave(&pisa_domains_lock, flags);
+		domain = pisa_domains[idx].domain;
+		base = pisa_domains[idx].base;
+		spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+		/* Free, or draining: pisa_domain_release() handles the bit. */
+		if (!domain)
+			continue;
+
+		unmapped = iommu_unmap(domain, base + offset, PISA_SIZE);
+		if (unlikely(unmapped != PISA_SIZE))
+			pisa_quarantine_meta(meta, domain, idx, base + offset, unmapped);
+	}
+}
+
+/* Clear domain @idx from all populated slots. */
+static unsigned int pisa_forget_domain(int idx)
+{
+	unsigned int slot, dropped = 0;
+
+	guard(rcu)();
+	for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+		struct pisa_meta *meta = &pisa_meta_array[slot];
+
+		if (test_bit(idx, meta->domains_mapped) &&
+		    test_and_clear_bit(idx, meta->domains_mapped))
+			dropped++;
+	}
+	return dropped;
+}
+
+/*
+ * Release @domain's registry index and clear its bit in all active
+ * PISA pages, waiting out concurrent unmappers on @pisa_srcu before freeing
+ * the slot.
+ */
+void __pisa_domain_release(struct iommu_domain *domain)
+{
+	struct pisa_iova *pi = pisa_dma_iova(domain);
+	unsigned int dropped;
+	unsigned long flags;
+	int idx;
+
+	/* Pairs with smp_store_release() in pisa_domain_init(). */
+	if (!pi || !smp_load_acquire(&pi->base))
+		return;
+
+	idx = pi->domain_idx - PISA_IDX_FIRST;
+	spin_lock_irqsave(&pisa_domains_lock, flags);
+	/* A window (@pi->base) is only published once @domain is registered at @idx. */
+	if (WARN_ON_ONCE(pisa_domains[idx].domain != domain)) {
+		spin_unlock_irqrestore(&pisa_domains_lock, flags);
+		return;
+	}
+	/* Leave @base set, mark the slot as DRAINING. */
+	pisa_domains[idx].domain = NULL;
+	spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+	dropped = pisa_forget_domain(idx);
+
+	/* Wait out in-flight pisa_unmap_all_domains() readers before reusing @idx. */
+	synchronize_srcu(&pisa_srcu);
+
+	spin_lock_irqsave(&pisa_domains_lock, flags);
+	/* Mark slot FREE and bump @pisa_domains_generation so NOIDX domains retry. */
+	pisa_domains[idx].base = 0;
+	WRITE_ONCE(pisa_domains_generation, pisa_domains_generation + 1);
+	spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+	if (dropped)
+		pr_debug("pisa: released %u PISA page mappings for domain %p (index %d)\n",
+			 dropped, domain, idx);
+}
+
+/*
+ * Unmap all PISA pages from a domain whose last DMA API user detached,
+ * keeping its window and domain index so mappings are re-installed on demand.
+ */
+void __pisa_domain_quiesce(struct iommu_domain *domain)
+{
+	struct pisa_iova *pi = pisa_dma_iova(domain);
+	struct iommu_iotlb_gather gather;
+	unsigned int slot, done = 0;
+	bool released = false;
+	dma_addr_t base;
+	int idx;
+
+	/* Pairs with smp_store_release() in pisa_domain_init(). */
+	base = pi ? smp_load_acquire(&pi->base) : 0;
+	if (!base)
+		return;
+	idx = pi->domain_idx - PISA_IDX_FIRST;
+
+	iommu_iotlb_gather_init(&gather);
+	/* Keep call_rcu()-freed pages from reaching the buddy allocator before IOTLB sync. */
+	rcu_read_lock();
+	for_each_set_bit(slot, pisa_slot_bitmap, pisa_nr_slots) {
+		struct pisa_meta *meta = &pisa_meta_array[slot];
+		dma_addr_t iova;
+		size_t unmapped;
+
+		if (!test_bit(idx, meta->domains_mapped) ||
+		    !test_and_clear_bit(idx, meta->domains_mapped))
+			continue;
+
+		/* Unmap the page, batching the IOTLB flush into @gather. */
+		iova = base + pisa_meta2iova_offset(meta);
+		unmapped = iommu_unmap_fast(domain, iova, PISA_SIZE, &gather);
+		if (unlikely(unmapped != PISA_SIZE))
+			released |= pisa_quarantine_meta(meta, domain, idx, iova, unmapped);
+		done++;
+	}
+
+	if (done)
+		iommu_iotlb_sync(domain, &gather);
+	rcu_read_unlock();
+	if (released)
+		pisa_schedule_reclaim();
+	if (done)
+		pr_debug("pisa: unmapped %u PISA pages from idle domain %p (index %d)\n",
+			 done, domain, idx);
+}
+
+/* Return negative errno if @pi's domain was permanently/currently declined, or 0. */
+int pisa_iova_declined(const struct pisa_iova *pi)
+{
+	switch (READ_ONCE(pi->domain_idx)) {
+	case PISA_IDX_NOHUGE:
+		return -EOPNOTSUPP;
+	case PISA_IDX_NOIOVA:
+		return -ENOSPC;
+	case PISA_IDX_NOIDX:
+		return READ_ONCE(pi->idx_gen) == READ_ONCE(pisa_domains_generation) ?
+		       -EBUSY : 0;
+	default:
+		return 0;
+	}
+}
+
+/* Allocate the initial IOVA window of @size bytes for @domain. */
+static dma_addr_t pisa_domain_alloc_iova(struct device *dev, struct iommu_domain *domain,
+					 struct iova_domain *iovad, u64 size)
+{
+	u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+	u64 align = min_t(u64, roundup_pow_of_two(size), SZ_1G);
+	unsigned long shift, iova_len;
+	struct iova *new_iova;
+
+	if (!iovad)
+		return 0;
+
+	limit = min_not_zero(limit, dev->coherent_dma_mask);
+	if (domain->geometry.force_aperture)
+		limit = min_t(u64, limit, domain->geometry.aperture_end);
+
+	/* Ignore devices with too many constraints. */
+	if (limit <= DMA_BIT_MASK(32) || size > limit - align)
+		return 0;
+
+	shift = iova_shift(iovad);
+	/* Overallocate to guarantee a 1GB aligned range */
+	iova_len = (size + align) >> shift;
+	new_iova = alloc_iova(iovad, iova_len, limit >> shift, false);
+	return new_iova ? ALIGN((dma_addr_t)new_iova->pfn_lo << shift, align) : 0;
+}
+
+/*
+ * Assign @domain an IOVA window and dense domain index on first use, recording
+ * permanent or retryable failures in @pi->domain_idx.
+ */
+static int pisa_domain_init(struct device *dev, struct iommu_domain *domain,
+			    struct pisa_iova *pi)
+{
+	u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+	u64 size = pisa_iova_size();
+	unsigned long flags;
+	dma_addr_t base;
+	int idx, ret;
+
+	limit = min_not_zero(limit, dev->coherent_dma_mask);
+
+	if (!dev_is_dma_coherent(dev) || (dev_is_pci(dev) && to_pci_dev(dev)->untrusted))
+		return -EOPNOTSUPP;
+
+	if (limit <= DMA_BIT_MASK(32) || size > limit - PISA_SIZE)
+		return -ENOSPC;
+
+	if (!pisa_domain_pgsize_ok(domain)) {
+		if (cmpxchg(&pi->domain_idx, PISA_IDX_NONE, PISA_IDX_NOHUGE) == PISA_IDX_NONE)
+			dev_warn_ratelimited(dev, "pisa: domain cannot map %luKB pages, not PISA\n",
+					     (unsigned long)(PISA_SIZE >> 10));
+		return -EOPNOTSUPP;
+	}
+
+	spin_lock_irqsave(&pisa_domains_lock, flags);
+
+	/* Bail if another CPU got here first, or if declined earlier. */
+	ret = pi->base ? 0 : pisa_iova_declined(pi);
+	if (pi->base || ret) {
+		spin_unlock_irqrestore(&pisa_domains_lock, flags);
+		return ret;
+	}
+
+	for (idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+		if (!pisa_domains[idx].domain && !pisa_domains[idx].base)
+			break;
+	}
+	if (idx == PISA_MAX_DOMAINS) {
+		pi->idx_gen = pisa_domains_generation;
+		WRITE_ONCE(pi->domain_idx, PISA_IDX_NOIDX);
+		spin_unlock_irqrestore(&pisa_domains_lock, flags);
+		pr_warn_ratelimited("pisa: all %d domain indices in use, not PISA for domain %p until one is released\n",
+				    PISA_MAX_DOMAINS, domain);
+		return -EBUSY;
+	}
+
+	base = pisa_domain_alloc_iova(dev, domain, pi->iovad, size);
+	if (!base) {
+		WRITE_ONCE(pi->domain_idx, PISA_IDX_NOIOVA);
+		spin_unlock_irqrestore(&pisa_domains_lock, flags);
+		dev_warn_ratelimited(dev,
+				     "pisa: no free %llu MB IOVA range for a PISA window\n",
+				     size >> 20);
+		return -ENOSPC;
+	}
+
+	pisa_domains[idx].domain = domain;
+	pisa_domains[idx].base = base;
+	spin_lock_init(&pisa_domains[idx].map_lock);
+	WRITE_ONCE(pi->domain_idx, idx + PISA_IDX_FIRST);
+	/* Publish .domain_idx before .base (paired with is_pisa_iova()). */
+	smp_store_release(&pi->base, base);
+	spin_unlock_irqrestore(&pisa_domains_lock, flags);
+
+	dev_info(dev, "pisa: domain %p index %d window %pad + %llu MB%s\n",
+		 domain, idx, &base, size >> 20,
+		 (domain->pgsize_bitmap & PISA_SIZE) ? "" : ", no leaf PTE of PISA page size");
+	return 0;
+}
+
+/*
+ * Set *@iova for address @phys for @dev. @phys is known to be PISA memory.
+ * On first use in a domain, reserves @pi via alloc_iova(), and on first use
+ * of a PISA page in that domain, maps it (with one leaf PTE if the domain has
+ * that page size) via iommu_map(..., GFP_ATOMIC) under spinlocks.
+ * Returns true on success, or false if @phys/@prot/@dev cannot use the window
+ * (callers only need to know whether to fall back to regular DMA mapping, not
+ * the specific error).
+ */
+bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+		     struct iommu_domain *domain, phys_addr_t phys,
+		     size_t size, int prot)
+{
+	struct pisa_iova *pi = pisa_dma_iova(domain);
+	unsigned long pfn = phys >> PAGE_SHIFT;
+	unsigned int domain_idx;
+	struct pisa_meta *meta;
+	int block_prot, ret;
+	dma_addr_t addr;
+	u64 limit;
+
+	if (unlikely(!iova || !dev || !pi || !dev_is_dma_coherent(dev) ||
+		     (dev_is_pci(dev) && to_pci_dev(dev)->untrusted) ||
+		     (phys & (PISA_SIZE - 1)) + size > PISA_SIZE))
+		return false;
+
+	meta = pisa_meta_of_pfn(pfn);
+
+	/* Every mapping of the page carries IOMMU_CACHE | @meta->prot. */
+	block_prot = IOMMU_CACHE | meta->prot;
+	if (unlikely(prot & ~block_prot))
+		return false;
+
+	/* Acquire pairs with the .base release in pisa_domain_init(). */
+	if (unlikely(!smp_load_acquire(&pi->base))) {
+		/* Test cached decline in @domain_idx before taking @pisa_domains_lock. */
+		ret = pisa_iova_declined(pi);
+		if (likely(!ret))
+			ret = pisa_domain_init(dev, domain, pi);
+
+		if (ret)
+			return false;
+	}
+
+	addr = pisa_phys_to_iova(pi, meta, phys);
+
+	/* Check @dev's own DMA mask and bus limit against the window IOVA. */
+	limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+	limit = min_not_zero(limit, dev->coherent_dma_mask);
+	if (unlikely(addr + size - 1 > limit))
+		return false;
+
+	domain_idx = pi->domain_idx - PISA_IDX_FIRST;
+	if (likely(test_bit(domain_idx, meta->domains_mapped))) {
+		*iova = addr;
+		return true;
+	}
+
+	/* First use of this PISA page in this domain: map it. */
+	guard(spinlock_irqsave)(&pisa_domains[domain_idx].map_lock);
+	if (!test_bit(domain_idx, meta->domains_mapped)) {
+		phys_addr_t pisa_phys = phys & PISA_MASK;
+
+		ret = iommu_map(domain, pisa_phys_to_iova(pi, meta, pisa_phys),
+				pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+		if (unlikely(ret))
+			return false;
+		/* Ensure the mapping is visible before setting the domain bit. */
+		smp_mb__before_atomic();
+		set_bit(domain_idx, meta->domains_mapped);
+	}
+
+	*iova = addr;
+	return true;
+}
diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index b29b405804193..d4c01207dca72 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -5,6 +5,7 @@
 
 #include <linux/bitmap.h>
 #include <linux/dma-direct.h>
+#include <linux/iommu.h>
 #include <linux/list.h>
 #include <linux/llist.h>
 #include <linux/pisa.h>
@@ -17,20 +18,46 @@ struct pisa_alloc;
 #define PISA_SIZE		(1UL << PISA_SHIFT)
 #define PISA_MASK		(~(PISA_SIZE - 1))
 
+/*
+ * Can @domain map a PISA page? One leaf PTE of PISA_SIZE is best, but smaller
+ * IOMMU pages that divide PISA_SIZE work too.
+ */
+static inline bool pisa_domain_pgsize_ok(const struct iommu_domain *domain)
+{
+	return domain->pgsize_bitmap &&
+	       IS_ALIGNED(PISA_SIZE, 1UL << __ffs(domain->pgsize_bitmap));
+}
+
 /* size of struct pisa_meta (256B, power of 2) and max blocks per PISA page. */
 #define _PISA_META_SHIFT	8
 #define _PISA_META_SIZE		(1U << _PISA_META_SHIFT)
 #define _PISA_MAX_BLOCKS	512U
 
+/* Max simultaneous PISA IOMMU domains (bit index in pisa_meta.domains_mapped). */
+#define PISA_MAX_DOMAINS	128
+
+/* Sentinel values of pisa_iova.domain_idx below PISA_IDX_FIRST. */
+enum {
+	PISA_IDX_NONE,		/* not attempted yet */
+	PISA_IDX_NOHUGE,	/* @domain can never use PISA */
+	PISA_IDX_NOIOVA,	/* no IOVA range for the window */
+	PISA_IDX_NOIDX,		/* all domain indices were in use */
+	PISA_IDX_FIRST,		/* first valid domain index */
+};
+
 /* Metadata for a single PISA page in @pisa_meta_array (indexed by @slot). */
 struct pisa_meta {
 	unsigned long flags;	/* PISA_ACTIVE / PISA_QUARANTINED / PISA_DECRYPTED */
 	unsigned long base_pfn;	/* base PFN of this PISA page */
+	u8 prot;		/* owning @pa's IOMMU_READ/WRITE protection bits */
 	u16 nr_clean;		/* zeroed available blocks in @clean_bitmap */
 	u16 nr_dirty;		/* dirty available blocks in @dirty_bitmap */
 	u8 on_list;		/* current placement in @pa (PISA_LIST_*) */
 	bool drain;		/* sparse old page draining back to @idle */
 	u32 slot;		/* dense index in @pisa_meta_array and IOVA windows */
+
+	DECLARE_BITMAP(domains_mapped, PISA_MAX_DOMAINS);
+
 	struct pisa_alloc *pa;	/* NULL for reservoir pages */
 	struct list_head list;	/* node in @pa lists or pisa_node.pages */
 
@@ -47,21 +74,34 @@ struct pisa_meta {
 static_assert(sizeof(struct pisa_meta) == _PISA_META_SIZE);
 
 extern struct pisa_meta *pisa_meta_array;
-extern unsigned int pisa_nr_slots;
+extern unsigned long *pisa_slot_bitmap;
 
 struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
 bool pisa_quarantine_page(struct pisa_meta *meta);
 void pisa_schedule_reclaim(void);
 
+/* IOVA of @phys inside active RAM page @meta in @pi. */
+static inline dma_addr_t pisa_phys_to_iova(const struct pisa_iova *pi,
+					   const struct pisa_meta *meta, phys_addr_t phys)
+{
+	return pi->base + ((dma_addr_t)meta->slot << PISA_SHIFT) +
+	       (phys & (PISA_SIZE - 1));
+}
+
 #endif /* CONFIG_PISA */
 
 #if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
 
+/* Bumped under pisa_domains_lock whenever a domain index is released. */
+extern unsigned int pisa_domains_generation;
+
 void pisa_unmap_all_domains(struct pisa_meta *meta);
+int pisa_iova_declined(const struct pisa_iova *pi);
 
 #else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
 
 static inline void pisa_unmap_all_domains(struct pisa_meta *meta) {}
+static inline int pisa_iova_declined(const struct pisa_iova *pi) { return -EOPNOTSUPP; }
 
 #endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
 #endif /* _LINUX_PISA_IOMMU_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 3307fae1dbd66..cc33e181df097 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -7,10 +7,36 @@
 #include <linux/bitops.h>
 #include <linux/dma-mapping.h>
 #include <linux/jump_label.h>
+#include <linux/device.h>
 #include <linux/mm.h>
 #include <linux/uaccess.h>
 #include <linux/rcupdate.h>
 
+struct iommu_domain;
+struct iova_domain;
+
+/**
+ * struct pisa_iova - A domain's IOVA window reserved for PISA pages
+ * @iovad: Domain's IOVA allocator (&domain->iova_cookie->iovad)
+ * @base: Window base IOVA (PISA_SIZE aligned), or 0 if unassigned
+ * @domain_idx: Dense index + PISA_IDX_FIRST, or PISA_IDX_* sentinel while @base == 0
+ * @idx_gen: Value of pisa_domains_generation when @domain_idx was set to NOIDX
+ *
+ * Lives in iommu_dma_cookie. Active PISA pages map at @base + slot * PISA_SIZE.
+ */
+struct pisa_iova {
+	struct iova_domain *iovad;
+	dma_addr_t base;
+	u16 domain_idx;
+	u32 idx_gen;
+};
+
+#ifdef CONFIG_IOMMU_DMA
+struct pisa_iova *pisa_dma_iova(struct iommu_domain *domain);
+#else
+static inline struct pisa_iova *pisa_dma_iova(struct iommu_domain *domain) { return NULL; }
+#endif
+
 #ifdef CONFIG_PISA
 
 /* PISA page geometry (log2 byte size and base-page order). */
@@ -49,6 +75,13 @@ DECLARE_STATIC_KEY_FALSE(pisa_key);
 DECLARE_STATIC_KEY_FALSE(pisa_recycle_check_key);
 
 extern u32 *pisa_pfn2slot;
+extern unsigned int pisa_nr_slots;
+
+/* Size of the IOVA window each domain reserves for PISA pages: one per metadata slot. */
+static inline u64 pisa_iova_size(void)
+{
+	return (u64)pisa_nr_slots << PISA_SHIFT;
+}
 
 int pisa_meta_init(void);
 bool __pisa_page_can_recycle(const struct page *page);
@@ -131,4 +164,66 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 }
 
 #endif /* CONFIG_PISA */
+
+#if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
+
+bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
+		     struct iommu_domain *domain, phys_addr_t phys,
+		     size_t size, int prot);
+void __pisa_domain_release(struct iommu_domain *domain);
+void __pisa_domain_quiesce(struct iommu_domain *domain);
+
+/* Hook for __iommu_dma_map() and iommu_dma_map_sg() (guarded by @pisa_key). */
+static inline bool pisa_map_phys(dma_addr_t *iova, struct device *dev,
+				 struct iommu_domain *domain, phys_addr_t phys,
+				 size_t size, int prot)
+{
+	return is_pisa_phys(phys) &&
+	       __pisa_map_phys(iova, dev, domain, phys, size, prot);
+}
+
+/* Hook for __iommu_dma_unmap() and __iommu_dma_unmap_phys() (guarded by @pisa_key). */
+static inline bool is_pisa_iova(const struct pisa_iova *pi, dma_addr_t dma)
+{
+	dma_addr_t base;
+
+	if (!pisa_enabled())
+		return false;
+	/* Acquire pairs with the release of @base in pisa_domain_init(). */
+	base = smp_load_acquire(&pi->base);
+	return base && dma - base < pisa_iova_size();
+}
+
+/* Hook for iommu_put_dma_cookie() (guarded by @pisa_key). */
+static inline void pisa_domain_release(struct iommu_domain *domain)
+{
+	if (pisa_enabled())
+		__pisa_domain_release(domain);
+}
+
+/* Hook for iommu_device_unuse_default_domain() (guarded by @pisa_key). */
+static inline void pisa_domain_quiesce(struct iommu_domain *domain)
+{
+	if (pisa_enabled())
+		__pisa_domain_quiesce(domain);
+}
+
+#else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
+
+static inline bool pisa_map_phys(dma_addr_t *iova, struct device *dev,
+				 struct iommu_domain *domain, phys_addr_t phys,
+				 size_t size, int prot)
+{
+	return false;
+}
+
+static inline bool is_pisa_iova(const struct pisa_iova *pi, dma_addr_t dma)
+{
+	return false;
+}
+
+static inline void pisa_domain_release(struct iommu_domain *domain) {}
+static inline void pisa_domain_quiesce(struct iommu_domain *domain) {}
+
+#endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
 #endif /* _LINUX_PISA_H */
diff --git a/mm/Kconfig b/mm/Kconfig
index 6e7811569d33a..f96c4017519ee 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1436,6 +1436,7 @@ config PISA
 	bool "Pre-mapped IOVA Superpage Allocator"
 	depends on (X86_64 && !UML) || (ARM64 && ARM64_4K_PAGES && PGTABLE_LEVELS >= 4)
 	depends on SPARSEMEM && !PREEMPT_RT
+	select NEED_SG_DMA_FLAGS
 	default y
 	help
 	  PISA is a specialized memory allocator with an API similar to
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index c09f321b07eeb..01d8bbc8d72ff 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -13,6 +13,7 @@
 #include <linux/pisa.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
+#include <linux/iommu.h>
 #include <linux/kasan.h>
 #include <linux/list.h>
 #include <linux/llist.h>
@@ -261,6 +262,8 @@ struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_
 	spin_lock_init(&pa->lock);
 	pa->order = order;
 	pa->nid = nid;
+	pa->prot = (dir != DMA_FROM_DEVICE ? IOMMU_READ : 0) |
+		     (dir != DMA_TO_DEVICE ? IOMMU_WRITE : 0);
 	pisa_alloc_set_name(pa, "%ps", (void *)_RET_IP_);
 	INIT_LIST_HEAD(&pa->partial);
 	INIT_LIST_HEAD(&pa->partial_dirty);
@@ -585,6 +588,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 			goto out;
 		spin_lock_irqsave(&pa->lock, flags);
 		meta->pa = pa;
+		meta->prot = pa->prot;
 		WRITE_ONCE(meta->alloc_jiffies, jiffies);
 		/* Set PISA_ACTIVE after @meta is fully set up. */
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index cd2bfbfb0f1a7..eb51007e66de5 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -6,6 +6,8 @@
 #include <linux/delay.h>
 #include <linux/gfp.h>
 #include <linux/pisa.h>
+#include <linux/iommu.h>
+#include <linux/iova.h>
 #include <linux/mm.h>
 #include <linux/shrinker.h>
 #include <linux/skbuff.h>
@@ -277,6 +279,170 @@ static void test_quarantine(struct kunit *test)
 	__free_pages(page, PISA_ORDER);
 }
 
+#ifdef CONFIG_IOMMU_DMA
+static void test_iova_helpers(struct kunit *test)
+{
+	struct device dev = { .numa_node = NUMA_NO_NODE };
+	struct iommu_domain dummy_domain = {};
+	struct pisa_iova pi = {};
+	dma_addr_t d;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+
+	/* Empty window (base == 0) never owns any IOVA. */
+	KUNIT_EXPECT_FALSE(test, is_pisa_iova(&pi, 0));
+	KUNIT_EXPECT_FALSE(test, is_pisa_iova(&pi, SZ_4G));
+
+	pi.base = SZ_4G;
+	KUNIT_EXPECT_FALSE(test, is_pisa_iova(&pi, SZ_4G - 1));
+	KUNIT_EXPECT_TRUE(test, is_pisa_iova(&pi, SZ_4G));
+	KUNIT_EXPECT_TRUE(test, is_pisa_iova(&pi, SZ_4G + pisa_iova_size() - 1));
+	KUNIT_EXPECT_FALSE(test, is_pisa_iova(&pi, SZ_4G + pisa_iova_size()));
+
+	/* Sentinel values in pisa_iova_declined(). */
+	pi.domain_idx = PISA_IDX_NONE;
+	KUNIT_EXPECT_EQ(test, pisa_iova_declined(&pi), 0);
+	pi.domain_idx = PISA_IDX_NOHUGE;
+	KUNIT_EXPECT_EQ(test, pisa_iova_declined(&pi), -EOPNOTSUPP);
+	pi.domain_idx = PISA_IDX_NOIOVA;
+	KUNIT_EXPECT_EQ(test, pisa_iova_declined(&pi), -ENOSPC);
+	pi.domain_idx = PISA_IDX_NOIDX;
+	pi.idx_gen = READ_ONCE(pisa_domains_generation);
+	KUNIT_EXPECT_EQ(test, pisa_iova_declined(&pi), -EBUSY);
+	pi.idx_gen--;
+	KUNIT_EXPECT_EQ(test, pisa_iova_declined(&pi), 0);
+
+	/* A device without an IOMMU group has no PISA window. */
+	KUNIT_EXPECT_FALSE(test, pisa_map_phys(&d, NULL, NULL, 0, PAGE_SIZE, 0));
+	KUNIT_EXPECT_FALSE(test, pisa_map_phys(&d, &dev, NULL, 0, PAGE_SIZE, 0));
+	/* Releasing an unregistered domain with an active allocator is a safe no-op. */
+	pisa_domain_release(&dummy_domain);
+}
+
+/*
+ * Fake DMA domain for the mapping and coherent allocation tests, backed by a
+ * real iommu_dma_cookie and no-op page table ops.
+ */
+static int fake_map_pages(struct iommu_domain *domain, unsigned long iova,
+			  phys_addr_t paddr, size_t pgsize, size_t pgcount,
+			  int prot, gfp_t gfp, size_t *mapped)
+{
+	*mapped = pgsize * pgcount;
+	return 0;
+}
+
+static size_t fake_unmap_pages(struct iommu_domain *domain, unsigned long iova,
+			       size_t pgsize, size_t pgcount,
+			       struct iommu_iotlb_gather *iotlb_gather)
+{
+	return pgsize * pgcount;
+}
+
+static const struct iommu_domain_ops fake_domain_ops = {
+	.map_pages = fake_map_pages,
+	.unmap_pages = fake_unmap_pages,
+};
+
+static struct iommu_domain fake_domain = {
+	.type = IOMMU_DOMAIN_DMA,
+	.pgsize_bitmap = SZ_4K | PISA_SIZE,
+	.ops = &fake_domain_ops,
+};
+
+static void fake_domain_teardown(void *unused)
+{
+	iommu_put_dma_cookie(&fake_domain);
+	fake_domain.cookie_type = IOMMU_COOKIE_NONE;
+	fake_domain.iova_cookie = NULL;
+	iova_cache_put();
+}
+
+static int fake_domain_setup(struct kunit *test)
+{
+	int ret = iova_cache_get();
+
+	if (ret)
+		return ret;
+	ret = iommu_get_dma_cookie(&fake_domain);
+	if (ret) {
+		iova_cache_put();
+		return ret;
+	}
+	init_iova_domain(pisa_dma_iova(&fake_domain)->iovad, SZ_4K, 1);
+	return kunit_add_action_or_reset(test, fake_domain_teardown, NULL);
+}
+
+static void test_map_and_domain_lifecycle(struct kunit *test)
+{
+	struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
+			      .coherent_dma_mask = DMA_BIT_MASK(64) };
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+	struct pisa_iova *pi;
+	struct pisa_meta *meta;
+	struct pisa_alloc *pa;
+	unsigned int domain_idx;
+	dma_addr_t d1, d2;
+	struct page *p;
+	phys_addr_t phys;
+
+	dev.dma_mask = &dev.coherent_dma_mask;
+	set_dev_node(&dev, first_memory_node);
+	dev_set_dma_coherent(&dev);
+	KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+	pi = pisa_dma_iova(&fake_domain);
+
+	pa = pisa_node_pa(first_memory_node, SKB_FRAG_PAGE_ORDER, true);
+	KUNIT_ASSERT_NOT_NULL(test, pa);
+	p = pisa_alloc(pa, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, p);
+
+	phys = page_to_phys(p);
+	meta = pisa_meta_of_pfn(page_to_pfn(p));
+	KUNIT_EXPECT_TRUE(test, is_pisa_phys(phys));
+
+	/* TX-only allocator declines IOMMU_WRITE mappings. */
+	KUNIT_EXPECT_FALSE(test,
+			   pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+					 IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE));
+
+	/* First map assigns the window and maps the page. */
+	KUNIT_ASSERT_TRUE(test,
+			  pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+					IOMMU_READ | IOMMU_CACHE));
+	KUNIT_EXPECT_TRUE(test, is_pisa_iova(pi, d1));
+	domain_idx = pi->domain_idx - PISA_IDX_FIRST;
+	KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+	/* Second map at an offset hits the lockless fast path. */
+	KUNIT_ASSERT_TRUE(test,
+			  pisa_map_phys(&d2, &dev, &fake_domain, phys + 128, 256,
+					IOMMU_READ | IOMMU_CACHE));
+	KUNIT_EXPECT_EQ(test, d2, d1 + 128);
+
+	/* Quiescing the domain unmaps the PISA page but keeps the window. */
+	pisa_domain_quiesce(&fake_domain);
+	KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+	KUNIT_ASSERT_TRUE(test,
+			  pisa_map_phys(&d2, &dev, &fake_domain, phys, PAGE_SIZE,
+					IOMMU_READ | IOMMU_CACHE));
+	KUNIT_EXPECT_EQ(test, d2, d1);
+	KUNIT_EXPECT_TRUE(test, test_bit(domain_idx, meta->domains_mapped));
+
+	/* Releasing the domain forgets the mapping bit across allocators. */
+	pisa_domain_release(&fake_domain);
+	KUNIT_EXPECT_FALSE(test, test_bit(domain_idx, meta->domains_mapped));
+
+	/* Re-map into a fresh window so shrinking exercises pisa_unmap_all_domains(). */
+	*pi = (struct pisa_iova){ .iovad = pi->iovad };
+	KUNIT_EXPECT_TRUE(test,
+			  pisa_map_phys(&d1, &dev, &fake_domain, phys, PAGE_SIZE,
+					IOMMU_READ | IOMMU_CACHE));
+
+	__free_pages(p, SKB_FRAG_PAGE_ORDER);
+	pisa_shrink_scan(NULL, &sc);
+}
+#endif /* CONFIG_IOMMU_DMA */
+
 static void test_reservoir_and_scrubber(struct kunit *test)
 {
 	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -289,6 +455,10 @@ static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_meta_active_toggle),
 	KUNIT_CASE(test_alloc_and_recycle),
 	KUNIT_CASE(test_quarantine),
+#ifdef CONFIG_IOMMU_DMA
+	KUNIT_CASE(test_iova_helpers),
+	KUNIT_CASE(test_map_and_domain_lifecycle),
+#endif
 	KUNIT_CASE(test_reservoir_and_scrubber),
 	{}
 };
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index 29f6ba866b633..a39251b952263 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -171,7 +171,8 @@ void pisa_release_page(struct pisa_meta *meta)
 	/* Count it now: pisa_alloc_held_pages() must not see a retiring page as held. */
 	atomic64_inc(&meta->pa->stats.page_free);
 
-	/* pisa_unmap_all_domains(meta) will be called here once IOMMU mappings are added. */
+	/* Quarantined on unmap failure and leaked in pisa_release_page_rcu(). */
+	pisa_unmap_all_domains(meta);
 
 	/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
 	pisa_meta_clear_flags(meta, PISA_ACTIVE);
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index a191ede9bf3ce..8736b8d168727 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -16,6 +16,11 @@
 #include <linux/workqueue.h>
 
 
+#ifdef CONFIG_IOMMU_DMA
+int iommu_get_dma_cookie(struct iommu_domain *domain);
+void iommu_put_dma_cookie(struct iommu_domain *domain);
+#endif
+
 #ifdef CONFIG_PISA
 
 /* Number of order-@order blocks in one PISA page. */
@@ -31,6 +36,10 @@
  * 2. Metadata tables (mm/pisa-meta.c):
  *    pisa_meta_mutex (mutex): init and hotplug page-table population.
  *    pisa_slot_lock (spinlock): slot allocation and @pisa_meta_array PTE install.
+ *
+ * 3. IOMMU domains (drivers/iommu/pisa-map.c):
+ *    pisa_srcu -> pisa_domains_lock (spinlock, leaf over pisa_domains[]).
+ *    pisa_domains[].map_lock (spinlock): standalone per-domain iommu_map() lock.
  */
 
 /*
@@ -90,6 +99,7 @@ struct pisa_alloc {
 	/* First cacheline: hot fields touched on block alloc and free. */
 	spinlock_t		lock;
 	u8			order;		/* block order (<= PISA_ORDER) */
+	u8			prot;		/* IOMMU_READ and/or IOMMU_WRITE */
 	struct list_head	partial;	/* partial pages with nr_clean > 0 */
 	struct list_head	partial_dirty;	/* partial pages with nr_clean == 0 */
 	struct list_head	idle;		/* completely free PISA pages */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (5 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
                     ` (19 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Because of the performance and security implications, the use of PISA
features is opt-in and configurable per device with a fine granularity.

Add boolean attributes to struct device exposed under
/sys/devices/*/*/pisa_*:
  - pisa_coherent:     enable PISA for coherent allocations
  - pisa_rx_bufs:      enable PISA for RX buffers / page_pool
  - pisa_tx_bufs:      enable PISA for TX socket page frags
  - pisa_tx_skb_heads: enable PISA slab cache for small TX skb heads
  - pisa_tx_zerocopy:  migrate MSG_ZEROCOPY user pages into PISA

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/base/core.c    | 75 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/device.h | 15 +++++++++
 include/linux/pisa.h   | 27 +++++++++++++++
 mm/pisa-alloc.c        | 21 ++++++++++++
 mm/pisa-kunit.c        | 71 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 209 insertions(+)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index cbe05197d4a0b..df93b6b6e125e 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -25,6 +25,7 @@
 #include <linux/notifier.h>
 #include <linux/of.h>
 #include <linux/of_device.h>
+#include <linux/pisa.h>
 #include <linux/pm_runtime.h>
 #include <linux/sched/mm.h>
 #include <linux/sched/signal.h>
@@ -2907,6 +2908,65 @@ static ssize_t removable_show(struct device *dev, const struct device_attribute
 }
 static const DEVICE_ATTR_RO(removable);
 
+/* Per-device configuration to selectively enable PISA */
+static ssize_t pisa_attr_store(struct device *dev, const char *buf, size_t count,
+			       bool *field, int (*setter)(struct device *, bool))
+{
+	bool val;
+	int ret = kstrtobool(buf, &val);
+
+	if (ret < 0)
+		return ret;
+	if (val) {
+		ret = pisa_meta_init();
+		if (ret < 0)
+			return ret;
+	}
+	if (setter) {
+		ret = setter(dev, val);
+		if (ret < 0)
+			return ret;
+	} else {
+		WRITE_ONCE(*field, val);
+	}
+	return count;
+}
+
+#define PISA_SET_ATTR(name, setter)					\
+static ssize_t name##_show(struct device *dev,				\
+			   struct device_attribute *attr, char *buf)	\
+{									\
+	return sysfs_emit(buf, "%u\n", READ_ONCE(dev->name));		\
+}									\
+static ssize_t name##_store(struct device *dev,				\
+			    struct device_attribute *attr,		\
+			    const char *buf, size_t count)		\
+{									\
+	return pisa_attr_store(dev, buf, count, &dev->name, setter);	\
+}									\
+static DEVICE_ATTR_RW(name)
+
+#define PISA_BOOL_ATTR(name)	PISA_SET_ATTR(name, NULL)
+
+PISA_BOOL_ATTR(pisa_coherent);
+PISA_BOOL_ATTR(pisa_rx_bufs);
+PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_BOOL_ATTR(pisa_tx_skb_heads);
+PISA_BOOL_ATTR(pisa_tx_zerocopy);
+
+static struct attribute *dev_attr_pisa[] = {
+	&dev_attr_pisa_coherent.attr,
+	&dev_attr_pisa_rx_bufs.attr,
+	&dev_attr_pisa_tx_bufs.attr,
+	&dev_attr_pisa_tx_skb_heads.attr,
+	&dev_attr_pisa_tx_zerocopy.attr,
+	NULL,
+};
+
+static const struct attribute_group dev_attr_pisa_group = {
+	.attrs = dev_attr_pisa,
+};
+
 int device_add_groups(struct device *dev,
 		      const struct attribute_group *const *groups)
 {
@@ -3014,6 +3074,18 @@ static int device_add_attrs(struct device *dev)
 			goto err_remove_dev_removable;
 	}
 
+	if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
+		error = device_add_group(dev, &dev_attr_pisa_group);
+		if (error) {
+			if (dev->physical_location) {
+				device_remove_group(dev, &dev_attr_physical_location_group);
+				kfree(dev->physical_location);
+				dev->physical_location = NULL;
+			}
+			goto err_remove_dev_removable;
+		}
+	}
+
 	return 0;
 
  err_remove_dev_removable:
@@ -3039,6 +3111,9 @@ static void device_remove_attrs(struct device *dev)
 	const struct class *class = dev->class;
 	const struct device_type *type = dev->type;
 
+	if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+		device_remove_group(dev, &dev_attr_pisa_group);
+
 	if (dev->physical_location) {
 		device_remove_group(dev, &dev_attr_physical_location_group);
 		kfree(dev->physical_location);
diff --git a/include/linux/device.h b/include/linux/device.h
index 429e39f647cf9..e705df38e49af 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -695,6 +695,15 @@ enum struct_device_flags {
  * @removable:  Whether the device can be removed from the system. This
  *              should be set by the subsystem / bus driver that discovered
  *              the device.
+ * @pisa_coherent: Back coherent dma_alloc_attrs() with PISA
+ *		(sysfs pisa_coherent).
+ * @pisa_rx_bufs: Use PISA allocators for RX buffers (sysfs pisa_rx_bufs).
+ * @pisa_tx_bufs: Use PISA allocators for socket TX page fragments
+ *		(sysfs pisa_tx_bufs).
+ * @pisa_tx_skb_heads: Use PISA slab cache for small TX skb heads
+ *		(sysfs pisa_tx_skb_heads).
+ * @pisa_tx_zerocopy: Migrate MSG_ZEROCOPY user pages into PISA
+ *		(sysfs pisa_tx_zerocopy).
  * @flags:	DEV_FLAG_XXX flags. Use atomic bitfield operations to modify.
  *
  * At the lowest level, every device in a Linux system is represented by an
@@ -799,6 +808,12 @@ struct device {
 
 	enum device_removable	removable;
 
+	bool			pisa_coherent;
+	bool			pisa_rx_bufs;
+	bool			pisa_tx_bufs;
+	bool			pisa_tx_skb_heads;
+	bool			pisa_tx_zerocopy;
+
 	DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
 };
 
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index cc33e181df097..5222cb207257f 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -84,6 +84,7 @@ static inline u64 pisa_iova_size(void)
 }
 
 int pisa_meta_init(void);
+bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
 bool __pisa_page_can_recycle(const struct page *page);
 
 static inline bool pisa_enabled(void)
@@ -142,6 +143,21 @@ static inline struct page *pisa_virt_to_page(const void *addr)
 	return is_pisa_page(page) ? page : NULL;
 }
 
+/* Hook for __page_pool_alloc_netmems_slow() (guarded by @pisa_key). */
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+	return pisa_enabled() && dev && READ_ONCE(dev->pisa_rx_bufs) ?
+	       __GFP_PISA_RW : 0;
+}
+
+/* Hook for NIC driver RX buffer allocation (guarded by @pisa_key). */
+static inline bool pisa_has_free_blocks_rx(const struct device *dev,
+					   unsigned int order, int nid)
+{
+	return pisa_enabled() && dev && READ_ONCE(dev->pisa_rx_bufs) &&
+	       __pisa_has_free_blocks_rx(order, nid);
+}
+
 /* Hook for skb_page_frag_refill() and __pisa_free_page() (@pisa_recycle_check_key). */
 static inline bool pisa_page_can_recycle(const struct page *page)
 {
@@ -158,6 +174,17 @@ static inline bool is_pisa_page(const struct page *page) { return false; }
 static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
 static inline struct page *pisa_virt_to_page(const void *addr) { return NULL; }
 
+static inline gfp_t pisa_rx_gfp(const struct device *dev)
+{
+	return 0;
+}
+
+static inline bool pisa_has_free_blocks_rx(const struct device *dev,
+					   unsigned int order, int nid)
+{
+	return false;
+}
+
 static inline bool pisa_page_can_recycle(const struct page *page)
 {
 	return true;
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 01d8bbc8d72ff..ff8565c3206ad 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -9,6 +9,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/cache.h>
+#include <linux/device.h>
 #include <linux/dma-mapping.h>
 #include <linux/pisa.h>
 #include <linux/export.h>
@@ -374,6 +375,26 @@ unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
 	return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
 }
 
+/**
+ * __pisa_has_free_blocks_rx - Check if per-node RX allocator of @order on @nid has free blocks
+ * @order: Block order
+ * @nid: Target NUMA node (or NUMA_NO_NODE)
+ *
+ * Return: true if the per-node RX allocator has a free block available, or
+ * the node's reservoir has a PISA page to give it.
+ */
+bool __pisa_has_free_blocks_rx(unsigned int order, int nid)
+{
+	struct pisa_alloc *pa = pisa_node_pa(nid, order, false);
+
+	return pa &&
+	       (!list_empty_careful(&pa->partial) ||
+		!list_empty_careful(&pa->partial_dirty) ||
+		!list_empty_careful(&pa->idle) ||
+		READ_ONCE(pisa_nodes[pa->nid].nr_pages));
+}
+EXPORT_SYMBOL(__pisa_has_free_blocks_rx);
+
 /*
  * Split a non-compound page of @old_order into 1 << (@old_order - @new_order)
  * pages of @new_order (compound if @new_order > 0), all frozen at refcount 0.
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index eb51007e66de5..2adaeb1ffe43e 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -6,6 +6,8 @@
 #include <linux/delay.h>
 #include <linux/gfp.h>
 #include <linux/pisa.h>
+#include <linux/kernfs.h>
+#include <linux/sysfs.h>
 #include <linux/iommu.h>
 #include <linux/iova.h>
 #include <linux/mm.h>
@@ -443,6 +445,74 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
 }
 #endif /* CONFIG_IOMMU_DMA */
 
+static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
+				  const char *name, bool *field)
+{
+	const struct sysfs_ops *ops = dev->kobj.ktype->sysfs_ops;
+	struct kernfs_node *kn;
+	char *buf;
+
+	buf = (char *)__get_free_page(GFP_KERNEL);
+	KUNIT_EXPECT_NOT_NULL(test, buf);
+	if (!buf)
+		return;
+	kn = kernfs_find_and_get(dev->kobj.sd, name);
+	KUNIT_EXPECT_NOT_NULL(test, kn);
+	if (!kn)
+		goto out_free;
+
+	KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+	KUNIT_EXPECT_STREQ(test, buf, "0\n");
+	KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+	KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "1\n", 2), 2);
+	KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+	KUNIT_EXPECT_EQ(test, ops->show(&dev->kobj, kn->priv, buf), 2);
+	KUNIT_EXPECT_STREQ(test, buf, "1\n");
+
+	KUNIT_EXPECT_LT(test, ops->store(&dev->kobj, kn->priv, "bad", 3), 0);
+	KUNIT_EXPECT_TRUE(test, READ_ONCE(*field));
+
+	KUNIT_EXPECT_EQ(test, ops->store(&dev->kobj, kn->priv, "0\n", 2), 2);
+	KUNIT_EXPECT_FALSE(test, READ_ONCE(*field));
+
+	kernfs_put(kn);
+out_free:
+	free_page((unsigned long)buf);
+}
+
+static void kunit_attr_dev_release(struct device *dev)
+{
+	kfree(dev);
+}
+
+static void test_dev_pisa_attrs(struct kunit *test)
+{
+	struct device *dev;
+
+	if (!IS_ENABLED(CONFIG_SYSFS))
+		kunit_skip(test, "requires CONFIG_SYSFS");
+
+	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, dev);
+	dev->init_name = "pisa_attr_kunit";
+	dev->release = kunit_attr_dev_release;
+	dev->coherent_dma_mask = DMA_BIT_MASK(64);
+	dev->dma_mask = &dev->coherent_dma_mask;
+
+	if (device_register(dev)) {
+		put_device(dev);
+		KUNIT_FAIL(test, "device_register failed");
+		return;
+	}
+	kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
+	kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
+	kunit_check_pisa_attr(test, dev, "pisa_tx_zerocopy", &dev->pisa_tx_zerocopy);
+	kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+	kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+	device_unregister(dev);
+}
+
 static void test_reservoir_and_scrubber(struct kunit *test)
 {
 	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -459,6 +529,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_iova_helpers),
 	KUNIT_CASE(test_map_and_domain_lifecycle),
 #endif
+	KUNIT_CASE(test_dev_pisa_attrs),
 	KUNIT_CASE(test_reservoir_and_scrubber),
 	{}
 };
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (6 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators Luigi Rizzo
                     ` (18 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Coherent DMA allocations (e.g. TX/RX descriptor rings allocated via
dma_alloc_attrs() / dma_alloc_coherent()) are long-lived and accessed
on every packet or I/O operation, so they benefit from sharing 2MB IOMMU
leaf mappings.

When dev->pisa_coherent is enabled, serve coherent DMA allocations
(dma_alloc_attrs()) up to 2MB with contiguous memory from PISA pages.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/iommu/dma-iommu.c  |   6 ++
 drivers/iommu/pisa-map.c   |  16 +++
 include/linux/pisa-iommu.h |   7 ++
 include/linux/pisa.h       |  38 +++++++
 kernel/dma/direct.c        |   7 ++
 mm/pisa-alloc.c            | 205 +++++++++++++++++++++++++++++++++++++
 mm/pisa-kunit.c            |  71 +++++++++++++
 mm/pisa-priv.h             |   3 +
 8 files changed, 353 insertions(+)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 9b26d0aee9a97..45e9432d2a020 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1653,6 +1653,8 @@ static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr)
 		dma_common_free_remap(cpu_addr, alloc_size);
 	} else {
 		/* Lowmem means a coherent atomic or CMA allocation */
+		if (pisa_free_coherent(cpu_addr, alloc_size))
+			return;
 		page = virt_to_page(cpu_addr);
 	}
 
@@ -1723,6 +1725,10 @@ void *iommu_dma_alloc(struct device *dev, size_t size, dma_addr_t *handle,
 
 	gfp |= __GFP_ZERO;
 
+	cpu_addr = pisa_alloc_map_coherent(dev, size, handle, gfp, attrs);
+	if (cpu_addr)
+		return cpu_addr;
+
 	if (gfpflags_allow_blocking(gfp) &&
 	    !(attrs & DMA_ATTR_FORCE_CONTIGUOUS)) {
 		return iommu_dma_alloc_remap(dev, size, handle, gfp, attrs);
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 3ff9f1a6c5b85..bbf330877f824 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -10,6 +10,7 @@
 #include <linux/dma-map-ops.h>
 #include <linux/dma-mapping.h>
 #include <linux/iommu.h>
+#include <linux/iommu-dma.h>
 #include <linux/iova.h>
 #include <linux/log2.h>
 #include <linux/mm.h>
@@ -18,6 +19,8 @@
 #include <linux/spinlock.h>
 #include <linux/srcu.h>
 
+#include "iommu-priv.h"
+
 
 /*
  * Registry of PISA IOMMU domains. Each slot is FREE (!domain && !base),
@@ -191,6 +194,19 @@ void __pisa_domain_quiesce(struct iommu_domain *domain)
 			 done, domain, idx);
 }
 
+/*
+ * Set *@domain to the DMA domain of @dev, or to NULL if @dev does not use
+ * dma-iommu. Returns an error if a deferred attach of @dev fails.
+ */
+int pisa_dev_dma_domain(struct device *dev, struct iommu_domain **domain)
+{
+	*domain = NULL;
+	if (!use_dma_iommu(dev))
+		return 0;
+	*domain = iommu_get_dma_domain(dev);
+	return iommu_deferred_attach(dev, *domain);
+}
+
 /* Return negative errno if @pi's domain was permanently/currently declined, or 0. */
 int pisa_iova_declined(const struct pisa_iova *pi)
 {
diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index d4c01207dca72..568108747f575 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -96,6 +96,7 @@ static inline dma_addr_t pisa_phys_to_iova(const struct pisa_iova *pi,
 extern unsigned int pisa_domains_generation;
 
 void pisa_unmap_all_domains(struct pisa_meta *meta);
+int pisa_dev_dma_domain(struct device *dev, struct iommu_domain **domain);
 int pisa_iova_declined(const struct pisa_iova *pi);
 
 #else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
@@ -103,5 +104,11 @@ int pisa_iova_declined(const struct pisa_iova *pi);
 static inline void pisa_unmap_all_domains(struct pisa_meta *meta) {}
 static inline int pisa_iova_declined(const struct pisa_iova *pi) { return -EOPNOTSUPP; }
 
+static inline int pisa_dev_dma_domain(struct device *dev, struct iommu_domain **domain)
+{
+	*domain = NULL;
+	return 0;
+}
+
 #endif /* CONFIG_PISA && CONFIG_IOMMU_DMA */
 #endif /* _LINUX_PISA_IOMMU_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 5222cb207257f..edd34771ad1c4 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -84,6 +84,9 @@ static inline u64 pisa_iova_size(void)
 }
 
 int pisa_meta_init(void);
+void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
+				    dma_addr_t *dma_handle, gfp_t gfp,
+				    unsigned long attrs);
 bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
 bool __pisa_page_can_recycle(const struct page *page);
 
@@ -143,6 +146,29 @@ static inline struct page *pisa_virt_to_page(const void *addr)
 	return is_pisa_page(page) ? page : NULL;
 }
 
+/* Hook for iommu_dma_alloc() and dma_direct_alloc() (guarded by @pisa_key). */
+static inline void *pisa_alloc_map_coherent(struct device *dev, size_t size,
+					    dma_addr_t *dma_handle, gfp_t gfp,
+					    unsigned long attrs)
+{
+	if (!pisa_enabled() || !READ_ONCE(dev->pisa_coherent))
+		return NULL;
+	return __pisa_alloc_map_coherent_dev(dev, size, dma_handle, gfp, attrs);
+}
+
+/* Hook for __iommu_dma_free() and dma_direct_free() (guarded by @pisa_key). */
+static inline bool pisa_free_coherent(const void *cpu_addr, size_t size)
+{
+	unsigned long count = PAGE_ALIGN(size) >> PAGE_SHIFT;
+	struct page *page = pisa_virt_to_page(cpu_addr);
+
+	if (!page)
+		return false;
+	while (count--)
+		__free_page(page++);
+	return true;
+}
+
 /* Hook for __page_pool_alloc_netmems_slow() (guarded by @pisa_key). */
 static inline gfp_t pisa_rx_gfp(const struct device *dev)
 {
@@ -174,6 +200,18 @@ static inline bool is_pisa_page(const struct page *page) { return false; }
 static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
 static inline struct page *pisa_virt_to_page(const void *addr) { return NULL; }
 
+static inline void *pisa_alloc_map_coherent(struct device *dev, size_t size,
+					    dma_addr_t *dma_handle, gfp_t gfp,
+					    unsigned long attrs)
+{
+	return NULL;
+}
+
+static inline bool pisa_free_coherent(const void *cpu_addr, size_t size)
+{
+	return false;
+}
+
 static inline gfp_t pisa_rx_gfp(const struct device *dev)
 {
 	return 0;
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 25e9a86835fd3..9ac4912e4a0a1 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -15,6 +15,7 @@
 #include <linux/slab.h>
 #include <linux/pci-p2pdma.h>
 #include <linux/cc_platform.h>
+#include <linux/pisa.h>
 
 #include "direct.h"
 
@@ -244,6 +245,10 @@ void *dma_direct_alloc(struct device *dev, size_t size,
 	if (attrs & DMA_ATTR_NO_WARN)
 		gfp |= __GFP_NOWARN;
 
+	cpu_addr = pisa_alloc_map_coherent(dev, size, dma_handle, gfp, attrs);
+	if (cpu_addr)
+		return cpu_addr;
+
 	if (dma_direct_use_no_mapping(dev, attrs))
 		return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp);
 
@@ -424,6 +429,8 @@ void dma_direct_free(struct device *dev, size_t size,
 
 	if (is_vmalloc_addr(cpu_addr)) {
 		vunmap(cpu_addr);
+	} else if (pisa_free_coherent(cpu_addr, size)) {
+		return;
 	} else {
 		if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED))
 			arch_dma_clear_uncached(cpu_addr, size);
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index ff8565c3206ad..eda12d475c9a7 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -10,11 +10,14 @@
 #include <linux/bitops.h>
 #include <linux/cache.h>
 #include <linux/device.h>
+#include <linux/dma-direct.h>
+#include <linux/dma-map-ops.h>
 #include <linux/dma-mapping.h>
 #include <linux/pisa.h>
 #include <linux/export.h>
 #include <linux/gfp.h>
 #include <linux/iommu.h>
+#include <linux/iommu-dma.h>
 #include <linux/kasan.h>
 #include <linux/list.h>
 #include <linux/llist.h>
@@ -25,6 +28,7 @@
 #include <linux/page_owner.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/page_table_check.h>
+#include <linux/pci.h>
 #include <linux/rcupdate.h>
 #include <linux/sched/mm.h>
 #include <linux/shrinker.h>
@@ -673,6 +677,207 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
 	return i;
 }
 
+/* Find and reserve @nblocks aligned contiguous blocks in @meta under @pa->lock. */
+static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblocks,
+				     unsigned long align_mask, bool *clean)
+{
+	DECLARE_BITMAP(busy, _PISA_MAX_BLOCKS);
+	struct pisa_alloc *pa = meta->pa;
+	unsigned int bpp = PISA_BLOCKS(0), dirty = 0;
+	bool poisoned = false;
+	unsigned long off;
+	struct page *page;
+
+	if (pisa_meta_avail(meta) < nblocks)
+		return NULL;
+
+	bitmap_or(busy, meta->clean_bitmap, meta->dirty_bitmap, bpp);
+	bitmap_complement(busy, busy, bpp);
+
+	off = 0;
+	for (;;) {
+		unsigned int i;
+
+		off = bitmap_find_next_zero_area(busy, bpp, off, nblocks, align_mask);
+		if (off >= bpp) {
+			if (poisoned) {
+				if (meta->on_list & PISA_LIST_IDLE) {
+					pa->num_idle_pages--;
+					WRITE_ONCE(meta->alloc_jiffies, jiffies);
+				}
+				pisa_place_meta(meta);
+			}
+			return NULL;
+		}
+
+		page = pisa_meta_to_page(meta) + off;
+		for (i = 0; i < nblocks; i++) {
+			if (!is_pisa_block_poisoned(pa, page + i))
+				continue;
+			if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
+				meta->nr_clean--;
+				pa->blocks_in_use++;
+			} else if (__test_and_clear_bit(off + i, meta->dirty_bitmap)) {
+				meta->nr_dirty--;
+				pa->blocks_in_use++;
+			}
+			__set_bit(off + i, busy);
+			poisoned = true;
+			off += i + 1;
+			break;
+		}
+		if (i == nblocks)
+			break;
+	}
+
+	if (meta->on_list & PISA_LIST_IDLE) {
+		pa->num_idle_pages--;
+		WRITE_ONCE(meta->alloc_jiffies, jiffies);
+	}
+
+	for (unsigned int i = 0; i < nblocks; i++) {
+		if (__test_and_clear_bit(off + i, meta->clean_bitmap)) {
+			meta->nr_clean--;
+		} else {
+			__clear_bit(off + i, meta->dirty_bitmap);
+			meta->nr_dirty--;
+			dirty++;
+		}
+		pisa_block_prep(page + i);
+		page_ref_unfreeze(page + i, 1);
+	}
+	pa->blocks_in_use += nblocks;
+	*clean = !dirty;
+	pisa_place_meta(meta);
+	return page;
+}
+
+/* The device checks of pisa_domain_init() and __pisa_map_phys(), before allocating. */
+static bool pisa_coherent_dev_fits(struct device *dev, const struct pisa_iova *pi)
+{
+	u64 limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
+	/* Pairs with smp_store_release() in pisa_domain_init(). */
+	dma_addr_t base = smp_load_acquire(&pi->base);
+
+	limit = min_not_zero(limit, dev->coherent_dma_mask);
+	if (limit <= DMA_BIT_MASK(32))
+		return false;
+	return base ? base + pisa_iova_size() - 1 <= limit :
+		      pisa_iova_size() <= limit - PISA_SIZE;
+}
+
+/*
+ * Specialized memory provider for iommu_dma_alloc() and dma_direct_alloc():
+ * allocate a zeroed, naturally aligned contiguous range of order-0 pages for
+ * @size bytes (<= PISA_SIZE) from @dev's per-node coherent allocator, map it
+ * into *@dma_handle, and return its kernel virtual address, or NULL.
+ */
+VISIBLE_IF_KUNIT void *
+__pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
+			  size_t size, dma_addr_t *dma_handle, gfp_t gfp,
+			  unsigned long attrs)
+{
+	bool should_release = false, clean = false;
+	struct list_head *lists[3];
+	unsigned long align_mask, flags;
+	struct pisa_meta *meta, *tmp;
+	unsigned int nblocks;
+	struct pisa_alloc *pa;
+	struct page *page = NULL;
+
+	size = PAGE_ALIGN(size);
+	if (!dev_is_dma_coherent(dev) || !size || size > PISA_SIZE)
+		return NULL;
+	/* As in pisa_domain_init(): no shared PISA pages for untrusted devices. */
+	if (dev_is_pci(dev) && to_pci_dev(dev)->untrusted)
+		return NULL;
+	if (min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) <= DMA_BIT_MASK(32))
+		return NULL;
+	if ((current_gfp_context(gfp) & ~(__GFP_ZERO | __GFP_NOWARN)) != GFP_KERNEL ||
+	    (attrs & ~(DMA_ATTR_NO_WARN | __DMA_ATTR_ALLOC_CC_SHARED)))
+		return NULL;
+
+	if (domain) {
+		struct pisa_iova *pi = pisa_dma_iova(domain);
+
+		if (!pi || pisa_iova_declined(pi) || !pisa_domain_pgsize_ok(domain) ||
+		    !pisa_coherent_dev_fits(dev, pi))
+			return NULL;
+	} else if (is_swiotlb_for_alloc(dev) ||
+		   force_dma_unencrypted(dev) ||
+		   !dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
+		return NULL;
+	}
+
+	pa = pisa_node_pa(dev_to_node(dev), 0, false);
+	if (!pa)
+		return NULL;
+
+	nblocks = size >> PAGE_SHIFT;
+	align_mask = (1UL << get_order(size)) - 1;
+	lists[0] = &pa->partial;
+	lists[1] = &pa->partial_dirty;
+	lists[2] = &pa->idle;
+
+	spin_lock_irqsave(&pa->lock, flags);
+	for (unsigned int i = 0; i < ARRAY_SIZE(lists) && !page; i++) {
+		list_for_each_entry_safe(meta, tmp, lists[i], list) {
+			page = pisa_take_contig(meta, nblocks, align_mask, &clean);
+			if (page)
+				break;
+		}
+	}
+	if (!page) {
+		spin_unlock_irqrestore(&pa->lock, flags);
+		meta = pisa_alloc_split_page(0, gfp, pa->nid);
+		if (IS_ERR(meta))
+			return NULL;
+		spin_lock_irqsave(&pa->lock, flags);
+		meta->pa = pa;
+		meta->prot = pa->prot;
+		WRITE_ONCE(meta->alloc_jiffies, jiffies);
+		pisa_meta_set_flags(meta, PISA_ACTIVE);
+		pa->stats.page_alloc++;
+		page = pisa_take_contig(meta, nblocks, align_mask, &clean);
+		if (unlikely(!page))
+			should_release = pisa_place_meta(meta);
+	}
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (should_release)
+		pisa_schedule_reclaim();
+	if (!page)
+		return NULL;
+	if (!domain && unlikely(!dma_coherent_ok(dev, page_to_phys(page), size))) {
+		/* E.g. a hole in dev->dma_range_map, or RAM hotplugged above max_pfn. */
+		pisa_free_coherent(page_address(page), size);
+		return NULL;
+	}
+	if (domain) {
+		if (!pisa_map_phys(dma_handle, dev, domain, page_to_phys(page),
+				   size, IOMMU_READ | IOMMU_WRITE | IOMMU_CACHE)) {
+			pisa_free_coherent(page_address(page), size);
+			return NULL;
+		}
+	} else {
+		*dma_handle = phys_to_dma(dev, page_to_phys(page));
+	}
+	if (!clean)
+		memset(page_address(page), 0, size);
+	return page_address(page);
+}
+
+void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
+				    dma_addr_t *dma_handle, gfp_t gfp,
+				    unsigned long attrs)
+{
+	struct iommu_domain *domain;
+
+	if (pisa_dev_dma_domain(dev, &domain))
+		return NULL;
+	return __pisa_alloc_map_coherent(dev, domain, size, dma_handle, gfp, attrs);
+}
+
 /**
  * __pisa_free_page - Recycle a freed block back into its PISA allocator
  * @page: Block being freed (called from __free_pages_prepare() under RCU)
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 2adaeb1ffe43e..1040786330cef 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -13,6 +13,7 @@
 #include <linux/mm.h>
 #include <linux/shrinker.h>
 #include <linux/skbuff.h>
+#include <linux/swiotlb.h>
 #include <net/sock.h>
 
 #include "pisa-priv.h"
@@ -445,6 +446,75 @@ static void test_map_and_domain_lifecycle(struct kunit *test)
 }
 #endif /* CONFIG_IOMMU_DMA */
 
+static void test_coherent_alloc_and_free(struct kunit *test)
+{
+	struct device dev = { .init_name = "pisa_kunit", .numa_node = NUMA_NO_NODE,
+			      .coherent_dma_mask = DMA_BIT_MASK(64),
+			      .pisa_coherent = true };
+	dma_addr_t d1, d2, d3;
+	void *v1, *v2, *v3 = NULL;
+#ifdef CONFIG_IOMMU_DMA
+	struct pisa_iova *pi;
+#endif
+
+	dev.dma_mask = &dev.coherent_dma_mask;
+	/* NUMA_NO_NODE would follow the node of the CPU the test runs on. */
+	set_dev_node(&dev, first_memory_node);
+	/* is_swiotlb_for_alloc() dereferences dev->dma_io_tlb_mem. */
+	swiotlb_dev_init(&dev);
+	dev_set_dma_coherent(&dev);
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	KUNIT_EXPECT_NULL(test, pisa_alloc_map_coherent(&dev, SZ_4K, &d1,
+							GFP_KERNEL | __GFP_ACCOUNT, 0));
+	KUNIT_EXPECT_NULL(test, pisa_alloc_map_coherent(&dev, PISA_SIZE + SZ_4K, &d1,
+							GFP_KERNEL, 0));
+
+#ifdef CONFIG_IOMMU_DMA
+	/* Uninitialized domain has no DMA cookie/window yet. */
+	KUNIT_EXPECT_NULL(test, __pisa_alloc_map_coherent(&dev, &fake_domain, SZ_4K,
+							  &d1, GFP_KERNEL | __GFP_NOWARN, 0));
+
+	KUNIT_ASSERT_EQ(test, fake_domain_setup(test), 0);
+	pi = pisa_dma_iova(&fake_domain);
+
+	v1 = __pisa_alloc_map_coherent(&dev, &fake_domain, SZ_64K, &d1, GFP_KERNEL, 0);
+	KUNIT_ASSERT_NOT_NULL(test, v1);
+	KUNIT_EXPECT_TRUE(test, is_pisa_iova(pi, d1));
+
+	v2 = __pisa_alloc_map_coherent(&dev, &fake_domain, SZ_64K, &d2, GFP_KERNEL, 0);
+	KUNIT_EXPECT_NOT_NULL(test, v2);
+	if (v2) {
+		KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+		KUNIT_EXPECT_EQ(test, d2, d1 + SZ_64K);
+	}
+
+	KUNIT_EXPECT_TRUE(test, pisa_free_coherent(v1, SZ_64K));
+	v3 = __pisa_alloc_map_coherent(&dev, &fake_domain, SZ_64K, &d3, GFP_KERNEL, 0);
+	KUNIT_EXPECT_NOT_NULL(test, v3);
+	if (v3)
+		KUNIT_EXPECT_PTR_EQ(test, v3, v1);
+#else
+	v1 = pisa_alloc_map_coherent(&dev, SZ_64K, &d1, GFP_KERNEL, 0);
+	KUNIT_ASSERT_NOT_NULL(test, v1);
+	v2 = pisa_alloc_map_coherent(&dev, SZ_64K, &d2, GFP_KERNEL, 0);
+	KUNIT_EXPECT_NOT_NULL(test, v2);
+	if (v2)
+		KUNIT_EXPECT_PTR_EQ(test, v2, v1 + SZ_64K);
+
+	/* Freeing v1 returns its 16 order-0 pages; next 64K alloc reuses them. */
+	KUNIT_EXPECT_TRUE(test, pisa_free_coherent(v1, SZ_64K));
+	v3 = pisa_alloc_map_coherent(&dev, SZ_64K, &d3, GFP_KERNEL, 0);
+	KUNIT_EXPECT_NOT_NULL(test, v3);
+	if (v3)
+		KUNIT_EXPECT_PTR_EQ(test, v3, v1);
+#endif
+
+	if (v2)
+		KUNIT_EXPECT_TRUE(test, pisa_free_coherent(v2, SZ_64K));
+	if (v3)
+		KUNIT_EXPECT_TRUE(test, pisa_free_coherent(v3, SZ_64K));
+}
+
 static void kunit_check_pisa_attr(struct kunit *test, struct device *dev,
 				  const char *name, bool *field)
 {
@@ -529,6 +599,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_iova_helpers),
 	KUNIT_CASE(test_map_and_domain_lifecycle),
 #endif
+	KUNIT_CASE(test_coherent_alloc_and_free),
 	KUNIT_CASE(test_dev_pisa_attrs),
 	KUNIT_CASE(test_reservoir_and_scrubber),
 	{}
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 8736b8d168727..fb26eb3aa9b37 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -165,6 +165,9 @@ struct shrink_control;
 struct shrinker;
 #if IS_ENABLED(CONFIG_KUNIT)
 unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc);
+void *__pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
+				size_t size, dma_addr_t *dma_handle, gfp_t gfp,
+				unsigned long attrs);
 #endif
 
 /* Global PISA page budget across all allocators and per-node reservoirs. */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (7 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 10/25] mm: Add background page scrubber for " Luigi Rizzo
                     ` (17 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Every block allocation and free takes the allocator's lock. Highly
concurrent users e.g. socket tx fragments, can cause significant lock
contention.

Add pisa_enable_pcp(), an opt-in per-CPU front cache, similar
to the page allocator's per-CPU lists:

- Each per-CPU cache keeps up to 64 free blocks, and refills it from
  the main allocator in batches of 32.
- returned blocks belonging to the local NUMA node are kept in the local cache.
- returned blocks belonging to a remote NUMA node are returned to the
  proper per-node allocator.
The cache is fully integrated with the shrinker and CPU offlining to
drain the cache.

Enable per-CPU caches for the per-node order-0 RW and SKB_FRAG_PAGE_ORDER RO
allocators.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 mm/pisa-alloc.c | 272 ++++++++++++++++++++++++++++++++++++++++++++++--
 mm/pisa-kunit.c |  79 ++++++++++++++
 mm/pisa-priv.h  |  11 +-
 3 files changed, 348 insertions(+), 14 deletions(-)

diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index eda12d475c9a7..896709303f258 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -9,6 +9,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/cache.h>
+#include <linux/cpuhotplug.h>
 #include <linux/device.h>
 #include <linux/dma-direct.h>
 #include <linux/dma-map-ops.h>
@@ -26,6 +27,7 @@
 #include <linux/moduleparam.h>
 #include <linux/mutex.h>
 #include <linux/page_owner.h>
+#include <linux/percpu.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/page_table_check.h>
 #include <linux/pci.h>
@@ -324,13 +326,168 @@ static bool pisa_return_block_locked(struct pisa_alloc *pa, struct page *block)
 	return pisa_place_meta(meta);
 }
 
+/* Return @n freed blocks to @pa under a single acquisition of @pa->lock. */
+static void pisa_free_bulk(struct pisa_alloc *pa, unsigned int n, struct page **blocks)
+{
+	bool released = false;
+	unsigned long flags;
+
+	spin_lock_irqsave(&pa->lock, flags);
+	for (unsigned int i = 0; i < n; i++) {
+		if (likely(!is_pisa_block_poisoned(pa, blocks[i])))
+			released |= pisa_return_block_locked(pa, blocks[i]);
+	}
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (released)
+		pisa_schedule_reclaim();
+}
+
+/*
+ * Per-CPU block cache: each CPU caches up to PISA_PCP_HIGH free blocks
+ * (refcount 0, not in bitmaps), refilling and flushing in PISA_PCP_BATCH
+ * batches. Drained by the shrinker and CPU hotplug.
+ */
+#define PISA_PCP_BATCH	32U	/* blocks moved per refill or flush */
+
+struct pisa_pcp {
+	spinlock_t	lock;	/* by this CPU, or by a drain from any CPU */
+	unsigned int	count;
+	struct page	*blocks[PISA_PCP_HIGH];
+};
+
+/* Max blocks moved per per-CPU cache refill or flush (1..PISA_PCP_BATCH). */
+static unsigned int pisa_pcp_batch __read_mostly = PISA_PCP_BATCH;
+
+static int pisa_pcp_batch_set(const char *val, const struct kernel_param *kp)
+{
+	return param_set_uint_minmax(val, kp, 1, PISA_PCP_BATCH);
+}
+
+static const struct kernel_param_ops pisa_pcp_batch_ops = {
+	.set = pisa_pcp_batch_set,
+	.get = param_get_uint,
+};
+module_param_cb(pcp_batch, &pisa_pcp_batch_ops, &pisa_pcp_batch, 0644);
+
+static inline unsigned int pisa_pcp_batch_order(const struct pisa_alloc *pa)
+{
+	return max(1U, READ_ONCE(pisa_pcp_batch) >> pa->order);
+}
+
+/* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
+static void pisa_pcp_push(struct pisa_alloc *pa, unsigned int n, struct page **blocks)
+{
+	unsigned int batch = pisa_pcp_batch_order(pa);
+	unsigned int high = max(2U, PISA_PCP_HIGH >> pa->order);
+	struct pisa_pcp *pcp;
+	unsigned long flags;
+
+	/* @pcp->lock protects the cache: migrating to another CPU here is harmless. */
+	pcp = raw_cpu_ptr(pa->pcp);
+	spin_lock_irqsave(&pcp->lock, flags);
+	for (unsigned int i = 0; i < n; i++) {
+		if (pcp->count >= high) {
+			/* Keep the most recently freed, likely still in cache. */
+			pisa_free_bulk(pa, batch, pcp->blocks);
+			pcp->count -= batch;
+			memmove(pcp->blocks, pcp->blocks + batch,
+				pcp->count * sizeof(*pcp->blocks));
+		}
+		pcp->blocks[pcp->count++] = blocks[i];
+	}
+	spin_unlock_irqrestore(&pcp->lock, flags);
+}
+
+/* Take up to @nr blocks, at refcount 0, from this CPU's cache. */
+static unsigned int pisa_pcp_pop(struct pisa_alloc *pa, unsigned int nr, struct page **out)
+{
+	struct pisa_pcp *pcp;
+	unsigned int got = 0;
+	unsigned long flags;
+
+	/* @pcp->lock protects the cache: migrating to another CPU here is harmless. */
+	pcp = raw_cpu_ptr(pa->pcp);
+	spin_lock_irqsave(&pcp->lock, flags);
+	while (got < nr && pcp->count)
+		out[got++] = pcp->blocks[--pcp->count];
+	spin_unlock_irqrestore(&pcp->lock, flags);
+
+	return got;
+}
+
+/* Return the blocks @cpu caches for @pa. */
+static void pisa_pcp_drain_cpu(struct pisa_alloc *pa, int cpu)
+{
+	struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+
+	if (!READ_ONCE(pcp->count))
+		return;
+	guard(spinlock_irqsave)(&pcp->lock);
+	if (pcp->count) {
+		pisa_free_bulk(pa, pcp->count, pcp->blocks);
+		pcp->count = 0;
+	}
+}
+
+/* Drain every non-empty CPU cache of @pa. */
+void pisa_pcp_drain(struct pisa_alloc *pa)
+{
+	int cpu;
+
+	/* Pairs with smp_store_release() in pisa_enable_pcp(). */
+	if (!smp_load_acquire(&pa->pcp))
+		return;
+	for_each_possible_cpu(cpu)
+		pisa_pcp_drain_cpu(pa, cpu);
+}
+
+/* Blocks in @pa's caches, for shrinker and debugfs. */
+static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+{
+	unsigned long n = 0;
+	int cpu;
+
+	if (!pa->pcp)
+		return 0;
+	for_each_possible_cpu(cpu) {
+		/* A racy sum is good enough for reclaim and statistics. */
+		n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+	}
+	return n;
+}
+
+/**
+ * pisa_enable_pcp - Put per-CPU block caches in front of an allocator
+ * @pa: Allocator to cache
+ *
+ * Return: 0, or -ENOMEM; @pa works without the cache.
+ */
+int pisa_enable_pcp(struct pisa_alloc *pa)
+{
+	struct pisa_pcp __percpu *pcp;
+	int cpu;
+
+	if (!pa || pa->pcp)
+		return 0;
+	pcp = alloc_percpu(struct pisa_pcp);
+	if (!pcp)
+		return -ENOMEM;
+	for_each_possible_cpu(cpu)
+		spin_lock_init(&per_cpu_ptr(pcp, cpu)->lock);
+	/* Publish initialized locks; readers depend on the pointer. */
+	if (cmpxchg_release(&pa->pcp, NULL, pcp))
+		free_percpu(pcp);
+	return 0;
+}
+
 static DEFINE_MUTEX(pisa_init_mutex);
 
 /* Return *@pap, creating and publishing it under @pisa_init_mutex if NULL. */
-__printf(5, 6)
+__printf(6, 7)
 struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
 				     unsigned int order, enum dma_data_direction dir,
-				     const char *fmt, ...)
+				     bool pcp, const char *fmt, ...)
 {
 	/* Pairs with smp_store_release() below. */
 	struct pisa_alloc *pa = smp_load_acquire(pap);
@@ -349,6 +506,8 @@ struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
 		va_start(args, fmt);
 		vsnprintf(pa->name, sizeof(pa->name), fmt, args);
 		va_end(args);
+		if (pcp)
+			pisa_enable_pcp(pa);
 		/* Pairs with smp_load_acquire() in lockless readers. */
 		smp_store_release(pap, pa);
 	}
@@ -392,7 +551,9 @@ bool __pisa_has_free_blocks_rx(unsigned int order, int nid)
 	struct pisa_alloc *pa = pisa_node_pa(nid, order, false);
 
 	return pa &&
-	       (!list_empty_careful(&pa->partial) ||
+	       /* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+	       ((smp_load_acquire(&pa->pcp) && raw_cpu_read(pa->pcp->count)) ||
+		!list_empty_careful(&pa->partial) ||
 		!list_empty_careful(&pa->partial_dirty) ||
 		!list_empty_careful(&pa->idle) ||
 		READ_ONCE(pisa_nodes[pa->nid].nr_pages));
@@ -636,6 +797,58 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 	return n;
 }
 
+/* Allocate @nr_pages blocks through this CPU's cache, refilling in batches on miss. */
+static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
+				    struct page **page_array, bool frozen)
+{
+	unsigned int pcp_batch = pisa_pcp_batch_order(pa);
+	unsigned long got, j = 0;
+
+	got = pisa_pcp_pop(pa, nr_pages, page_array);
+	if (got < nr_pages) {
+		if (nr_pages - got >= pcp_batch) {
+			got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
+						 page_array + got, false, true);
+		} else {
+			struct page *batch[PISA_PCP_BATCH];
+			unsigned long n, i;
+
+			n = __pisa_alloc_bulk(pa, gfp, pcp_batch, batch, false, true);
+			for (i = 0; i < n && got < nr_pages; i++)
+				page_array[got++] = batch[i];
+			if (i < n) {
+				if (pa->nid == numa_mem_id())
+					pisa_pcp_push(pa, n - i, batch + i);
+				else
+					pisa_free_bulk(pa, n - i, batch + i);
+			}
+		}
+	}
+
+	for (unsigned long i = 0; i < got; i++) {
+		struct page *block = page_array[i];
+
+		page_array[i] = NULL;
+		/* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
+		if (is_pisa_block_poisoned(pa, block))
+			continue;
+		pisa_block_prep(block);
+		if (want_init_on_alloc(gfp)) {
+			memset(page_address(block), 0, PAGE_SIZE << pa->order);
+			if (frozen) /* Order zeroing before caller publishes the page. */
+				smp_wmb();
+		}
+		if (!frozen)
+			page_ref_unfreeze(block, 1);
+		page_array[j++] = block;
+	}
+	/* Replace blocks dropped as poisoned, so that callers do not see a short allocation. */
+	if (unlikely(j < got))
+		j += __pisa_alloc_bulk(pa, gfp, nr_pages - j, page_array + j,
+				       want_init_on_alloc(gfp), frozen);
+	return j;
+}
+
 /**
  * pisa_alloc_bulk - Allocate up to @nr_pages blocks from @pa
  * @pa: Allocator to allocate from
@@ -665,11 +878,16 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
 			i++;
 			continue;
 		}
+
 		/* Fill the run of @n NULL entries at @i. */
 		for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
 			;
-		got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
-					want_init_on_alloc(gfp), frozen);
+		/* Pairs with smp_store_release() in pisa_enable_pcp(). */
+		if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id())
+			got = pisa_pcp_alloc(pa, gfp, n, page_array + i, frozen);
+		else
+			got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
+						want_init_on_alloc(gfp), frozen);
 		i += got;
 		if (got < n)
 			break;
@@ -812,6 +1030,7 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 	pa = pisa_node_pa(dev_to_node(dev), 0, false);
 	if (!pa)
 		return NULL;
+	pisa_pcp_drain(pa);
 
 	nblocks = size >> PAGE_SHIFT;
 	align_mask = (1UL << get_order(size)) - 1;
@@ -911,6 +1130,16 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 	if (want_init_on_free())
 		memset(page_address(page), 0, PAGE_SIZE << pa->order);
 
+	/* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+	if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
+	    page_to_nid(page) == numa_mem_id()) {
+		if (pisa_page_can_recycle(page)) {
+			pisa_pcp_push(pa, 1, &page);
+			return true;
+		}
+		pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
+	}
+
 	spin_lock_irqsave(&pa->lock, flags);
 	should_release = pisa_return_block_locked(pa, page);
 	spin_unlock_irqrestore(&pa->lock, flags);
@@ -924,14 +1153,16 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 /* Shrinker: releases reservoir pages first, then idle allocator pages. */
 static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
 {
-	unsigned long nr = pisa_reservoir_count();
+	unsigned long nr = pisa_reservoir_count() << PISA_ORDER;
 	struct pisa_alloc *pa;
 
 	guard(read_lock)(&pisa_allocs_lock);
-	list_for_each_entry(pa, &pisa_allocs, node)
-		nr += READ_ONCE(pa->num_idle_pages);
+	list_for_each_entry(pa, &pisa_allocs, node) {
+		nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
+		nr += pisa_pcp_count(pa) << pa->order;
+	}
 
-	return nr << PISA_ORDER;
+	return nr;
 }
 
 /* Detach PISA pages from @list onto @release_list up to @nr_to_scan base pages. */
@@ -967,6 +1198,8 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
 	if (freed < sc->nr_to_scan) {
 		read_lock(&pisa_allocs_lock);
 		list_for_each_entry(pa, &pisa_allocs, node) {
+			/* Cached blocks keep PISA pages from going idle. */
+			pisa_pcp_drain(pa);
 			freed += pisa_shrink_list(&pa->lock, &pa->idle,
 						  &pa->num_idle_pages, &release_list,
 						  sc->nr_to_scan - freed);
@@ -989,6 +1222,19 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
 	return freed ?: SHRINK_STOP;
 }
 
+/* Return the blocks a dead CPU cached, which nothing else would. */
+static int pisa_cpu_dead(unsigned int cpu)
+{
+	struct pisa_alloc *pa;
+
+	guard(read_lock)(&pisa_allocs_lock);
+	list_for_each_entry(pa, &pisa_allocs, node)
+		/* Pairs with smp_store_release() in pisa_enable_pcp(). */
+		if (smp_load_acquire(&pa->pcp))
+			pisa_pcp_drain_cpu(pa, cpu);
+	return 0;
+}
+
 static int __init pisa_init(void)
 {
 	struct shrinker *shrink;
@@ -1001,9 +1247,9 @@ static int __init pisa_init(void)
 	/* Preallocate the supported per-node allocators. */
 	for_each_node_state(nid, N_MEMORY) {
 		pisa_alloc_ensure(&pisa_nodes[nid].pa[0][0], nid, 0,
-				  DMA_BIDIRECTIONAL, "pisa_o0_n%d", nid);
+				  DMA_BIDIRECTIONAL, true, "pisa_o0_n%d", nid);
 		pisa_alloc_ensure(&pisa_nodes[nid].pa[1][SKB_FRAG_PAGE_ORDER],
-				  nid, SKB_FRAG_PAGE_ORDER, DMA_TO_DEVICE,
+				  nid, SKB_FRAG_PAGE_ORDER, DMA_TO_DEVICE, true,
 				  "pisa_ro_o%u_n%d", SKB_FRAG_PAGE_ORDER, nid);
 	}
 
@@ -1012,6 +1258,10 @@ static int __init pisa_init(void)
 	if (!pisa_wq)
 		pr_warn("pisa: no reclaim workqueue, falling back to system_wq\n");
 
+	if (cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN, "iommu/pisa:dead", NULL,
+				      pisa_cpu_dead) < 0)
+		pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
+
 	shrink = shrinker_alloc(0, "pisa");
 	if (!shrink) {
 		pr_warn("pisa: shrinker registration failed\n");
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 1040786330cef..0f8cc9baf5d28 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -583,6 +583,83 @@ static void test_dev_pisa_attrs(struct kunit *test)
 	device_unregister(dev);
 }
 
+static void test_pcp_cache(struct kunit *test)
+{
+	struct page *blocks[PISA_PCP_HIGH + 1];
+	struct pisa_alloc *pa;
+	struct page *p1, *p2;
+	unsigned int i, n = 0;
+
+	KUNIT_EXPECT_EQ(test, pisa_enable_pcp(NULL), 0);
+	pa = pisa_node_pa(numa_mem_id(), 0, false);
+	KUNIT_ASSERT_NOT_NULL(test, pa);
+	KUNIT_ASSERT_EQ(test, pisa_enable_pcp(pa), 0);
+
+	p1 = pisa_alloc(pa, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, p1);
+	__free_page(p1);
+	p2 = pisa_alloc(pa, GFP_KERNEL | __GFP_ZERO);
+	KUNIT_EXPECT_PTR_EQ(test, p2, p1);
+	if (p2)
+		__free_page(p2);
+
+	/* Overflowing PISA_PCP_HIGH flushes a batch back to the allocator. */
+	for (i = 0; i < ARRAY_SIZE(blocks); i++) {
+		blocks[i] = pisa_alloc(pa, GFP_KERNEL);
+		if (!blocks[i])
+			break;
+		n++;
+	}
+	KUNIT_EXPECT_EQ(test, n, (unsigned int)ARRAY_SIZE(blocks));
+	for (i = 0; i < n; i++)
+		__free_page(blocks[i]);
+}
+
+static void test_gfp_pisa(struct kunit *test)
+{
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+	struct device dev = { .numa_node = first_memory_node };
+	struct page *p, *p_ro, *bulk[4] = {};
+	unsigned long n;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	pisa_shrink_scan(NULL, &sc);
+	KUNIT_EXPECT_EQ(test, pisa_rx_gfp(NULL), 0);
+	KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), 0);
+	KUNIT_EXPECT_FALSE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+
+	dev.pisa_rx_bufs = true;
+	KUNIT_EXPECT_EQ(test, pisa_rx_gfp(&dev), __GFP_PISA_RW);
+
+	p = alloc_pages_node(first_memory_node, GFP_KERNEL | pisa_rx_gfp(&dev), 0);
+	KUNIT_ASSERT_NOT_NULL(test, p);
+	if (p) {
+		KUNIT_EXPECT_TRUE(test, is_pisa_page(p));
+		KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p))->prot,
+				IOMMU_READ | IOMMU_WRITE);
+		__free_page(p);
+		KUNIT_EXPECT_TRUE(test, pisa_has_free_blocks_rx(&dev, 0, first_memory_node));
+	}
+
+	p_ro = alloc_pages(GFP_KERNEL | __GFP_COMP | __GFP_PISA,
+			   SKB_FRAG_PAGE_ORDER);
+	KUNIT_ASSERT_NOT_NULL(test, p_ro);
+	if (p_ro) {
+		KUNIT_EXPECT_TRUE(test, is_pisa_page(p_ro));
+		KUNIT_EXPECT_EQ(test, pisa_meta_of_pfn(page_to_pfn(p_ro))->prot,
+				IOMMU_READ);
+		__free_pages(p_ro, SKB_FRAG_PAGE_ORDER);
+	}
+
+	n = alloc_pages_bulk_node(GFP_KERNEL | __GFP_PISA_RW, first_memory_node,
+				  ARRAY_SIZE(bulk), bulk);
+	KUNIT_EXPECT_EQ(test, n, ARRAY_SIZE(bulk));
+	for (int i = 0; i < n; i++) {
+		KUNIT_EXPECT_TRUE(test, is_pisa_page(bulk[i]));
+		__free_page(bulk[i]);
+	}
+}
+
 static void test_reservoir_and_scrubber(struct kunit *test)
 {
 	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
@@ -601,6 +678,8 @@ static struct kunit_case pisa_meta_test_cases[] = {
 #endif
 	KUNIT_CASE(test_coherent_alloc_and_free),
 	KUNIT_CASE(test_dev_pisa_attrs),
+	KUNIT_CASE(test_pcp_cache),
+	KUNIT_CASE(test_gfp_pisa),
 	KUNIT_CASE(test_reservoir_and_scrubber),
 	{}
 };
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index fb26eb3aa9b37..60120a05082a9 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -15,6 +15,7 @@
 #include <linux/topology.h>
 #include <linux/workqueue.h>
 
+struct pisa_pcp;
 
 #ifdef CONFIG_IOMMU_DMA
 int iommu_get_dma_cookie(struct iommu_domain *domain);
@@ -30,7 +31,7 @@ void iommu_put_dma_cookie(struct iommu_domain *domain);
  * Locking (disjoint groups; locks in different groups do not nest):
  *
  * 1. Allocators (mm/pisa-alloc.c, mm/pisa-node.c):
- *    pisa_allocs_lock (rwlock) -> pa->lock (spinlock)
+ *    pisa_allocs_lock (rwlock) -> pcp->lock (spinlock) -> pa->lock (spinlock)
  *    pisa_nodes[].lock (spinlock): leaf lock over a node's reservoir.
  *
  * 2. Metadata tables (mm/pisa-meta.c):
@@ -100,6 +101,7 @@ struct pisa_alloc {
 	spinlock_t		lock;
 	u8			order;		/* block order (<= PISA_ORDER) */
 	u8			prot;		/* IOMMU_READ and/or IOMMU_WRITE */
+	struct pisa_pcp __percpu *pcp;
 	struct list_head	partial;	/* partial pages with nr_clean > 0 */
 	struct list_head	partial_dirty;	/* partial pages with nr_clean == 0 */
 	struct list_head	idle;		/* completely free PISA pages */
@@ -200,10 +202,13 @@ static inline void pisa_alloc_backoff(struct pisa_node *node, bool can_block)
 struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir);
 __printf(2, 3)
 void pisa_alloc_set_name(struct pisa_alloc *pa, const char *fmt, ...);
-__printf(5, 6)
+__printf(6, 7)
 struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
 				     unsigned int order, enum dma_data_direction dir,
-				     const char *fmt, ...);
+				     bool pcp, const char *fmt, ...);
+#define PISA_PCP_HIGH		64U	/* blocks cached per CPU at most */
+int pisa_enable_pcp(struct pisa_alloc *pa);
+void pisa_pcp_drain(struct pisa_alloc *pa);
 bool pisa_place_meta(struct pisa_meta *meta);
 extern struct llist_head pisa_free_list;
 extern unsigned int pisa_idle_ms;
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 10/25] mm: Add background page scrubber for PISA allocators
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (8 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
                     ` (16 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Blocks returned to a pisa_alloc are marked dirty and need to be zeroed
before they can serve __GFP_ZERO requests. Zeroing blocks inline at
allocation time adds CPU cost and latency.

Add an optional background scrubber (enabled via pisa.scrub_max) that
runs on the allocator's home NUMA node work item, zeroes dirty blocks
outside pa->lock in batches, and moves them to meta->clean_bitmap[].
Scrubbing is demand-driven by the number of dirty blocks that __GFP_ZERO
allocations had to zero inline.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa-iommu.h |   1 +
 mm/pisa-alloc.c            | 187 ++++++++++++++++++++++++++++++++++++-
 mm/pisa-kunit.c            |  51 +++++++++-
 mm/pisa-meta.c             |   7 +-
 mm/pisa-node.c             |  33 +++++++
 mm/pisa-priv.h             |  10 +-
 6 files changed, 281 insertions(+), 8 deletions(-)

diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index 568108747f575..7dd91ff47288a 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -55,6 +55,7 @@ struct pisa_meta {
 	u8 on_list;		/* current placement in @pa (PISA_LIST_*) */
 	bool drain;		/* sparse old page draining back to @idle */
 	u32 slot;		/* dense index in @pisa_meta_array and IOVA windows */
+	u16 nr_scrub;		/* blocks being zeroed by the scrubber, in neither bitmap */
 
 	DECLARE_BITMAP(domains_mapped, PISA_MAX_DOMAINS);
 
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 896709303f258..ccba2bf01bd59 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -32,6 +32,7 @@
 #include <linux/page_table_check.h>
 #include <linux/pci.h>
 #include <linux/rcupdate.h>
+#include <linux/sched/clock.h>
 #include <linux/sched/mm.h>
 #include <linux/shrinker.h>
 #include <linux/skbuff.h>
@@ -80,6 +81,10 @@ static unsigned int pisa_max_idle __read_mostly = 1;
 unsigned int pisa_idle_ms __read_mostly = 200;
 module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
 
+/* Max pages zeroed in each scrubber run (0 = disabled, capped at PISA_SCRUB_MAX_LIMIT). */
+unsigned int pisa_scrub_max __read_mostly;
+#define PISA_SCRUB_MAX_LIMIT	65536U
+
 /* All live allocators, protected by pisa_allocs_lock. */
 static LIST_HEAD(pisa_allocs);
 static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -113,6 +118,25 @@ static const struct kernel_param_ops pisa_max_idle_ops = {
 };
 module_param_cb(max_idle, &pisa_max_idle_ops, &pisa_max_idle, 0644);
 
+static int pisa_scrub_set(const char *val, const struct kernel_param *kp)
+{
+	int ret = param_set_uint(val, kp);
+
+	if (ret)
+		return ret;
+	WRITE_ONCE(pisa_scrub_max, min(pisa_scrub_max, PISA_SCRUB_MAX_LIMIT));
+	/* Serve demand accumulated while the scrubber was off (once initialized). */
+	if (pisa_enabled() && pisa_scrub_max)
+		pisa_schedule_scrub_all();
+	return 0;
+}
+
+static const struct kernel_param_ops pisa_scrub_ops = {
+	.set = pisa_scrub_set,
+	.get = param_get_uint,
+};
+module_param_cb(scrub_max, &pisa_scrub_ops, &pisa_scrub_max, 0644);
+
 static void pisa_reclaim_work_fn(struct work_struct *work);
 
 /* Asynchronous reclaim worker for pisa_free_list and expired idle pages. */
@@ -245,6 +269,140 @@ static bool is_pisa_block_poisoned(struct pisa_alloc *pa, struct page *block)
 	return false;
 }
 
+/*
+ * Background page scrubber: pre-zeroes dirty blocks in @pa->nid's work item
+ * to match inline-zeroed demand (@pa->scrub_demand) when scrub_max > 0.
+ */
+static struct pisa_meta *pisa_pick_dirty_meta(struct pisa_alloc *pa)
+{
+	struct pisa_meta *meta;
+
+	meta = list_first_entry_or_null(&pa->partial_dirty, struct pisa_meta, list);
+	if (meta)
+		return meta;
+
+	list_for_each_entry(meta, &pa->partial, list) {
+		if (meta->nr_dirty)
+			return meta;
+	}
+
+	/* Idle pages are left alone: scrubbing one would restart its idle clock. */
+	return NULL;
+}
+
+/* Dirty blocks per allocator, inline-zeroed since the last run, that queue a run. */
+#define PISA_SCRUB_TRIGGER	32U
+/* Cap on @pa->scrub_demand, so a long-off scrubber does not owe too much. */
+#define PISA_SCRUB_DEMAND_MAX	4096U
+/* Bytes zeroed per batch: a few microseconds of memset(). */
+#define PISA_SCRUB_BYTES	SZ_128K
+#define PISA_SCRUB_BATCH	(PISA_SCRUB_BYTES >> PAGE_SHIFT)
+static_assert(PISA_SCRUB_BATCH * sizeof(u16) <= 128);
+
+/*
+ * Scrub one batch of dirty blocks in @pa outside @pa->lock.
+ * Returns the number of PAGE_SIZE pages taken off the dirty bitmap.
+ */
+static unsigned long pisa_alloc_scrub_blocks(struct pisa_alloc *pa)
+{
+	unsigned int bpp, order, idx, count = 0, want;
+	unsigned long flags, used = 0;
+	u16 idxs[PISA_SCRUB_BATCH];
+	struct page *page, *block;
+	struct pisa_meta *meta;
+	bool should_release;
+
+	spin_lock_irqsave(&pa->lock, flags);
+	if (!pa->scrub_demand) {
+		spin_unlock_irqrestore(&pa->lock, flags);
+		return 0;
+	}
+
+	meta = pisa_pick_dirty_meta(pa);
+	if (!meta) {
+		pa->scrub_demand = 0;
+		spin_unlock_irqrestore(&pa->lock, flags);
+		return 0;
+	}
+
+	order = pa->order;
+	bpp = PISA_BLOCKS(order);
+	page = pisa_meta_to_page(meta);
+	want = min(max(PISA_SCRUB_BATCH >> order, 1U), pa->scrub_demand);
+
+	for_each_set_bit(idx, meta->dirty_bitmap, bpp) {
+		block = page + (idx << order);
+		__clear_bit(idx, meta->dirty_bitmap);
+		used++;
+		if (!is_pisa_block_poisoned(pa, block))
+			idxs[count++] = idx;
+		if (used == meta->nr_dirty || count == want)
+			break;
+	}
+	meta->nr_dirty -= used;
+	pa->blocks_in_use += used - count;
+	meta->nr_scrub = count;
+	pisa_place_meta(meta);
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (unlikely(!count))
+		return used << order;
+
+	for (idx = 0; idx < count; idx++) {
+		block = page + (idxs[idx] << order);
+		memset(page_address(block), 0, PAGE_SIZE << order);
+	}
+
+	spin_lock_irqsave(&pa->lock, flags);
+	for (idx = 0; idx < count; idx++)
+		__set_bit(idxs[idx], meta->clean_bitmap);
+	meta->nr_clean += count;
+	meta->nr_scrub = 0;
+	pa->scrub_demand -= min(count, pa->scrub_demand);
+	should_release = pisa_place_meta(meta);
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (should_release)
+		pisa_schedule_reclaim();
+
+	return used << order;
+}
+
+/*
+ * Scrub dirty blocks for allocators homed on @nid up to @pisa_scrub_max pages or
+ * PISA_SCRUB_RUN_NS. Returns true if stopped early with demand remaining.
+ */
+bool pisa_scrub_node(int nid)
+{
+	unsigned long budget = READ_ONCE(pisa_scrub_max), n;
+	u64 deadline = local_clock() + PISA_SCRUB_RUN_NS;
+	struct pisa_alloc *pa;
+	bool progress;
+
+	if (!budget)
+		return false;
+
+	do {
+		progress = false;
+		read_lock(&pisa_allocs_lock);
+		list_for_each_entry(pa, &pisa_allocs, node) {
+			if (pa->nid != nid || !READ_ONCE(pa->scrub_demand))
+				continue;
+			read_unlock(&pisa_allocs_lock);
+			n = pisa_alloc_scrub_blocks(pa);
+			if (n >= budget || local_clock() >= deadline)
+				return true;
+			budget -= n;
+			progress |= n != 0;
+			cond_resched();
+			read_lock(&pisa_allocs_lock);
+		}
+		read_unlock(&pisa_allocs_lock);
+	} while (progress);
+
+	return false;
+}
+
 /**
  * pisa_alloc_create - Create a PISA page allocator
  * @nid: Home NUMA node for this allocator
@@ -731,6 +889,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
 		for (unsigned int i = got; i < got + dirty; i++)
 			set_page_private(out[i], 1);
 		got += dirty;
+		pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
 	} else {
 		/* First choice are dirty blocks, followed by clean */
 		got += pisa_scan_bitmap(meta, true, want, out, !frozen);
@@ -754,6 +913,7 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
 static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
 				       struct page **pages, bool zero, bool frozen)
 {
+	bool should_scrub = false;
 	unsigned long n, flags;
 	struct pisa_meta *meta;
 	unsigned int want;
@@ -767,6 +927,7 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 			continue;
 
 		/* Fallback to a new allocation */
+		should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
 		spin_unlock_irqrestore(&pa->lock, flags);
 		/* With @zero, fresh dirty blocks are zeroed below. */
 		meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
@@ -780,6 +941,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
 		pa->stats.page_alloc++;
 	}
+	/* Queue the scrubber when the inline zeroing crosses the trigger. */
+	should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
 	spin_unlock_irqrestore(&pa->lock, flags);
 
 out:
@@ -794,6 +957,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 		else
 			page_ref_unfreeze(pages[i], 1);
 	}
+	if (should_scrub)
+		pisa_schedule_scrub(pa->nid);
 	return n;
 }
 
@@ -882,8 +1047,14 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
 		/* Fill the run of @n NULL entries at @i. */
 		for (n = 1; i + n < nr_pages && !page_array[i + n]; n++)
 			;
-		/* Pairs with smp_store_release() in pisa_enable_pcp(). */
-		if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id())
+		/*
+		 * With the scrubber on, zeroed allocations skip the per-CPU
+		 * cache: __pisa_alloc_bulk() prefers clean blocks and records
+		 * the demand that drives the scrubber.
+		 * smp_load_acquire() pairs with smp_store_release() in pisa_enable_pcp().
+		 */
+		if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
+		    !(want_init_on_alloc(gfp) && READ_ONCE(pisa_scrub_max)))
 			got = pisa_pcp_alloc(pa, gfp, n, page_array + i, frozen);
 		else
 			got = __pisa_alloc_bulk(pa, gfp, n, page_array + i,
@@ -965,6 +1136,7 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
 		page_ref_unfreeze(page + i, 1);
 	}
 	pa->blocks_in_use += nblocks;
+	pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
 	*clean = !dirty;
 	pisa_place_meta(meta);
 	return page;
@@ -995,7 +1167,7 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 			  size_t size, dma_addr_t *dma_handle, gfp_t gfp,
 			  unsigned long attrs)
 {
-	bool should_release = false, clean = false;
+	bool should_scrub = false, should_release = false, clean = false;
 	struct list_head *lists[3];
 	unsigned long align_mask, flags;
 	struct pisa_meta *meta, *tmp;
@@ -1047,10 +1219,14 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 		}
 	}
 	if (!page) {
+		should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
 		spin_unlock_irqrestore(&pa->lock, flags);
 		meta = pisa_alloc_split_page(0, gfp, pa->nid);
-		if (IS_ERR(meta))
+		if (IS_ERR(meta)) {
+			if (should_scrub)
+				pisa_schedule_scrub(pa->nid);
 			return NULL;
+		}
 		spin_lock_irqsave(&pa->lock, flags);
 		meta->pa = pa;
 		meta->prot = pa->prot;
@@ -1061,10 +1237,13 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 		if (unlikely(!page))
 			should_release = pisa_place_meta(meta);
 	}
+	should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
 	spin_unlock_irqrestore(&pa->lock, flags);
 
 	if (should_release)
 		pisa_schedule_reclaim();
+	if (should_scrub)
+		pisa_schedule_scrub(pa->nid);
 	if (!page)
 		return NULL;
 	if (!domain && unlikely(!dma_coherent_ok(dev, page_to_phys(page), size))) {
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 0f8cc9baf5d28..4effbc8610079 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -662,8 +662,57 @@ static void test_gfp_pisa(struct kunit *test)
 
 static void test_reservoir_and_scrubber(struct kunit *test)
 {
+	unsigned int old_res = READ_ONCE(pisa_reservoir_max);
+	unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
+	struct pisa_meta *meta;
+	struct pisa_alloc *pa;
+	struct page *p;
+	int nid;
+
 	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
-	KUNIT_EXPECT_EQ(test, READ_ONCE(pisa_reservoir_max), 16U);
+	pisa_shrink_scan(NULL, &sc);
+
+	/* Refill reservoir to 1 PISA page per memory node, then consume one. */
+	WRITE_ONCE(pisa_reservoir_max, 1);
+	pisa_schedule_reservoir();
+	for_each_node_state(nid, N_MEMORY)
+		flush_work(&pisa_nodes[nid].work);
+	KUNIT_EXPECT_GE(test, pisa_reservoir_count(), 1UL);
+
+	pa = pisa_node_pa(first_memory_node, SKB_FRAG_PAGE_ORDER, true);
+	KUNIT_ASSERT_NOT_NULL(test, pa);
+	p = pisa_alloc(pa, GFP_KERNEL);
+	if (p) {
+		meta = pisa_meta_of_pfn(page_to_pfn(p));
+		/* Reservoir pages come pre-zeroed (all remaining blocks clean). */
+		KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+		__free_pages(p, SKB_FRAG_PAGE_ORDER);
+		pisa_pcp_drain(pa);
+		if (!want_init_on_free()) {
+			KUNIT_EXPECT_GE(test, meta->nr_dirty, 1U);
+
+			/* Scrub the freed block and its PCP batch back into clean_bitmap. */
+			pa->scrub_demand = meta->nr_dirty;
+			WRITE_ONCE(pisa_scrub_max, 512);
+			pisa_scrub_node(pa->nid);
+			KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+		}
+	}
+	pisa_shrink_scan(NULL, &sc);
+
+	/* Trim reservoirs back to 0. */
+	for_each_node_state(nid, N_MEMORY)
+		flush_work(&pisa_nodes[nid].work);
+	WRITE_ONCE(pisa_reservoir_max, 0);
+	pisa_schedule_reservoir();
+	for_each_node_state(nid, N_MEMORY)
+		flush_work(&pisa_nodes[nid].work);
+	KUNIT_EXPECT_EQ(test, pisa_reservoir_count(), 0UL);
+
+	WRITE_ONCE(pisa_reservoir_max, old_res);
+	pisa_schedule_reservoir();
+	WRITE_ONCE(pisa_scrub_max, old_scrub);
 }
 
 static struct kunit_case pisa_meta_test_cases[] = {
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 30f0f2f6ef628..49fa5271a392f 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -213,10 +213,13 @@ bool pisa_place_meta(struct pisa_meta *meta)
 		 * (head of partial list, preferred for allocation). In between,
 		 * keep the current dense/sparse state.
 		 */
+		/* Blocks that the scrubber is zeroing are free: count them here. */
+		unsigned int nfree = avail + meta->nr_scrub;
+
 		to_list = meta->nr_clean ? PISA_LIST_CLEAN : PISA_LIST_DIRTY;
-		if (avail >= bpp - (bpp >> 2))
+		if (nfree >= bpp - (bpp >> 2))
 			to_list |= PISA_LIST_SPARSE;
-		else if (avail > (bpp >> 1))
+		else if (nfree > (bpp >> 1))
 			to_list |= meta->on_list & PISA_LIST_SPARSE;
 	}
 
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index a39251b952263..c013e384b1c7e 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -5,6 +5,9 @@
  */
 
 #include <linux/atomic.h>
+#include <linux/bitops.h>
+#include <linux/device.h>
+#include <linux/export.h>
 #include <linux/pisa.h>
 #include <linux/err.h>
 #include <linux/gfp.h>
@@ -37,6 +40,9 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
 	},
 };
 
+/* Bit number in pisa_node.flags: background scrub requested on this node. */
+#define PISA_NODE_SCRUB	0
+
 static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
 {
 	int ret = param_set_uint(val, kp);
@@ -237,6 +243,30 @@ static void pisa_reservoir_trim_node(struct pisa_node *node, unsigned int max)
 	}
 }
 
+/* Schedule node @nid's work item to scrub allocators homed on @nid. */
+void pisa_schedule_scrub(int nid)
+{
+	struct pisa_node *node;
+
+	if (!READ_ONCE(pisa_scrub_max))
+		return;
+
+	node = &pisa_nodes[nid];
+	set_bit(PISA_NODE_SCRUB, &node->flags);
+	/* Pairs with the clearing of PENDING before the work runs. */
+	smp_mb__after_atomic();
+	if (!work_pending(&node->work))
+		queue_work_node(nid, system_dfl_wq, &node->work);
+}
+
+void pisa_schedule_scrub_all(void)
+{
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY)
+		pisa_schedule_scrub(nid);
+}
+
 /* Per-node worker: trim/refill the reservoir, then scrub allocators homed on @nid. */
 static void pisa_node_work_fn(struct work_struct *work)
 {
@@ -246,6 +276,9 @@ static void pisa_node_work_fn(struct work_struct *work)
 	pisa_reservoir_trim_node(node, READ_ONCE(pisa_reservoir_max));
 	if (pisa_enabled() && READ_ONCE(pisa_reservoir_max))
 		pisa_reservoir_refill_node(node, nid);
+
+	if (test_and_clear_bit(PISA_NODE_SCRUB, &node->flags) && pisa_scrub_node(nid))
+		pisa_schedule_scrub(nid);
 }
 
 /* Queue node @nid's work if its reservoir is below or above the target. */
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 60120a05082a9..9f01173d30f5f 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -108,6 +108,7 @@ struct pisa_alloc {
 	struct list_head	full;		/* PISA pages with 0 free blocks */
 	unsigned int		num_idle_pages;
 	unsigned int		max_idle_pages;
+	unsigned int		scrub_demand;	/* dirty blocks zeroed inline */
 	unsigned long		idle_jiffies;	/* last change of @idle at/above @max_idle_pages */
 	unsigned long		blocks_in_use;	/* blocks handed out and not yet freed */
 	/* Cold. */
@@ -136,7 +137,8 @@ struct pisa_node {
 	spinlock_t		lock;		/* protects @pages and @nr_pages */
 	unsigned int		nr_pages;	/* entries in @pages */
 	struct list_head	pages;		/* reservoir */
-	struct work_struct	work;		/* refill/trim, on this node */
+	unsigned long		flags;		/* PISA_NODE_* requests */
+	struct work_struct	work;		/* refill/trim/scrub, on this node */
 } ____cacheline_aligned_in_smp;
 
 /* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
@@ -229,11 +231,17 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
 unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
 			       unsigned int *nr, struct list_head *release_list,
 			       unsigned long nr_to_scan);
+/* Time after which a scrubber run stops and requeues. */
+#define PISA_SCRUB_RUN_NS	(50 * NSEC_PER_USEC)
+extern unsigned int pisa_scrub_max;
+bool pisa_scrub_node(int nid);
 
 /* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
 extern unsigned int pisa_reservoir_max;
 void pisa_node_init(void);
 void pisa_schedule_reservoir(void);
+void pisa_schedule_scrub(int nid);
+void pisa_schedule_scrub_all(void);
 struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
 unsigned long pisa_reservoir_count(void);
 unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 11/25] mm: Migrate user pages into PISA
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (9 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 10/25] mm: Add background page scrubber for " Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20 14:10     ` Lorenzo Stoakes (ARM)
  2026-11-20  1:06   ` [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration Luigi Rizzo
                     ` (15 subsequent siblings)
  26 siblings, 1 reply; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Zero-copy DMA paths (such as O_DIRECT disk I/O and MSG_ZEROCOPY sockets)
map each user page with a per-buffer IOMMU mapping and IOTLB flush.
Moving user buffers into PISA blocks once allows subsequent DMA
operations to reuse pre-mapped 2MB IOMMU entries.

Add mm/pisa-user.c to migrate private anonymous order-0 user pages into
per-node DMA_TO_DEVICE PISA allocators, controlled by
pisa.user_max_pages (0 = disabled), via madvise(MADV_PISA) /
MADV_NOPISA (also supported via process_madvise()) and
pisa_upgrade_user_pages().

Large anonymous folios are split to order 0 first, while budget remains,
with split_folio_to_list_keep_zero(): a variant of split_folio_to_list()
that leaves zero-filled subpages mapped instead of remapping them to the
shared zeropage, so that pages the caller is about to write to (e.g. the
target of an O_DIRECT read) are still there to migrate.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/huge_mm.h                |   7 +
 include/linux/pisa-mm.h                |   8 +
 include/linux/pisa.h                   |  18 ++
 include/linux/swap.h                   |   1 +
 include/uapi/asm-generic/mman-common.h |   2 +
 mm/Makefile                            |   2 +-
 mm/huge_memory.c                       |  36 ++-
 mm/madvise.c                           |  19 ++
 mm/pisa-alloc.c                        |   2 +
 mm/pisa-priv.h                         |   2 +
 mm/pisa-user.c                         | 394 +++++++++++++++++++++++++
 11 files changed, 483 insertions(+), 8 deletions(-)
 create mode 100644 mm/pisa-user.c

diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index fac05120ab774..d679de8cf261d 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -404,6 +404,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
 int folio_split_unmapped(struct folio *folio, unsigned int new_order);
 unsigned int min_order_for_split(struct folio *folio);
 int split_folio_to_list(struct folio *folio, struct list_head *list);
+int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list);
 int folio_check_splittable(struct folio *folio, unsigned int new_order,
 			   enum split_type split_type);
 int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
@@ -671,6 +672,12 @@ static inline int split_folio_to_list(struct folio *folio, struct list_head *lis
 	return -EINVAL;
 }
 
+static inline int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list)
+{
+	VM_WARN_ON_ONCE_FOLIO(1, folio);
+	return -EINVAL;
+}
+
 static inline int folio_split(struct folio *folio, unsigned int new_order,
 		struct page *page, struct list_head *list)
 {
diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
index d55a42397c4b3..1e82e081b4004 100644
--- a/include/linux/pisa-mm.h
+++ b/include/linux/pisa-mm.h
@@ -11,6 +11,8 @@ unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
 				      int preferred_nid, int nr_pages,
 				      struct page **page_array, bool frozen);
 bool __pisa_free_page(struct page *page, unsigned int order);
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+		 unsigned long end, bool to_pisa);
 
 /*
  * ============================================================================
@@ -77,5 +79,11 @@ static inline bool pisa_free_page(struct page *page, unsigned int order)
 	return false;
 }
 
+static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
+			       unsigned long end, bool to_pisa)
+{
+	return -EINVAL;
+}
+
 #endif /* CONFIG_PISA */
 #endif /* _LINUX_PISA_MM_H */
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index edd34771ad1c4..9321e7ff00042 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -14,6 +14,7 @@
 
 struct iommu_domain;
 struct iova_domain;
+struct iov_iter;
 
 /**
  * struct pisa_iova - A domain's IOVA window reserved for PISA pages
@@ -70,9 +71,11 @@ enum {
  * Static keys guarding all kernel hooks:
  * - @pisa_key: enabled on first device opt-in (pisa_meta_init()).
  * - @pisa_recycle_check_key: enabled when pisa.compact or pisa.target_usage_percent > 0.
+ * - @pisa_user_key: enabled when pisa.user_max_pages > 0.
  */
 DECLARE_STATIC_KEY_FALSE(pisa_key);
 DECLARE_STATIC_KEY_FALSE(pisa_recycle_check_key);
+DECLARE_STATIC_KEY_FALSE(pisa_user_key);
 
 extern u32 *pisa_pfn2slot;
 extern unsigned int pisa_nr_slots;
@@ -89,12 +92,19 @@ void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
 				    unsigned long attrs);
 bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
 bool __pisa_page_can_recycle(const struct page *page);
+bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
+			     size_t size, size_t offset);
 
 static inline bool pisa_enabled(void)
 {
 	return static_branch_unlikely(&pisa_key);
 }
 
+static inline bool pisa_user_enabled(void)
+{
+	return static_branch_unlikely(&pisa_user_key);
+}
+
 static inline u32 pisa_pfn_flags(unsigned long pfn)
 {
 	unsigned long frame = pfn >> PISA_ORDER;
@@ -195,6 +205,7 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 
 static inline int pisa_meta_init(void) { return -EOPNOTSUPP; }
 static inline bool pisa_enabled(void) { return false; }
+static inline bool pisa_user_enabled(void) { return false; }
 static inline bool is_pisa_pfn(unsigned long pfn) { return false; }
 static inline bool is_pisa_page(const struct page *page) { return false; }
 static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
@@ -228,6 +239,13 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 	return true;
 }
 
+static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
+					   struct page **pages, size_t size,
+					   size_t offset)
+{
+	return false;
+}
+
 #endif /* CONFIG_PISA */
 
 #if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 4f686709d71f0..3736786ccc5a3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -314,6 +314,7 @@ extern unsigned long totalreserve_pages;
 /* linux/mm/folio.c */
 void folio_add_lru(struct folio *folio);
 void folio_mark_accessed(struct folio *folio);
+void lru_add_drain(void);
 void lru_add_drain_all(void);
 
 enum lru_cache_drained {
diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
index ef1c27fa3c570..c4332db8e56ff 100644
--- a/include/uapi/asm-generic/mman-common.h
+++ b/include/uapi/asm-generic/mman-common.h
@@ -81,6 +81,8 @@
 
 #define MADV_GUARD_INSTALL 102		/* fatal signal on access to range */
 #define MADV_GUARD_REMOVE 103		/* unguard range */
+#define MADV_PISA		210	/* move pages into PISA memory */
+#define MADV_NOPISA		211	/* move pages out of PISA memory */
 
 /* compatibility flags */
 #define MAP_FILE	0
diff --git a/mm/Makefile b/mm/Makefile
index 7d624e4bad456..d6b54a6eb3dc0 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -147,7 +147,7 @@ obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
 obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_PISA) += pisa.o
-pisa-y := pisa-meta.o pisa-alloc.o pisa-node.o
+pisa-y := pisa-meta.o pisa-alloc.o pisa-node.o pisa-user.o
 obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 7791872f14af7..ffa3912f2418b 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4113,7 +4113,8 @@ static void folio_reset_partially_mapped(struct folio *folio)
 
 static int __folio_freeze_split_anon(struct folio *folio,
 		unsigned int new_order, struct page *split_at, bool do_lru,
-		struct list_head *list, enum split_type split_type)
+		struct list_head *list, enum split_type split_type,
+		bool keep_zero)
 {
 	struct folio *end_folio = folio_next(folio);
 	struct swap_cluster_info *ci = NULL;
@@ -4207,7 +4208,13 @@ static int __folio_freeze_split_anon(struct folio *folio,
 out_no_split:
 	local_irq_enable();
 	if (anon_vma) {
-		if (!ret && !folio_is_device_private(folio))
+		/*
+		 * Zero-filled anon subpages are normally unmapped, freed and
+		 * replaced by the shared zeropage. A @keep_zero caller needs
+		 * the subpages themselves, e.g. to migrate them before a
+		 * device writes to them.
+		 */
+		if (!ret && !folio_is_device_private(folio) && !keep_zero)
 			ttu_flags = TTU_USE_SHARED_ZEROPAGE;
 		remap_anon_folio(folio, 1 << old_order, ttu_flags);
 	}
@@ -4387,6 +4394,7 @@ static int __folio_freeze_split_file(struct folio *folio,
  * @lock_at: a page within @folio to be left locked to caller
  * @list: after-split folios will be put on it if non NULL
  * @split_type: perform uniform split or not (non-uniform split)
+ * @keep_zero: do not remap zero-filled anon subpages to the shared zeropage
  *
  * It calls __split_frozen_folio() to perform uniform and non-uniform split.
  * It is in charge of checking whether the split is supported or not and
@@ -4402,7 +4410,8 @@ static int __folio_freeze_split_file(struct folio *folio,
  */
 static int __folio_split(struct folio *folio, unsigned int new_order,
 		struct page *split_at, struct page *lock_at,
-		struct list_head *list, enum split_type split_type)
+		struct list_head *list, enum split_type split_type,
+		bool keep_zero)
 {
 	struct folio *end_folio = folio_next(folio);
 	const bool is_anon = folio_test_anon(folio);
@@ -4432,7 +4441,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
 
 	if (is_anon)
 		ret = __folio_freeze_split_anon(folio, new_order, split_at,
-						true, list, split_type);
+						true, list, split_type,
+						keep_zero);
 	else
 		ret = __folio_freeze_split_file(folio, new_order, split_at,
 						list, split_type);
@@ -4495,7 +4505,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
 	VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
 
 	return __folio_freeze_split_anon(folio, new_order, &folio->page,
-					 false, NULL, SPLIT_TYPE_UNIFORM);
+					 false, NULL, SPLIT_TYPE_UNIFORM, false);
 }
 
 /*
@@ -4551,7 +4561,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
 	struct folio *folio = page_folio(page);
 
 	return __folio_split(folio, new_order, &folio->page, page, list,
-			     SPLIT_TYPE_UNIFORM);
+			     SPLIT_TYPE_UNIFORM, false);
 }
 
 /**
@@ -4582,7 +4592,7 @@ int folio_split(struct folio *folio, unsigned int new_order,
 		struct page *split_at, struct list_head *list)
 {
 	return __folio_split(folio, new_order, split_at, &folio->page, list,
-			     SPLIT_TYPE_NON_UNIFORM);
+			     SPLIT_TYPE_NON_UNIFORM, false);
 }
 
 /**
@@ -4617,6 +4627,18 @@ int split_folio_to_list(struct folio *folio, struct list_head *list)
 	return split_huge_page_to_list_to_order(&folio->page, list, 0);
 }
 
+/*
+ * As split_folio_to_list(), but zero-filled subpages of an anon folio stay
+ * mapped instead of being remapped to the shared zeropage. For callers that
+ * are about to take write references on the subpages: PISA uses it to migrate
+ * the pages of an I/O buffer, e.g. the target of an O_DIRECT read.
+ */
+int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list)
+{
+	return __folio_split(folio, 0, &folio->page, &folio->page, list,
+			     SPLIT_TYPE_UNIFORM, true);
+}
+
 /*
  * __folio_unqueue_deferred_split() is not to be called directly:
  * the folio_unqueue_deferred_split() inline wrapper in mm/internal.h
diff --git a/mm/madvise.c b/mm/madvise.c
index 0214efd2a014f..46981ad295aa5 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -6,6 +6,7 @@
  * Copyright (C) 2002  Christoph Hellwig
  */
 
+#include <linux/pisa-mm.h>
 #include <linux/mman.h>
 #include <linux/pagemap.h>
 #include <linux/syscalls.h>
@@ -1393,6 +1394,11 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)
 		return madvise_guard_install(madv_behavior);
 	case MADV_GUARD_REMOVE:
 		return madvise_guard_remove(madv_behavior);
+#ifdef MADV_PISA
+	case MADV_PISA:
+	case MADV_NOPISA:
+		return pisa_madvise(vma, range->start, range->end, behavior == MADV_PISA);
+#endif
 
 	/* The below behaviours update VMAs via madvise_update_vma(). */
 
@@ -1582,6 +1588,11 @@ madvise_behavior_valid(int behavior)
 	case MADV_HWPOISON:
 #endif
 		return true;
+#ifdef MADV_PISA
+	case MADV_PISA:
+	case MADV_NOPISA:
+		return IS_ENABLED(CONFIG_PISA);
+#endif
 
 	default:
 		return false;
@@ -1596,6 +1607,10 @@ static bool process_madvise_remote_valid(int behavior)
 	case MADV_PAGEOUT:
 	case MADV_WILLNEED:
 	case MADV_COLLAPSE:
+#ifdef MADV_PISA
+	case MADV_PISA:
+	case MADV_NOPISA:
+#endif
 		return true;
 	default:
 		return false;
@@ -1782,6 +1797,10 @@ static enum madvise_lock_mode get_lock_mode(struct madvise_behavior *madv_behavi
 	case MADV_POPULATE_READ:
 	case MADV_POPULATE_WRITE:
 	case MADV_COLLAPSE:
+#ifdef MADV_PISA
+	case MADV_PISA:
+	case MADV_NOPISA:
+#endif
 		return MADVISE_MMAP_READ_LOCK;
 	case MADV_GUARD_INSTALL:
 	case MADV_GUARD_REMOVE:
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index ccba2bf01bd59..2213a56de1242 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -1432,6 +1432,8 @@ static int __init pisa_init(void)
 				  "pisa_ro_o%u_n%d", SKB_FRAG_PAGE_ORDER, nid);
 	}
 
+	pisa_user_boot_enable();
+
 	/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
 	pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
 	if (!pisa_wq)
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 9f01173d30f5f..b18d461198ac5 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -133,6 +133,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
 struct pisa_node {
 	unsigned long		next_alloc;	/* atomic alloc backoff jiffies */
 	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
+	struct pisa_alloc	*user_pa;	/* user page migration pa */
 
 	spinlock_t		lock;		/* protects @pages and @nr_pages */
 	unsigned int		nr_pages;	/* entries in @pages */
@@ -245,6 +246,7 @@ void pisa_schedule_scrub_all(void);
 struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
 unsigned long pisa_reservoir_count(void);
 unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+void pisa_user_boot_enable(void);
 
 #endif /* CONFIG_PISA */
 #endif /* _MM_PISA_PRIV_H */
diff --git a/mm/pisa-user.c b/mm/pisa-user.c
new file mode 100644
index 0000000000000..6c3efaee4ecda
--- /dev/null
+++ b/mm/pisa-user.c
@@ -0,0 +1,394 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA user and page-cache folios: allocate or migrate order-0 folios into
+ * per-node PISA allocators so zero-copy TX and block I/O use the PISA IOMMU mappings.
+ */
+#include <linux/pisa.h>
+#include <linux/gfp.h>
+#include <linux/huge_mm.h>
+#include <linux/migrate.h>
+#include <linux/mm.h>
+#include <linux/moduleparam.h>
+#include <linux/nodemask.h>
+#include <linux/pagewalk.h>
+#include <linux/swap.h>
+#include <linux/uio.h>
+
+#include "pisa-priv.h"
+
+/* Soft cap on user (PTE) pages held in PISA blocks, in PISA pages (0 = disabled). */
+static unsigned long pisa_user_max_pages __read_mostly;
+DEFINE_STATIC_KEY_FALSE(pisa_user_key);
+
+/* Create per-node user allocators and enable pisa_user_key (serialized by param_lock/init). */
+static int pisa_user_init(void)
+{
+	int nid;
+
+	if (pisa_user_enabled())
+		return 0;
+	if (pisa_meta_init())
+		return -ENOMEM;
+	for_each_node_state(nid, N_MEMORY) {
+		if (!pisa_alloc_ensure(&pisa_nodes[nid].user_pa, nid, 0,
+				       DMA_TO_DEVICE, false, "user_tx_n%d", nid))
+			return -ENOMEM;
+	}
+	static_branch_enable(&pisa_user_key);
+	return 0;
+}
+
+static int pisa_user_max_set(const char *val, const struct kernel_param *kp)
+{
+	unsigned long max_pages;
+	int ret;
+
+	ret = kstrtoul(val, 0, &max_pages);
+	if (ret)
+		return ret;
+	WRITE_ONCE(pisa_user_max_pages, max_pages);
+	if (!max_pages) {
+		static_branch_disable(&pisa_user_key);
+	} else if (pisa_nodes[first_memory_node].pa[0][0] && pisa_user_init()) {
+		WRITE_ONCE(pisa_user_max_pages, 0);
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static const struct kernel_param_ops pisa_user_max_ops = {
+	.set = pisa_user_max_set,
+	.get = param_get_ulong,
+};
+module_param_cb(user_max_pages, &pisa_user_max_ops, &pisa_user_max_pages, 0644);
+
+/*
+ * Initialize per-node user allocators if pisa.user_max_pages was set on
+ * the kernel command line before pisa_init() ran.
+ */
+void __init pisa_user_boot_enable(void)
+{
+	if (READ_ONCE(pisa_user_max_pages))
+		pisa_user_init();
+}
+
+/* User pages currently held in PISA blocks. */
+static unsigned long pisa_user_in_use(void)
+{
+	unsigned long used = 0;
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY) {
+		/* Pairs with smp_store_release() in pisa_user_init(). */
+		const struct pisa_alloc *pa = smp_load_acquire(&pisa_nodes[nid].user_pa);
+
+		used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+	}
+	return used;
+}
+
+/* Pages that can still be moved in before reaching pisa.user_max_pages. */
+static long pisa_user_budget(void)
+{
+	return (long)((READ_ONCE(pisa_user_max_pages) << PISA_ORDER) - pisa_user_in_use());
+}
+
+/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
+static bool pisa_user_want(struct folio *folio, bool to_pisa)
+{
+	if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
+	    folio_test_ksm(folio) || is_huge_zero_folio(folio) ||
+	    (folio_test_large(folio) && !to_pisa) ||
+	    folio_maybe_mapped_shared(folio))
+		return false;
+	return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+}
+
+/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
+static bool pisa_user_busy(struct folio *folio, int extra)
+{
+	return folio_ref_count(folio) > folio_expected_ref_count(folio) + extra;
+}
+
+struct pisa_user_mtc {
+	gfp_t	gfp;
+	long	budget;
+};
+
+/* new_folio_t for migration into PISA memory. */
+static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
+{
+	struct pisa_user_mtc *mtc = (void *)private;
+	int nid = folio_nid(src);
+	struct page *page;
+
+	/* Callers split large folios first (see pisa_user_split()); none is expected here. */
+	if (mtc->budget <= 0 || folio_test_large(src))
+		return NULL;
+	page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+	if (!page)
+		return NULL;
+	mtc->budget--;
+	return page_folio(page);
+}
+
+/* free_folio_t: give back an unused destination block and the budget it was charged. */
+static void pisa_user_put_folio(struct folio *dst, unsigned long private)
+{
+	struct pisa_user_mtc *mtc = (void *)private;
+
+	mtc->budget++;
+	folio_put(dst);
+}
+
+/* new_folio_t for migration out of PISA memory: allocators never serve these. */
+static struct folio *pisa_user_old_folio(struct folio *src, unsigned long private)
+{
+	return __folio_alloc_node(GFP_HIGHUSER_MOVABLE | __GFP_NOWARN, 0, folio_nid(src));
+}
+
+/* Migrate the isolated folios on @list; failures are put back. */
+static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+{
+	struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+	enum migrate_mode mode = MIGRATE_SYNC;
+
+	if (list_empty(list))
+		return;
+	if (to_pisa) {
+		mtc.budget = pisa_user_budget();
+		if (!gfpflags_allow_blocking(gfp))
+			mode = MIGRATE_ASYNC;
+	}
+	if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
+			  to_pisa ? pisa_user_put_folio : NULL, (unsigned long)&mtc,
+			  mode, MR_SYSCALL, NULL))
+		putback_movable_pages(list);
+}
+
+/*
+ * Split large anon @folio to order 0, with the caller holding the only extra
+ * reference. Zero-filled subpages stay mapped instead of being replaced by the
+ * shared zeropage, so that a GUP for write that follows (e.g. an O_DIRECT
+ * read) finds them. After-split folios go to @list, or to the LRU if NULL.
+ */
+static bool pisa_user_split(struct folio *folio, struct list_head *list, bool can_wait)
+{
+	int ret;
+
+	if (can_wait)
+		folio_lock(folio);
+	else if (!folio_trylock(folio))
+		return false;
+	ret = split_folio_to_list_keep_zero(folio, list);
+	folio_unlock(folio);
+	return !ret;
+}
+
+/*
+ * Split isolated large @folio and move to @out the subpages found in @pages.
+ * The others, or all of @folio if @split is false or the split fails, go back
+ * to the LRU. Returns the number of subpages moved to @out.
+ */
+static unsigned int pisa_user_split_isolated(struct folio *folio, struct list_head *out,
+					     struct page **pages, unsigned int nr, bool split)
+{
+	const struct page *lo = NULL, *hi = NULL;
+	unsigned int kept = 0;
+	struct folio *f, *next;
+	LIST_HEAD(pieces);
+
+	/* @folio is held isolated, so page_folio() of its subpages is stable. */
+	for (unsigned int i = 0; i < nr; i++) {
+		if (page_folio(pages[i]) != folio)
+			continue;
+		if (!lo || pages[i] < lo)
+			lo = pages[i];
+		if (!hi || pages[i] > hi)
+			hi = pages[i];
+	}
+	list_move(&folio->lru, &pieces);
+	if (split && pisa_user_split(folio, &pieces, false)) {
+		list_for_each_entry_safe(f, next, &pieces, lru) {
+			if (!folio_test_large(f) && &f->page >= lo && &f->page <= hi) {
+				list_move_tail(&f->lru, out);
+				kept++;
+			}
+		}
+	}
+	putback_movable_pages(&pieces);
+	return kept;
+}
+
+/**
+ * pisa_upgrade_user_pages - Migrate extracted user @pages into PISA
+ * @iter: Iterator from which @pages were just extracted
+ * @pages: Extracted page array, one reference each
+ * @size: Number of bytes extracted from @iter
+ * @offset: Byte offset within @pages[0]
+ *
+ * Return: false if no pages were isolated (@pages and @iter are untouched);
+ * true otherwise, so the caller must re-extract @pages from the reverted @iter.
+ */
+bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
+			     size_t size, size_t offset)
+{
+	unsigned int nr = DIV_ROUND_UP(offset + size, PAGE_SIZE);
+	unsigned int want = 0;
+	struct folio *f, *next;
+	bool drained = false;
+	LIST_HEAD(small);
+	LIST_HEAD(list);
+	long budget;
+
+	if (!pisa_user_enabled())
+		return false;
+	/*
+	 * First pass: check if any page needs migration and whether budget is
+	 * available before draining per-CPU LRU caches or isolating folios.
+	 */
+	for (unsigned int i = 0; i < nr; i++)
+		want += pisa_user_want(page_folio(pages[i]), true);
+	if (!want || pisa_user_budget() <= 0)
+		return false;
+	for (unsigned int i = 0; i < nr; i++) {
+		struct folio *folio = page_folio(pages[i]);
+		unsigned int refs = 0;
+
+		if (!pisa_user_want(folio, true))
+			continue;
+		/* Drain per-CPU LRU batches once so freshly faulted folios can be isolated. */
+		if (!drained && !folio_test_lru(folio)) {
+			lru_add_drain();
+			drained = true;
+		}
+		if (!folio_test_lru(folio))
+			continue;
+		for (unsigned int j = 0; j < nr; j++)
+			refs += page_folio(pages[j]) == folio;
+		/* The references from iov_iter_get_pages2() are expected. */
+		if (folio_maybe_dma_pinned(folio) || pisa_user_busy(folio, refs))
+			continue;
+		if (isolate_folio_to_list(folio, &list))
+			list_move_tail(&folio->lru, &list);
+	}
+	if (list_empty(&list))
+		return false;
+	/*
+	 * Release the caller's references on @pages before migrating
+	 * (migrate_pages() rejects folios with extra references).
+	 */
+	for (unsigned int i = 0; i < nr; i++)
+		put_page(pages[i]);
+	/*
+	 * Split large folios while budget remains and keep only their subpages
+	 * that are in @pages: migrate_pages() would split them regardless of
+	 * the budget, migrate every subpage and drop the zero-filled ones.
+	 */
+	budget = pisa_user_budget();
+	list_for_each_entry_safe(f, next, &list, lru) {
+		if (folio_test_large(f))
+			budget -= pisa_user_split_isolated(f, &small, pages, nr, budget > 0);
+		else
+			budget -= !is_pisa_pfn(folio_pfn(f));
+	}
+	list_splice_tail(&small, &list);
+	pisa_user_migrate(&list, true,
+			  GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+	/* Rewind @iter so the caller re-extracts the updated page pointers. */
+	iov_iter_revert(iter, size);
+	return true;
+}
+
+#define PISA_USER_BATCH	512
+
+struct pisa_madvise_walk {
+	struct list_head	list;	/* isolated folios waiting to be migrated */
+	unsigned int		n;	/* pages on @list */
+	bool			to_pisa;
+};
+
+#define PISA_MADVISE_GFP	(GFP_USER | __GFP_NOWARN)
+
+/* Isolate the wanted folios mapped in one PMD's range, splitting the large ones first. */
+static int pisa_madvise_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end,
+				  struct mm_walk *walk)
+{
+	struct pisa_madvise_walk *w = walk->private;
+	bool to_pisa = w->to_pisa;
+	struct folio_walk fw;
+	struct folio *folio;
+
+	while (addr < end) {
+		unsigned long nr = 1;
+
+		if (fatal_signal_pending(current))
+			return -EINTR;
+		/* Stop the walk, without an error, once the cap is reached. */
+		if (to_pisa && pisa_user_budget() <= (long)w->n)
+			return 1;
+		folio = folio_walk_start(&fw, walk->vma, addr, 0);
+		if (folio) {
+			bool want = pisa_user_want(folio, to_pisa) && folio_test_lru(folio);
+			bool busy = want && pisa_user_busy(folio, 0);
+			bool split = want && !busy && folio_test_large(folio);
+
+			/* A PMD mapping covers the rest of the range, a PTE maps one page. */
+			if (fw.level != FW_LEVEL_PTE)
+				nr = (end - addr) >> PAGE_SHIFT;
+			if (split) {
+				folio_get(folio);
+			} else if (want && !busy && isolate_folio_to_list(folio, &w->list)) {
+				list_move_tail(&folio->lru, &w->list);
+				w->n++;
+			}
+			folio_walk_end(&fw, walk->vma);
+			if (split) {
+				split = pisa_user_split(folio, NULL, true);
+				folio_put(folio);
+				/* Look at @addr again: it now maps an order-0 folio. */
+				if (split)
+					continue;
+			}
+		}
+		if (w->n >= PISA_USER_BATCH) {
+			pisa_user_migrate(&w->list, to_pisa, PISA_MADVISE_GFP);
+			w->n = 0;
+		}
+		addr += nr << PAGE_SHIFT;
+		cond_resched();
+	}
+	return 0;
+}
+
+static const struct mm_walk_ops pisa_madvise_walk_ops = {
+	.pmd_entry	= pisa_madvise_pmd_entry,
+	.walk_lock	= PGWALK_RDLOCK,
+};
+
+/**
+ * pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
+ * @vma: VMA, with the mmap lock held for reading
+ * @start: Start address
+ * @end: End address
+ * @to_pisa: true to move present pages into PISA, false to move them out
+ *
+ * Return: 0, -EINVAL for unsupported mappings, or -EOPNOTSUPP if disabled.
+ */
+int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool to_pisa)
+{
+	struct pisa_madvise_walk w = { .list = LIST_HEAD_INIT(w.list), .to_pisa = to_pisa };
+	int ret;
+
+	if (vma->vm_flags & (VM_SHARED | VM_IO | VM_PFNMAP | VM_HUGETLB))
+		return -EINVAL;
+	if (to_pisa && !pisa_user_enabled())
+		return -EOPNOTSUPP;
+	if (!pisa_enabled() && !to_pisa)
+		return 0;
+	lru_add_drain();
+	/* The page walker skips the unpopulated page-table levels of sparse mappings. */
+	ret = walk_page_range_vma(vma, start, end, &pisa_madvise_walk_ops, &w);
+	pisa_user_migrate(&w.list, to_pisa, PISA_MADVISE_GFP);
+	return ret < 0 ? ret : 0;
+}
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (10 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device Luigi Rizzo
                     ` (14 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Extend PISA user/page-cache allocators to support O_DIRECT and buffered
disk I/O, gated per block/controller device through pisa_rx_bufs (read)
and pisa_tx_bufs (write):

- Split per-node user_pa into DMA_BIDIRECTIONAL, DMA_TO_DEVICE, and
  DMA_FROM_DEVICE allocators, promoting pages to DMA_BIDIRECTIONAL if
  later accessed in the opposite direction.
- Upgrade pinned O_DIRECT user pages in bio_iov_iter_get_pages(), and
  allocate/promote order-0 page-cache folios for buffered I/O when PISA
  is enabled on the target block device.
- Add pisa.scrub_migrate to compact or demote lingering folios off old,
  sparse user_pa 2MB pages.

Assisted-by: gemini:2.5-pro
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 block/bio.c             |   9 +
 include/linux/pisa-mm.h |  76 ++++++
 include/linux/pisa.h    |   7 +-
 mm/filemap.c            |  12 +-
 mm/page-writeback.c     |   9 +
 mm/pisa-alloc.c         |  18 +-
 mm/pisa-meta.c          |  15 +-
 mm/pisa-node.c          |  10 +-
 mm/pisa-priv.h          |   9 +-
 mm/pisa-user.c          | 523 ++++++++++++++++++++++++++++++++++++----
 mm/readahead.c          |   3 +-
 11 files changed, 635 insertions(+), 56 deletions(-)

diff --git a/block/bio.c b/block/bio.c
index b0fcc69de6535..ad8632661dddf 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -6,6 +6,7 @@
 #include <linux/swap.h>
 #include <linux/bio-integrity.h>
 #include <linux/blkdev.h>
+#include <linux/pisa-mm.h>
 #include <linux/uio.h>
 #include <linux/iocontext.h>
 #include <linux/slab.h>
@@ -1346,6 +1347,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 {
 	iov_iter_extraction_t flags = 0;
 	struct iov_iter saved_iter;
+	bool migrated = false;
 	int ret;
 
 	if (WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED)))
@@ -1368,6 +1370,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		flags |= ITER_ALLOW_P2PDMA;
 
 	saved_iter = *iter;
+retry:
 	do {
 		ssize_t size;
 
@@ -1386,6 +1389,12 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
 		bio->bi_iter.bi_size += size;
 	} while (iov_iter_count(iter) && !bio_full(bio, 0));
 
+	/* Migrate suitable pages to PISA, re-extract if so. */
+	if (!migrated && pisa_bio_migrate_pages(bio, iter)) {
+		migrated = true;
+		goto retry;
+	}
+
 	if (is_pci_p2pdma_page(bio->bi_io_vec->bv_page))
 		bio->bi_opf |= REQ_NOMERGE;
 	ret = bio_iov_iter_align_down(bio, iter, len_align_mask);
diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
index 1e82e081b4004..8caa2afded6ff 100644
--- a/include/linux/pisa-mm.h
+++ b/include/linux/pisa-mm.h
@@ -3,14 +3,29 @@
 #ifndef _LINUX_PISA_MM_H
 #define _LINUX_PISA_MM_H
 
+#include <linux/bio.h>
 #include <linux/pisa.h>
 
+struct address_space;
+struct folio_batch;
+
 #ifdef CONFIG_PISA
 
 unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
 				      int preferred_nid, int nr_pages,
 				      struct page **page_array, bool frozen);
 bool __pisa_free_page(struct page *page, unsigned int order);
+bool __pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter,
+			      enum dma_data_direction dir);
+bool pisa_bdev_enabled(const struct block_device *bdev,
+		       enum dma_data_direction dir);
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+				   gfp_t gfp);
+bool __pisa_filemap_upgrade(const struct address_space *mapping,
+			    struct folio *folio);
+bool __pisa_writeback_upgrade(const struct address_space *mapping,
+			      struct folio *folio, pgoff_t *index,
+			      struct folio_batch *fbatch);
 int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
 		 unsigned long end, bool to_pisa);
 
@@ -57,6 +72,46 @@ static inline bool pisa_free_page(struct page *page, unsigned int order)
 	return is_pisa_page(page) && __pisa_free_page(page, order);
 }
 
+/*
+ * Hook for bio_iov_iter_get_pages() O_DIRECT migration (@pisa_user_key).
+ * Returns false if @bio and @iter were left untouched; returns true if the
+ * pages were unpinned, migrated to PISA, and @iter rewound for re-extraction.
+ */
+static inline bool pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter)
+{
+	enum dma_data_direction dir;
+
+	if (!pisa_user_enabled() || !bio->bi_iter.bi_size ||
+	    !bio_flagged(bio, BIO_PAGE_PINNED))
+		return false;
+	dir = op_is_write(bio_op(bio)) ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+	return pisa_bdev_enabled(bio->bi_bdev, dir) &&
+	       __pisa_bio_migrate_pages(bio, iter, dir);
+}
+
+/* Hook for __filemap_get_folio_mpol() and ractl_alloc_folio() (@pisa_user_key). */
+#define pisa_filemap_alloc_folio(gfp, order, policy, mapping)		\
+	((pisa_user_enabled() && !(order) ?				\
+	  __pisa_filemap_alloc(mapping, gfp) : NULL) ?:			\
+	 filemap_alloc_folio(gfp, order, policy))
+
+/* Hook for __filemap_get_folio_mpol() existing folio upgrade (@pisa_user_key). */
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+					struct folio *folio, bool write_wait)
+{
+	return pisa_user_enabled() && write_wait &&
+	       __pisa_filemap_upgrade(mapping, folio);
+}
+
+/* Hook for writeback_get_folio() upgrade (guarded by @pisa_user_key). */
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+					  struct folio *folio, pgoff_t *index,
+					  struct folio_batch *fbatch)
+{
+	return pisa_user_enabled() &&
+	       __pisa_writeback_upgrade(mapping, folio, index, fbatch);
+}
+
 #else /* !CONFIG_PISA */
 
 static inline unsigned long
@@ -79,6 +134,27 @@ static inline bool pisa_free_page(struct page *page, unsigned int order)
 	return false;
 }
 
+static inline bool pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter)
+{
+	return false;
+}
+
+#define pisa_filemap_alloc_folio(gfp, order, policy, mapping)		\
+	filemap_alloc_folio(gfp, order, policy)
+
+static inline bool pisa_filemap_upgrade(const struct address_space *mapping,
+					struct folio *folio, bool write_wait)
+{
+	return false;
+}
+
+static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
+					  struct folio *folio, pgoff_t *index,
+					  struct folio_batch *fbatch)
+{
+	return false;
+}
+
 static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
 			       unsigned long end, bool to_pisa)
 {
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 9321e7ff00042..6d184e393e297 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -93,7 +93,8 @@ void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
 bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
 bool __pisa_page_can_recycle(const struct page *page);
 bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
-			     size_t size, size_t offset);
+			     size_t size, size_t offset,
+			     enum dma_data_direction dir, bool pinned);
 
 static inline bool pisa_enabled(void)
 {
@@ -241,7 +242,9 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 
 static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
 					   struct page **pages, size_t size,
-					   size_t offset)
+					   size_t offset,
+					   enum dma_data_direction dir,
+					   bool pinned)
 {
 	return false;
 }
diff --git a/mm/filemap.c b/mm/filemap.c
index f9e34144c53c7..7d870c22abe53 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -15,6 +15,7 @@
 #include <linux/dax.h>
 #include <linux/fs.h>
 #include <linux/sched/signal.h>
+#include <linux/pisa-mm.h>
 #include <linux/uaccess.h>
 #include <linux/capability.h>
 #include <linux/kernel_stat.h>
@@ -1976,6 +1977,7 @@ void *filemap_get_entry(struct address_space *mapping, pgoff_t index)
 struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 		pgoff_t index, fgf_t fgp_flags, gfp_t gfp, struct mempolicy *policy)
 {
+	bool tried_pisa = false;
 	struct folio *folio;
 
 repeat:
@@ -1985,6 +1987,13 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 	if (!folio)
 		goto no_page;
 
+	if (!tried_pisa) {
+		tried_pisa = true;
+		if (pisa_filemap_upgrade(mapping, folio,
+					 (fgp_flags & (FGP_WRITE | FGP_NOWAIT)) == FGP_WRITE))
+			goto repeat;
+	}
+
 	if (fgp_flags & FGP_LOCK) {
 		if (fgp_flags & FGP_NOWAIT) {
 			if (!folio_trylock(folio)) {
@@ -2044,7 +2053,8 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
 			err = -ENOMEM;
 			if (order > min_order)
 				alloc_gfp |= __GFP_NORETRY | __GFP_NOWARN;
-			folio = filemap_alloc_folio(alloc_gfp, order, policy);
+			folio = pisa_filemap_alloc_folio(alloc_gfp, order,
+							 policy, mapping);
 			if (!folio)
 				continue;
 
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 0694ec4098f6d..5f1327c22c0e4 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -37,6 +37,7 @@
 #include <linux/sched/rt.h>
 #include <linux/sched/signal.h>
 #include <linux/mm_inline.h>
+#include <linux/pisa-mm.h>
 #include <linux/shmem_fs.h>
 #include <trace/events/writeback.h>
 
@@ -2409,6 +2410,7 @@ static pgoff_t wbc_end(struct writeback_control *wbc)
 static struct folio *writeback_get_folio(struct address_space *mapping,
 		struct writeback_control *wbc)
 {
+	bool tried_pisa = false;
 	struct folio *folio;
 
 retry:
@@ -2423,6 +2425,13 @@ static struct folio *writeback_get_folio(struct address_space *mapping,
 			return NULL;
 	}
 
+	if (!tried_pisa &&
+	    pisa_writeback_upgrade(mapping, folio, &wbc->index, &wbc->fbatch)) {
+		tried_pisa = true;
+		goto retry;
+	}
+	tried_pisa = false;
+
 	folio_lock(folio);
 	if (unlikely(!folio_prepare_writeback(mapping, wbc, folio))) {
 		folio_unlock(folio);
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 2213a56de1242..8c4c0f1f1cf50 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -574,6 +574,20 @@ static unsigned int pisa_pcp_pop(struct pisa_alloc *pa, unsigned int nr, struct
 	return got;
 }
 
+/* Free blocks in the per-CPU caches of @pa, which @pa->blocks_in_use counts as in use. */
+unsigned long pisa_pcp_cached(struct pisa_alloc *pa)
+{
+	unsigned long n = 0;
+	int cpu;
+
+	/* Pairs with smp_store_release() in pisa_enable_pcp(). */
+	if (!smp_load_acquire(&pa->pcp))
+		return 0;
+	for_each_possible_cpu(cpu)
+		n += READ_ONCE(per_cpu_ptr(pa->pcp, cpu)->count);
+	return n;
+}
+
 /* Return the blocks @cpu caches for @pa. */
 static void pisa_pcp_drain_cpu(struct pisa_alloc *pa, int cpu)
 {
@@ -1316,7 +1330,9 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 			pisa_pcp_push(pa, 1, &page);
 			return true;
 		}
-		pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
+		/* A compaction target drains the caches once, when it is done. */
+		if (!(READ_ONCE(meta->on_list) & PISA_LIST_HOLD))
+			pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
 	}
 
 	spin_lock_irqsave(&pa->lock, flags);
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 49fa5271a392f..25d3add7a4891 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -85,8 +85,8 @@ DEFINE_STATIC_KEY_FALSE(pisa_key);
 EXPORT_SYMBOL(pisa_key);
 
 static unsigned int pisa_compact __read_mostly;
-static unsigned int pisa_target_usage_percent __read_mostly;
-static unsigned int pisa_old_ms __read_mostly;
+unsigned int pisa_target_usage_percent __read_mostly;
+unsigned int pisa_old_ms __read_mostly;
 
 /* Enabled when pisa.compact > 0 or pisa.target_usage_percent > 0. */
 DEFINE_STATIC_KEY_FALSE(pisa_recycle_check_key);
@@ -189,6 +189,10 @@ bool pisa_place_meta(struct pisa_meta *meta)
 
 	lockdep_assert_held(&pa->lock);
 
+	/* A compaction target stays parked until pisa_user_scrub_alloc() is done. */
+	if (meta->on_list & PISA_LIST_HOLD)
+		return false;
+
 	if (avail == bpp) {
 		/* All blocks free: retire immediately if quarantined or over idle cap with idle_ms == 0. */
 		if (pa->num_idle_pages >= pa->max_idle_pages)
@@ -283,6 +287,10 @@ static bool pisa_meta_can_recycle(struct pisa_meta *meta)
 	if (unlikely(READ_ONCE(meta->flags) & PISA_QUARANTINED))
 		return false;
 
+	/* Blocks freed from a compaction target go back to it, not to a cache. */
+	if (READ_ONCE(meta->on_list) & PISA_LIST_HOLD)
+		return false;
+
 	if (!(READ_ONCE(meta->on_list) & PISA_LIST_SPARSE))
 		return true;
 
@@ -317,6 +325,9 @@ static bool pisa_meta_can_recycle(struct pisa_meta *meta)
 	}
 
 	WRITE_ONCE(meta->drain, true);
+	/* Let scrub migration move out what the callers still hold on @meta. */
+	if (READ_ONCE(pisa_scrub_migrate))
+		pisa_schedule_scrub(pa->nid);
 	return false;
 }
 
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index c013e384b1c7e..8d6fa9f36bdcf 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -248,7 +248,7 @@ void pisa_schedule_scrub(int nid)
 {
 	struct pisa_node *node;
 
-	if (!READ_ONCE(pisa_scrub_max))
+	if (!READ_ONCE(pisa_scrub_max) && !READ_ONCE(pisa_scrub_migrate))
 		return;
 
 	node = &pisa_nodes[nid];
@@ -277,8 +277,12 @@ static void pisa_node_work_fn(struct work_struct *work)
 	if (pisa_enabled() && READ_ONCE(pisa_reservoir_max))
 		pisa_reservoir_refill_node(node, nid);
 
-	if (test_and_clear_bit(PISA_NODE_SCRUB, &node->flags) && pisa_scrub_node(nid))
-		pisa_schedule_scrub(nid);
+	if (test_and_clear_bit(PISA_NODE_SCRUB, &node->flags)) {
+		bool more = pisa_scrub_node(nid);
+
+		if (pisa_user_scrub_node(nid) || more)
+			pisa_schedule_scrub(nid);
+	}
 }
 
 /* Queue node @nid's work if its reservoir is below or above the target. */
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index b18d461198ac5..c7f4e30579070 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -55,6 +55,7 @@ enum {
 	PISA_LIST_DIRTY		= BIT(3),	/* on @pa->partial_dirty */
 	PISA_LIST_MASK		= 0xf,
 	PISA_LIST_SPARSE	= BIT(4),	/* tail: list_move_tail() */
+	PISA_LIST_HOLD		= BIT(5),	/* parked on @pa->full by compaction */
 };
 
 /* Publish meta->flags into the pfn2slot array. */
@@ -133,10 +134,11 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
 struct pisa_node {
 	unsigned long		next_alloc;	/* atomic alloc backoff jiffies */
 	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
-	struct pisa_alloc	*user_pa;	/* user page migration pa */
+	struct pisa_alloc	*user_pa[3];	/* user/page-cache allocators by dir */
 
 	spinlock_t		lock;		/* protects @pages and @nr_pages */
 	unsigned int		nr_pages;	/* entries in @pages */
+	unsigned int		scrub_dir;	/* round-robin user_pa scrub index */
 	struct list_head	pages;		/* reservoir */
 	unsigned long		flags;		/* PISA_NODE_* requests */
 	struct work_struct	work;		/* refill/trim/scrub, on this node */
@@ -212,6 +214,7 @@ struct pisa_alloc *pisa_alloc_ensure(struct pisa_alloc **pap, int nid,
 #define PISA_PCP_HIGH		64U	/* blocks cached per CPU at most */
 int pisa_enable_pcp(struct pisa_alloc *pa);
 void pisa_pcp_drain(struct pisa_alloc *pa);
+unsigned long pisa_pcp_cached(struct pisa_alloc *pa);
 bool pisa_place_meta(struct pisa_meta *meta);
 extern struct llist_head pisa_free_list;
 extern unsigned int pisa_idle_ms;
@@ -229,6 +232,9 @@ static inline struct page *pisa_alloc(struct pisa_alloc *pa, gfp_t gfp)
 }
 
 #define PISA_OLD_MS_MAX		100000U
+extern unsigned int pisa_target_usage_percent;
+extern unsigned int pisa_old_ms;
+extern unsigned int pisa_scrub_migrate;
 unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
 			       unsigned int *nr, struct list_head *release_list,
 			       unsigned long nr_to_scan);
@@ -236,6 +242,7 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
 #define PISA_SCRUB_RUN_NS	(50 * NSEC_PER_USEC)
 extern unsigned int pisa_scrub_max;
 bool pisa_scrub_node(int nid);
+bool pisa_user_scrub_node(int nid);
 
 /* pisa-node.c: per-NUMA-node PISA page reservoir and work item */
 extern unsigned int pisa_reservoir_max;
diff --git a/mm/pisa-user.c b/mm/pisa-user.c
index 6c3efaee4ecda..5fb1ec7c45204 100644
--- a/mm/pisa-user.c
+++ b/mm/pisa-user.c
@@ -3,14 +3,22 @@
  * PISA user and page-cache folios: allocate or migrate order-0 folios into
  * per-node PISA allocators so zero-copy TX and block I/O use the PISA IOMMU mappings.
  */
+#include <linux/blkdev.h>
+#include <linux/device.h>
 #include <linux/pisa.h>
+#include <linux/fs.h>
 #include <linux/gfp.h>
 #include <linux/huge_mm.h>
+#include <linux/iommu.h>
+#include <linux/jiffies.h>
 #include <linux/migrate.h>
 #include <linux/mm.h>
 #include <linux/moduleparam.h>
 #include <linux/nodemask.h>
+#include <linux/pagemap.h>
 #include <linux/pagewalk.h>
+#include <linux/folio_batch.h>
+#include <linux/sched/clock.h>
 #include <linux/swap.h>
 #include <linux/uio.h>
 
@@ -20,9 +28,31 @@
 static unsigned long pisa_user_max_pages __read_mostly;
 DEFINE_STATIC_KEY_FALSE(pisa_user_key);
 
+/*
+ * Scrub-time migration of old sparse user_pa PISA pages when usage is below
+ * pisa.target_usage_percent (0 = off, 1 = compact in allocator, 2 = demote to the buddy allocator).
+ */
+unsigned int pisa_scrub_migrate __read_mostly;
+
+static int pisa_scrub_migrate_set(const char *val, const struct kernel_param *kp)
+{
+	int ret = param_set_uint_minmax(val, kp, 0, 2);
+
+	if (pisa_enabled() && !ret && pisa_scrub_migrate)
+		pisa_schedule_scrub_all();
+	return ret;
+}
+
+static const struct kernel_param_ops pisa_scrub_migrate_ops = {
+	.set = pisa_scrub_migrate_set,
+	.get = param_get_uint,
+};
+module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 0644);
+
 /* Create per-node user allocators and enable pisa_user_key (serialized by param_lock/init). */
 static int pisa_user_init(void)
 {
+	static const char * const dir_names[3] = { "bi", "tx", "rx" };
 	int nid;
 
 	if (pisa_user_enabled())
@@ -30,9 +60,11 @@ static int pisa_user_init(void)
 	if (pisa_meta_init())
 		return -ENOMEM;
 	for_each_node_state(nid, N_MEMORY) {
-		if (!pisa_alloc_ensure(&pisa_nodes[nid].user_pa, nid, 0,
-				       DMA_TO_DEVICE, false, "user_tx_n%d", nid))
-			return -ENOMEM;
+		for (unsigned int dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+			if (!pisa_alloc_ensure(&pisa_nodes[nid].user_pa[dir], nid, 0,
+					       dir, true, "user_%s_n%d", dir_names[dir], nid))
+				return -ENOMEM;
+		}
 	}
 	static_branch_enable(&pisa_user_key);
 	return 0;
@@ -73,16 +105,25 @@ void __init pisa_user_boot_enable(void)
 }
 
 /* User pages currently held in PISA blocks. */
-static unsigned long pisa_user_in_use(void)
+/* Blocks in use; @exact leaves out the free ones held in per-CPU caches (slower). */
+static unsigned long pisa_user_in_use(bool exact)
 {
-	unsigned long used = 0;
+	unsigned long used = 0, n;
 	int nid;
 
 	for_each_node_state(nid, N_MEMORY) {
-		/* Pairs with smp_store_release() in pisa_user_init(). */
-		const struct pisa_alloc *pa = smp_load_acquire(&pisa_nodes[nid].user_pa);
-
-		used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
+		for (unsigned int dir = 0; dir < ARRAY_SIZE(pisa_nodes[nid].user_pa); dir++) {
+			struct pisa_alloc *pa;
+
+			/* Pairs with smp_store_release() in pisa_user_init(). */
+			pa = smp_load_acquire(&pisa_nodes[nid].user_pa[dir]);
+			if (!pa)
+				continue;
+			n = READ_ONCE(pa->blocks_in_use);
+			if (exact)
+				n -= min(n, pisa_pcp_cached(pa));
+			used += n;
+		}
 	}
 	return used;
 }
@@ -90,18 +131,59 @@ static unsigned long pisa_user_in_use(void)
 /* Pages that can still be moved in before reaching pisa.user_max_pages. */
 static long pisa_user_budget(void)
 {
-	return (long)((READ_ONCE(pisa_user_max_pages) << PISA_ORDER) - pisa_user_in_use());
+	unsigned long max = READ_ONCE(pisa_user_max_pages) << PISA_ORDER;
+	long budget = (long)(max - pisa_user_in_use(false));
+
+	/* Free blocks in per-CPU caches count as in use: recount before refusing. */
+	if (budget <= 0)
+		budget = (long)(max - pisa_user_in_use(true));
+	return budget;
 }
 
-/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
-static bool pisa_user_want(struct folio *folio, bool to_pisa)
+/*
+ * Return true if @folio should be migrated into (@to_pisa) or out of PISA
+ * memory, promoting existing PISA folios to DMA_BIDIRECTIONAL when needed.
+ */
+static bool pisa_user_want_dir(struct folio *folio, bool to_pisa,
+			       enum dma_data_direction dir, bool allow_file)
 {
-	if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
-	    folio_test_ksm(folio) || is_huge_zero_folio(folio) ||
-	    (folio_test_large(folio) && !to_pisa) ||
-	    folio_maybe_mapped_shared(folio))
+	unsigned long pfn = folio_pfn(folio);
+
+	if (is_zero_folio(folio) || folio_is_zone_device(folio) ||
+	    folio_test_ksm(folio) || is_huge_zero_folio(folio))
+		return false;
+	if (folio_test_large(folio) && (!to_pisa || !folio_test_anon(folio)))
 		return false;
-	return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
+	if (folio_test_anon(folio)) {
+		if (folio_maybe_mapped_shared(folio))
+			return false;
+	} else if (!allow_file || !folio_mapping(folio) ||
+		   folio_test_locked(folio) || folio_test_writeback(folio)) {
+		return false;
+	}
+
+	if (!to_pisa)
+		return is_pisa_pfn(pfn);
+	if (!is_pisa_pfn(pfn)) {
+		struct address_space *mapping;
+		bool movable;
+
+		if (folio_test_anon(folio))
+			return true;
+		/* Only file folios whose mapping allows movable pages. */
+		rcu_read_lock();
+		mapping = folio_mapping(folio);
+		movable = mapping && (mapping_gfp_mask(mapping) & __GFP_MOVABLE);
+		rcu_read_unlock();
+		return movable;
+	}
+
+	if (dir == DMA_TO_DEVICE || dir == DMA_FROM_DEVICE) {
+		int need = (dir == DMA_FROM_DEVICE) ? IOMMU_WRITE : IOMMU_READ;
+
+		return (READ_ONCE(pisa_meta_of_pfn(pfn)->prot) & need) != need;
+	}
+	return false;
 }
 
 /* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
@@ -110,24 +192,46 @@ static bool pisa_user_busy(struct folio *folio, int extra)
 	return folio_ref_count(folio) > folio_expected_ref_count(folio) + extra;
 }
 
+/*
+ * Return true if @folio has references or pins beyond its mappings, swap
+ * cache, and the @refs entries in the caller's GUP batch.
+ */
+static bool pisa_user_batch_busy(struct folio *folio, int refs, bool pinned)
+{
+	int extra = (pinned && !folio_has_pincount(folio)) ?
+		    refs * GUP_PIN_COUNTING_BIAS : refs;
+
+	if (folio_has_pincount(folio) &&
+	    atomic_read(&folio->_pincount) > (pinned ? refs : 0))
+		return true;
+	return pisa_user_busy(folio, extra);
+}
+
 struct pisa_user_mtc {
-	gfp_t	gfp;
-	long	budget;
+	gfp_t			gfp;
+	long			budget;
+	enum dma_data_direction	dir;
+	bool			same_alloc;
 };
 
 /* new_folio_t for migration into PISA memory. */
 static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
 {
 	struct pisa_user_mtc *mtc = (void *)private;
+	bool already_pisa = is_pisa_pfn(folio_pfn(src));
+	enum dma_data_direction dir = mtc->dir;
 	int nid = folio_nid(src);
 	struct page *page;
 
 	/* Callers split large folios first (see pisa_user_split()); none is expected here. */
-	if (mtc->budget <= 0 || folio_test_large(src))
+	if ((!already_pisa && mtc->budget <= 0) || folio_test_large(src))
 		return NULL;
-	page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
+	if (already_pisa && !mtc->same_alloc)
+		dir = DMA_BIDIRECTIONAL;
+	page = pisa_alloc(pisa_nodes[nid].user_pa[dir], mtc->gfp);
 	if (!page)
 		return NULL;
+	/* Charged for every destination, so that pisa_user_put_folio() can undo it. */
 	mtc->budget--;
 	return page_folio(page);
 }
@@ -144,17 +248,31 @@ static void pisa_user_put_folio(struct folio *dst, unsigned long private)
 /* new_folio_t for migration out of PISA memory: allocators never serve these. */
 static struct folio *pisa_user_old_folio(struct folio *src, unsigned long private)
 {
-	return __folio_alloc_node(GFP_HIGHUSER_MOVABLE | __GFP_NOWARN, 0, folio_nid(src));
+	struct address_space *mapping = folio_mapping(src);
+	gfp_t gfp = GFP_HIGHUSER_MOVABLE;
+
+	/* A file folio goes back to memory that its mapping allows. */
+	if (!folio_test_anon(src) && mapping)
+		gfp = mapping_gfp_constraint(mapping, gfp);
+	return __folio_alloc_node(gfp | __GFP_NOWARN, 0, folio_nid(src));
 }
 
 /* Migrate the isolated folios on @list; failures are put back. */
-static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
+static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
+					  enum dma_data_direction dir,
+					  bool same_alloc, gfp_t gfp)
 {
-	struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
+	struct pisa_user_mtc mtc = {
+		.gfp = gfp,
+		.budget = LONG_MAX,
+		.dir = dir,
+		.same_alloc = same_alloc,
+	};
 	enum migrate_mode mode = MIGRATE_SYNC;
+	unsigned int done = 0;
 
 	if (list_empty(list))
-		return;
+		return 0;
 	if (to_pisa) {
 		mtc.budget = pisa_user_budget();
 		if (!gfpflags_allow_blocking(gfp))
@@ -162,8 +280,9 @@ static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
 	}
 	if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
 			  to_pisa ? pisa_user_put_folio : NULL, (unsigned long)&mtc,
-			  mode, MR_SYSCALL, NULL))
+			  mode, MR_SYSCALL, &done))
 		putback_movable_pages(list);
+	return done;
 }
 
 /*
@@ -221,22 +340,25 @@ static unsigned int pisa_user_split_isolated(struct folio *folio, struct list_he
 }
 
 /**
- * pisa_upgrade_user_pages - Migrate extracted user @pages into PISA
+ * pisa_upgrade_user_pages - Migrate extracted user @pages into PISA for @dir
  * @iter: Iterator from which @pages were just extracted
- * @pages: Extracted page array, one reference each
+ * @pages: Extracted page array
  * @size: Number of bytes extracted from @iter
  * @offset: Byte offset within @pages[0]
+ * @dir: I/O direction (DMA_TO_DEVICE or DMA_FROM_DEVICE)
+ * @pinned: true if @pages are FOLL_PIN-pinned, false if refcounted (+1)
  *
  * Return: false if no pages were isolated (@pages and @iter are untouched);
  * true otherwise, so the caller must re-extract @pages from the reverted @iter.
  */
 bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
-			     size_t size, size_t offset)
+			     size_t size, size_t offset,
+			     enum dma_data_direction dir, bool pinned)
 {
 	unsigned int nr = DIV_ROUND_UP(offset + size, PAGE_SIZE);
-	unsigned int want = 0;
+	unsigned int want = 0, need_budget = 0;
 	struct folio *f, *next;
-	bool drained = false;
+	bool drained = false, no_budget = false;
 	LIST_HEAD(small);
 	LIST_HEAD(list);
 	long budget;
@@ -247,15 +369,29 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 	 * First pass: check if any page needs migration and whether budget is
 	 * available before draining per-CPU LRU caches or isolating folios.
 	 */
-	for (unsigned int i = 0; i < nr; i++)
-		want += pisa_user_want(page_folio(pages[i]), true);
-	if (!want || pisa_user_budget() <= 0)
+	for (unsigned int i = 0; i < nr; i++) {
+		struct folio *folio = page_folio(pages[i]);
+
+		if (pisa_user_want_dir(folio, true, dir, false)) {
+			want++;
+			need_budget += !is_pisa_pfn(folio_pfn(folio));
+		}
+	}
+	if (!want)
 		return false;
+	if (need_budget && pisa_user_budget() <= 0) {
+		if (want == need_budget)
+			return false;
+		no_budget = true;
+	}
 	for (unsigned int i = 0; i < nr; i++) {
 		struct folio *folio = page_folio(pages[i]);
 		unsigned int refs = 0;
 
-		if (!pisa_user_want(folio, true))
+		if (!pisa_user_want_dir(folio, true, dir, false))
+			continue;
+		/* Out of budget: only PISA folios that need another direction. */
+		if (no_budget && !is_pisa_pfn(folio_pfn(folio)))
 			continue;
 		/* Drain per-CPU LRU batches once so freshly faulted folios can be isolated. */
 		if (!drained && !folio_test_lru(folio)) {
@@ -266,8 +402,7 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			continue;
 		for (unsigned int j = 0; j < nr; j++)
 			refs += page_folio(pages[j]) == folio;
-		/* The references from iov_iter_get_pages2() are expected. */
-		if (folio_maybe_dma_pinned(folio) || pisa_user_busy(folio, refs))
+		if (pisa_user_batch_busy(folio, refs, pinned))
 			continue;
 		if (isolate_folio_to_list(folio, &list))
 			list_move_tail(&folio->lru, &list);
@@ -275,11 +410,15 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 	if (list_empty(&list))
 		return false;
 	/*
-	 * Release the caller's references on @pages before migrating
+	 * Release the caller's references/pins on @pages before migrating
 	 * (migrate_pages() rejects folios with extra references).
 	 */
-	for (unsigned int i = 0; i < nr; i++)
-		put_page(pages[i]);
+	if (pinned) {
+		unpin_user_pages(pages, nr);
+	} else {
+		for (unsigned int i = 0; i < nr; i++)
+			put_page(pages[i]);
+	}
 	/*
 	 * Split large folios while budget remains and keep only their subpages
 	 * that are in @pages: migrate_pages() would split them regardless of
@@ -293,15 +432,307 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			budget -= !is_pisa_pfn(folio_pfn(f));
 	}
 	list_splice_tail(&small, &list);
-	pisa_user_migrate(&list, true,
-			  GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+	pisa_user_migrate_dir(&list, true, dir, false,
+			      GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
 	/* Rewind @iter so the caller re-extracts the updated page pointers. */
 	iov_iter_revert(iter, size);
 	return true;
 }
 
+/* Migrate the single-page segments already added to @bio; the caller then extracts them again. */
+bool __pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter,
+			      enum dma_data_direction dir)
+{
+	ssize_t size = bio->bi_iter.bi_size;
+	struct page *bv_pages[64];
+
+	if (!bio->bi_vcnt || bio->bi_vcnt > ARRAY_SIZE(bv_pages))
+		return false;
+	for (unsigned int i = 0; i < bio->bi_vcnt; i++) {
+		const struct bio_vec *bv = &bio->bi_io_vec[i];
+
+		/* Only segments that lie within one page. */
+		if ((bv->bv_offset + bv->bv_len - 1) / PAGE_SIZE != bv->bv_offset / PAGE_SIZE)
+			return false;
+		bv_pages[i] = bv->bv_page;
+	}
+	if (!pisa_upgrade_user_pages(iter, bv_pages, size,
+				     (size_t)bio->bi_vcnt * PAGE_SIZE - size,
+				     dir, true))
+		return false;
+	bio->bi_vcnt = 0;
+	bio->bi_iter.bi_size = 0;
+	return true;
+}
+
+/*
+ * Return true if @bdev (or one of its parent devices) has opted into PISA for
+ * @dir (pisa_rx_bufs for reads, pisa_tx_bufs for writes).
+ */
+bool pisa_bdev_enabled(const struct block_device *bdev,
+		       enum dma_data_direction dir)
+{
+	if (!bdev || !bdev->bd_disk)
+		return false;
+	for (struct device *dev = disk_to_dev(bdev->bd_disk); dev; dev = dev->parent) {
+		if (dir == DMA_FROM_DEVICE ?
+		    READ_ONCE(dev->pisa_rx_bufs) :
+		    READ_ONCE(dev->pisa_tx_bufs))
+			return true;
+	}
+	return false;
+}
+
+static bool pisa_mapping_enabled(const struct address_space *mapping,
+				 enum dma_data_direction dir)
+{
+	struct super_block *sb;
+
+	if (!mapping || !mapping->host || !mapping->host->i_sb)
+		return false;
+	sb = mapping->host->i_sb;
+	return pisa_bdev_enabled(sb_is_blkdev_sb(sb) ?
+				 I_BDEV(mapping->host) : sb->s_bdev, dir);
+}
+
+/**
+ * __pisa_filemap_alloc - Allocate an order-0 page-cache folio from a PISA user allocator
+ * @mapping: Page-cache address_space
+ * @gfp: GFP flags (__GFP_WRITE indicates buffered write path)
+ *
+ * Return: Newly allocated order-0 folio, or NULL on failure.
+ */
+struct folio *__pisa_filemap_alloc(const struct address_space *mapping,
+				   gfp_t gfp)
+{
+	bool is_write = gfp & __GFP_WRITE;
+	enum dma_data_direction dir = is_write ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+	struct page *page;
+	int nid;
+
+	/* User PISA pages are compacted by migration: no unmovable folios in them. */
+	if (!(gfp & __GFP_MOVABLE))
+		return NULL;
+	if (!pisa_mapping_enabled(mapping, dir))
+		return NULL;
+	if (pisa_mapping_enabled(mapping, is_write ? DMA_FROM_DEVICE : DMA_TO_DEVICE))
+		dir = DMA_BIDIRECTIONAL;
+	if (pisa_user_budget() <= 0)
+		return NULL;
+	nid = pisa_sanitize_nid(numa_node_id());
+	page = pisa_alloc(pisa_nodes[nid].user_pa[dir], gfp);
+	if (!page)
+		return NULL;
+	return page_folio(page);
+}
+
+static bool pisa_folio_upgrade(struct folio *folio, enum dma_data_direction dir,
+			       struct folio_batch *fbatch)
+{
+	LIST_HEAD(list);
+
+	if (!pisa_user_want_dir(folio, true, dir, true) ||
+	    (!is_pisa_pfn(folio_pfn(folio)) && pisa_user_budget() <= 0))
+		return false;
+	if (!folio_test_lru(folio))
+		lru_add_drain();
+	if (pisa_user_busy(folio, 1) || !isolate_folio_to_list(folio, &list))
+		return false;
+	/* Drop caller's reference so migrate_pages() sees only expected refs. */
+	if (fbatch)
+		folio_batch_release(fbatch);
+	else
+		folio_put(folio);
+	pisa_user_migrate_dir(&list, true, dir, false,
+			      GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
+	return true;
+}
+
+bool __pisa_filemap_upgrade(const struct address_space *mapping,
+			    struct folio *folio)
+{
+	return pisa_mapping_enabled(mapping, DMA_TO_DEVICE) &&
+	       pisa_folio_upgrade(folio, DMA_TO_DEVICE, NULL);
+}
+
+bool __pisa_writeback_upgrade(const struct address_space *mapping,
+			      struct folio *folio, pgoff_t *index,
+			      struct folio_batch *fbatch)
+{
+	pgoff_t idx;
+
+	if (!pisa_mapping_enabled(mapping, DMA_TO_DEVICE))
+		return false;
+	idx = folio->index;
+	if (!pisa_folio_upgrade(folio, DMA_TO_DEVICE, fbatch))
+		return false;
+	*index = idx;
+	return true;
+}
+
 #define PISA_USER_BATCH	512
 
+/* Under RCU: is @pfn still a block of @target, and @target still owned by @pa? */
+static bool pisa_user_block_of(unsigned long pfn, const struct pisa_meta *target,
+			       const struct pisa_alloc *pa)
+{
+	return is_pisa_pfn(pfn) && pisa_meta_of_pfn(pfn) == target &&
+	       READ_ONCE(target->pa) == pa;
+}
+
+static unsigned long pisa_user_scrub_alloc(struct pisa_alloc *pa,
+					   enum dma_data_direction dir,
+					   unsigned int mode,
+					   unsigned long budget)
+{
+	unsigned int target_pct = READ_ONCE(pisa_target_usage_percent);
+	unsigned int old_ms = READ_ONCE(pisa_old_ms);
+	unsigned long bpp = PISA_BLOCKS(0), in_use;
+	unsigned long max_age, base_pfn = 0, flags;
+	DECLARE_BITMAP(inuse_map, _PISA_MAX_BLOCKS);
+	struct list_head *lists[] = { &pa->partial_dirty, &pa->partial };
+	struct pisa_meta *meta, *target = NULL;
+	unsigned int idx, count = 0;
+	unsigned long migrated = 0;
+	LIST_HEAD(list);
+	long held;
+
+	if (!target_pct || !old_ms)
+		return 0;
+
+	pisa_pcp_drain(pa);
+
+	held = pisa_alloc_held_pages(pa);
+	in_use = READ_ONCE(pa->blocks_in_use);
+	if (held <= 0 || in_use * 100 >= (unsigned long)held * bpp * target_pct)
+		return 0;
+
+	max_age = msecs_to_jiffies(min(old_ms, PISA_OLD_MS_MAX));
+
+	spin_lock_irqsave(&pa->lock, flags);
+	for (unsigned int i = 0; i < ARRAY_SIZE(lists) && !target; i++) {
+		list_for_each_entry_reverse(meta, lists[i], list) {
+			if (pisa_meta_avail(meta) < (bpp >> 1))
+				break;
+			/* Compaction needs a sparse page and another partial page to fill. */
+			if (mode == 1 && (!(meta->on_list & PISA_LIST_SPARSE) ||
+					  (list_is_first(&meta->list, lists[i]) &&
+					   list_empty(lists[!i]))))
+				break;
+			if (time_after(jiffies, meta->alloc_jiffies + max_age)) {
+				target = meta;
+				break;
+			}
+		}
+	}
+	if (target) {
+		/*
+		 * Compaction: park @target on @pa->full, so that the blocks
+		 * freed from it are neither cached nor handed out again while
+		 * its folios move to other pages of @pa.
+		 */
+		if (mode == 1) {
+			WRITE_ONCE(target->on_list, PISA_LIST_FULL | PISA_LIST_HOLD);
+			list_move(&target->list, &pa->full);
+		}
+		base_pfn = pisa_meta_to_pfn(target);
+		bitmap_or(inuse_map, target->clean_bitmap, target->dirty_bitmap, bpp);
+		bitmap_complement(inuse_map, inuse_map, bpp);
+	}
+	spin_unlock_irqrestore(&pa->lock, flags);
+
+	if (!target)
+		return 0;
+
+	for_each_set_bit(idx, inuse_map, bpp) {
+		unsigned long pfn = base_pfn + idx;
+		struct folio *folio;
+
+		rcu_read_lock();
+		if (!pisa_user_block_of(pfn, target, pa)) {
+			rcu_read_unlock();
+			break;
+		}
+		folio = page_folio(pfn_to_page(pfn));
+		if (!folio_try_get(folio)) {
+			rcu_read_unlock();
+			continue;
+		}
+		if (!pisa_user_block_of(pfn, target, pa) ||
+		    folio != page_folio(pfn_to_page(pfn))) {
+			rcu_read_unlock();
+			folio_put(folio);
+			continue;
+		}
+		rcu_read_unlock();
+
+		if (!pisa_user_want_dir(folio, false, dir, true) ||
+		    pisa_user_busy(folio, 1) ||
+		    !isolate_folio_to_list(folio, &list)) {
+			folio_put(folio);
+			continue;
+		}
+		folio_put(folio);
+		if (++count >= min_t(unsigned long, budget, PISA_USER_BATCH))
+			break;
+	}
+
+	if (!list_empty(&list)) {
+		migrated = pisa_user_migrate_dir(&list, mode == 1, dir, true,
+						 GFP_KERNEL | __GFP_NOWARN);
+		pisa_pcp_drain(pa);
+	}
+
+	if (mode == 1) {
+		bool should_release;
+
+		/* Put @target back on the list that its free blocks call for. */
+		spin_lock_irqsave(&pa->lock, flags);
+		WRITE_ONCE(target->on_list, PISA_LIST_FULL);
+		should_release = pisa_place_meta(target);
+		spin_unlock_irqrestore(&pa->lock, flags);
+		if (should_release)
+			pisa_schedule_reclaim();
+	}
+
+	/* Nothing moved: do not pick @target again before it ages. */
+	if (!migrated && READ_ONCE(target->pa) == pa)
+		WRITE_ONCE(target->alloc_jiffies, jiffies);
+
+	return migrated;
+}
+
+bool pisa_user_scrub_node(int nid)
+{
+	unsigned long budget = READ_ONCE(pisa_scrub_max) ?: 512;
+	unsigned int mode = READ_ONCE(pisa_scrub_migrate);
+	unsigned int n_dirs = ARRAY_SIZE(pisa_nodes[nid].user_pa);
+	u64 deadline = local_clock() + PISA_SCRUB_RUN_NS;
+	bool progress = false;
+
+	if (!mode || mode > 2)
+		return false;
+
+	for (unsigned int i = 0; i < n_dirs; i++) {
+		unsigned int dir = pisa_nodes[nid].scrub_dir++ % n_dirs;
+		/* Pairs with smp_store_release() in pisa_user_init(). */
+		struct pisa_alloc *pa = smp_load_acquire(&pisa_nodes[nid].user_pa[dir]);
+		unsigned long n;
+
+		if (!pa)
+			continue;
+		n = pisa_user_scrub_alloc(pa, dir, mode, budget);
+		if (n >= budget || local_clock() >= deadline)
+			return true;
+		if (n) {
+			budget -= n;
+			progress = true;
+			cond_resched();
+		}
+	}
+	return progress;
+}
+
 struct pisa_madvise_walk {
 	struct list_head	list;	/* isolated folios waiting to be migrated */
 	unsigned int		n;	/* pages on @list */
@@ -329,7 +760,8 @@ static int pisa_madvise_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long
 			return 1;
 		folio = folio_walk_start(&fw, walk->vma, addr, 0);
 		if (folio) {
-			bool want = pisa_user_want(folio, to_pisa) && folio_test_lru(folio);
+			bool want = pisa_user_want_dir(folio, to_pisa, DMA_TO_DEVICE, false) &&
+				    folio_test_lru(folio);
 			bool busy = want && pisa_user_busy(folio, 0);
 			bool split = want && !busy && folio_test_large(folio);
 
@@ -352,7 +784,8 @@ static int pisa_madvise_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long
 			}
 		}
 		if (w->n >= PISA_USER_BATCH) {
-			pisa_user_migrate(&w->list, to_pisa, PISA_MADVISE_GFP);
+			pisa_user_migrate_dir(&w->list, to_pisa, DMA_TO_DEVICE, false,
+					      PISA_MADVISE_GFP);
 			w->n = 0;
 		}
 		addr += nr << PAGE_SHIFT;
@@ -389,6 +822,6 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
 	lru_add_drain();
 	/* The page walker skips the unpopulated page-table levels of sparse mappings. */
 	ret = walk_page_range_vma(vma, start, end, &pisa_madvise_walk_ops, &w);
-	pisa_user_migrate(&w.list, to_pisa, PISA_MADVISE_GFP);
+	pisa_user_migrate_dir(&w.list, to_pisa, DMA_TO_DEVICE, false, PISA_MADVISE_GFP);
 	return ret < 0 ? ret : 0;
 }
diff --git a/mm/readahead.c b/mm/readahead.c
index 6e5563290287e..82b03ac1161dd 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -114,6 +114,7 @@
  */
 
 #include <linux/blk_plug.h>
+#include <linux/pisa-mm.h>
 #include <linux/kernel.h>
 #include <linux/dax.h>
 #include <linux/gfp.h>
@@ -197,7 +198,7 @@ static struct folio *ractl_alloc_folio(struct readahead_control *ractl,
 {
 	struct folio *folio;
 
-	folio = filemap_alloc_folio(gfp_mask, order, NULL);
+	folio = pisa_filemap_alloc_folio(gfp_mask, order, NULL, ractl->mapping);
 	if (folio && ractl->dropbehind)
 		__folio_set_dropbehind(folio);
 
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (11 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
                     ` (13 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the
physical DMA parent (dev->dev.parent), whereas a socket route (dst->dev)
or skb->dev may point to an upper virtual net_device (bond, VLAN,
ipvlan, etc.). Walking lower devices via netdev_walk_all_lower_dev_rcu()
on every TX allocation would be too expensive.

Add dev->pisa_tx_flags_tstamp on struct net_device to cache the leaf device
PISA TX flags and refresh them at most once every 256 jiffies via
pisa_tx_flags_refresh(). Also add the net_pisa_tx_bufs_key static key
and helpers (use_pisa_tx_bufs(), pisa_tx_bufs_gfp()) for subsequent TX
buffer patches.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/base/core.c       |   6 ++-
 include/linux/device.h    |  11 ++++
 include/linux/netdevice.h |   4 ++
 include/net/pisa.h        |  81 +++++++++++++++++++++++++++++
 mm/pisa-kunit.c           |   3 +-
 net/core/Makefile         |   1 +
 net/core/pisa-net.c       | 104 ++++++++++++++++++++++++++++++++++++++
 7 files changed, 207 insertions(+), 3 deletions(-)
 create mode 100644 include/net/pisa.h
 create mode 100644 net/core/pisa-net.c

diff --git a/drivers/base/core.c b/drivers/base/core.c
index df93b6b6e125e..5f1291e644c5e 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2950,7 +2950,7 @@ static DEVICE_ATTR_RW(name)
 
 PISA_BOOL_ATTR(pisa_coherent);
 PISA_BOOL_ATTR(pisa_rx_bufs);
-PISA_BOOL_ATTR(pisa_tx_bufs);
+PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
 PISA_BOOL_ATTR(pisa_tx_skb_heads);
 PISA_BOOL_ATTR(pisa_tx_zerocopy);
 
@@ -3111,8 +3111,10 @@ static void device_remove_attrs(struct device *dev)
 	const struct class *class = dev->class;
 	const struct device_type *type = dev->type;
 
-	if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask)
+	if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
 		device_remove_group(dev, &dev_attr_pisa_group);
+		net_pisa_tx_bufs_set(dev, false);
+	}
 
 	if (dev->physical_location) {
 		device_remove_group(dev, &dev_attr_physical_location_group);
diff --git a/include/linux/device.h b/include/linux/device.h
index e705df38e49af..0d867d8255ba5 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -817,6 +817,17 @@ struct device {
 	DECLARE_BITMAP(flags, DEV_FLAG_COUNT);
 };
 
+#if defined(CONFIG_NET) && defined(CONFIG_PISA)
+int net_pisa_tx_bufs_set(struct device *dev, bool val);
+#else
+/* Without CONFIG_NET the flag still gates block-device writes (see pisa_bdev_enabled()). */
+static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+	WRITE_ONCE(dev->pisa_tx_bufs, val);
+	return 0;
+}
+#endif
+
 #define __create_dev_flag_accessors(accessor_name, flag_name) \
 static inline bool dev_##accessor_name(const struct device *dev) \
 { \
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 435be7880d7e5..f973faba4402c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2161,6 +2161,7 @@ enum netdev_reg_state {
  *	@neighbours:	List heads pointing to this device's neighbours'
  *			dev_list, one per address-family.
  *	@hwprov: Tracks which PTP performs hardware packet time stamping.
+ *	@pisa_tx_flags_tstamp: Cached leaf pisa flags and timestamp.
  *
  *	FIXME: cleanup struct net_device such that network protocol info
  *	moves out.
@@ -2276,6 +2277,9 @@ struct net_device {
 	} adj_list;
 
 	/* Read-mostly cache-line for fast-path access */
+#ifdef CONFIG_PISA
+	u32			pisa_tx_flags_tstamp;
+#endif
 	xdp_features_t		xdp_features;
 	const struct xdp_metadata_ops *xdp_metadata_ops;
 	const struct xsk_tx_metadata_ops *xsk_tx_metadata_ops;
diff --git a/include/net/pisa.h b/include/net/pisa.h
new file mode 100644
index 0000000000000..1f95e67e99bf8
--- /dev/null
+++ b/include/net/pisa.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause */
+/* PISA hooks for the networking stack. */
+#ifndef _NET_PISA_H
+#define _NET_PISA_H
+
+#include <linux/pisa.h>
+#include <net/sock.h>
+
+/*
+ * struct net_device caches the PISA enable flags for tx_bufs and tx_heads used
+ * by the leaf device, refreshed every 256 jiffies. The helpers below decide
+ * whether to use PISA memory for a given dst, sk, skb.
+ */
+#ifdef CONFIG_PISA
+
+enum pisa_tx_flags {
+	PISA_TX_FLAG_BUFS	= BIT(0),
+	PISA_TX_XFRM_OK		= BIT(2),
+	PISA_TX_TSTAMP_MASK	= ~0xffU,
+};
+
+/*
+ * Static keys guarding networking hooks:
+ * - @net_pisa_tx_bufs_key: enabled while any device has pisa_tx_bufs set.
+ */
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+
+u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now);
+
+/*
+ * May TX data routed through @dst use PISA pages? Those pages stay readable by
+ * every device that mapped them for as long as they are active, so only data
+ * that leaves the host in clear qualifies: IPsec payload before encryption has
+ * dst_xfrm(dst) != NULL and is rejected unless PISA_TX_XFRM_OK is set.
+ */
+static inline gfp_t __pisa_tx_gfp(const struct dst_entry *dst,
+				  enum pisa_tx_flags flag)
+{
+	const struct net_device *dev;
+	u32 stamp, now;
+
+	guard(rcu)();
+	if (!dst || (dst_xfrm(dst) && !(flag & PISA_TX_XFRM_OK)))
+		return 0;
+	dev = dst_dev_rcu(dst);
+	if (!dev)
+		return 0;
+	stamp = READ_ONCE(dev->pisa_tx_flags_tstamp);
+	now = (u32)jiffies & PISA_TX_TSTAMP_MASK;
+	if (unlikely((stamp & PISA_TX_TSTAMP_MASK) != now))
+		stamp = pisa_tx_flags_refresh((struct net_device *)dev, now);
+	return (stamp & flag) ? __GFP_PISA : 0;
+}
+
+/*
+ * ============================================================================
+ * Networking hooks (every hook starts inline with a static-key check)
+ * ============================================================================
+ */
+
+/* Hook for route-based TX payload fragment selection (@net_pisa_tx_bufs_key). */
+static inline bool use_pisa_tx_bufs(const struct dst_entry *dst)
+{
+	return static_branch_unlikely(&net_pisa_tx_bufs_key) &&
+	       __pisa_tx_gfp(dst, PISA_TX_FLAG_BUFS);
+}
+
+/* Hook for ESP output skb_page_frag_refill() (@net_pisa_tx_bufs_key). */
+static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb)
+{
+	return static_branch_unlikely(&net_pisa_tx_bufs_key) ?
+	       __pisa_tx_gfp(skb_dst(skb), PISA_TX_FLAG_BUFS | PISA_TX_XFRM_OK) : 0;
+}
+
+#else /* !CONFIG_PISA */
+
+static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) { return false; }
+static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) { return 0; }
+
+#endif /* CONFIG_PISA */
+#endif /* _NET_PISA_H */
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 4effbc8610079..d7670c76e98c5 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -578,7 +578,8 @@ static void test_dev_pisa_attrs(struct kunit *test)
 	kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
 	kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
 	kunit_check_pisa_attr(test, dev, "pisa_tx_zerocopy", &dev->pisa_tx_zerocopy);
-	kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
+	if (IS_ENABLED(CONFIG_NET))
+		kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
 	kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
 	device_unregister(dev);
 }
diff --git a/net/core/Makefile b/net/core/Makefile
index c20e520358b8e..3c84358d6b0c0 100644
--- a/net/core/Makefile
+++ b/net/core/Makefile
@@ -52,3 +52,4 @@ obj-$(CONFIG_NET_TEST) += net_test.o
 obj-$(CONFIG_NET_DEVMEM) += devmem.o
 obj-$(CONFIG_DEBUG_NET) += lock_debug.o
 obj-$(CONFIG_FAIL_SKB_REALLOC) += skb_fault_injection.o
+obj-$(CONFIG_PISA) += pisa-net.o
diff --git a/net/core/pisa-net.c b/net/core/pisa-net.c
new file mode 100644
index 0000000000000..6d4909c82070c
--- /dev/null
+++ b/net/core/pisa-net.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+/*
+ * PISA networking helpers: leaf device TX flag cache, small skb head slab
+ * cache, and MSG_ZEROCOPY user page upgrade.
+ * See Documentation/core-api/pisa.rst for the architecture overview.
+ */
+
+#include <linux/device.h>
+#include <linux/export.h>
+#include <linux/mutex.h>
+#include <linux/netdevice.h>
+#include <linux/pisa.h>
+#include <net/pisa.h>
+
+/* Enabled while at least one device has pisa_tx_bufs set. */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+EXPORT_SYMBOL(net_pisa_tx_bufs_key);
+
+int net_pisa_tx_bufs_set(struct device *dev, bool val)
+{
+	static DEFINE_MUTEX(lock);
+
+	guard(mutex)(&lock);
+	if (dev->pisa_tx_bufs != val) {
+		WRITE_ONCE(dev->pisa_tx_bufs, val);
+		if (val)
+			static_branch_inc(&net_pisa_tx_bufs_key);
+		else
+			static_branch_dec(&net_pisa_tx_bufs_key);
+	}
+	return 0;
+}
+
+/*
+ * The sysfs attribute pisa_tx_bufs lives on the device that does DMA: the
+ * closest ancestor of a leaf net_device that has a dma_mask (dev->dev.parent,
+ * or further up behind bus glue such as virtio). A socket route (dst->dev) may
+ * instead point to an upper virtual net_device (bond, VLAN, ipvlan, etc.),
+ * whose leaves are its lower devices. Walking lower devices via
+ * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
+ * expensive, so dev->pisa_tx_flags_tstamp caches the walk result on the net_device
+ * and refreshes it at most once every 256 jiffies:
+ *   bit 0 (PISA_TX_FLAG_BUFS):       pisa_tx_bufs enabled on lower device(s)
+ *   bits 31..8 (PISA_TX_TSTAMP_MASK):  upper 24 bits of (u32)jiffies
+ */
+static u32 pisa_tx_flags(const struct net_device *dev)
+{
+	const struct device *parent = dev->dev.parent;
+	u32 flags = 0;
+
+	while (parent && !parent->dma_mask)
+		parent = parent->parent;
+	if (parent && READ_ONCE(parent->pisa_tx_bufs))
+		flags |= PISA_TX_FLAG_BUFS;
+	return flags;
+}
+
+/* MACsec encrypts in software above its lower device: its cleartext must stay out of PISA. */
+static bool pisa_tx_dev_excluded(const struct net_device *dev)
+{
+	return netif_is_macsec(dev);
+}
+
+/* A leaf has no lower devices; what its parent device is does not matter. */
+static bool pisa_tx_dev_is_leaf(const struct net_device *dev)
+{
+	return list_empty(&dev->adj_list.lower);
+}
+
+/* Every leaf must have opted in; a leaf without a DMA device (veth, tun, ...) has not. */
+static int pisa_tx_flags_walk(struct net_device *dev, struct netdev_nested_priv *priv)
+{
+	uintptr_t flags = (uintptr_t)priv->data;
+
+	if (pisa_tx_dev_excluded(dev))
+		flags = BIT(2);
+	else if (pisa_tx_dev_is_leaf(dev))
+		flags = (flags & pisa_tx_flags(dev)) | BIT(2);
+	priv->data = (void *)flags;
+	return 0;
+}
+
+/* Slow path of __pisa_tx_gfp(): refresh dev->pisa_tx_flags_tstamp. */
+u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now)
+{
+	if (pisa_tx_dev_excluded(dev)) {
+		/* No flags. */
+	} else if (pisa_tx_dev_is_leaf(dev)) {
+		now |= pisa_tx_flags(dev);
+	} else {
+		struct netdev_nested_priv priv = {
+			.data = (void *)(uintptr_t)PISA_TX_FLAG_BUFS,
+		};
+
+		rcu_read_lock();
+		netdev_walk_all_lower_dev_rcu(dev, pisa_tx_flags_walk, &priv);
+		rcu_read_unlock();
+		if ((uintptr_t)priv.data & BIT(2))
+			now |= (uintptr_t)priv.data & ~BIT(2);
+	}
+	WRITE_ONCE(dev->pisa_tx_flags_tstamp, now);
+	return now;
+}
+EXPORT_SYMBOL(pisa_tx_flags_refresh);
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill()
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (12 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 15/25] net: Use PISA for page_pool memory Luigi Rizzo
                     ` (12 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Transmit socket buffers use memory coming from skb_page_frag_refill().
Sockets whose leaf device has enabled pisa_tx_bufs should use PISA
memory.

Add a dedicated per-task page_frag, current->task_frag_pisa, populated
with read-only PISA blocks via __GFP_PISA, and select it on TX when
the socket's (or cork's) route goes out in the clear through a PISA-enabled
device.

Special paths that hold plaintext before encryption (IPsec, kTLS) or
do not use a DMA device keep using regular pages; ESP output uses
pisa_tx_bufs_gfp() to opt in when the egress device has pisa_tx_bufs
enabled.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa.h  | 15 +++++++++++++++
 include/linux/sched.h |  4 ++++
 include/net/pisa.h    | 33 +++++++++++++++++++++++++++++++++
 kernel/exit.c         |  2 ++
 kernel/fork.c         |  2 ++
 net/core/sock.c       | 10 ++++++++++
 net/ipv4/esp4.c       |  7 +++++--
 net/ipv4/ip_output.c  | 12 ++++++++----
 net/ipv4/tcp.c        |  3 ++-
 net/ipv6/esp6.c       |  7 +++++--
 net/ipv6/ip6_output.c | 12 ++++++++----
 11 files changed, 94 insertions(+), 13 deletions(-)

diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 6d184e393e297..84692852299f1 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -202,6 +202,18 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 	       __pisa_page_can_recycle(page);
 }
 
+/* Hooks for dup_task_struct() and do_exit() (guarded by @pisa_key on release). */
+static inline void pisa_task_frag_init(struct task_struct *tsk)
+{
+	tsk->task_frag_pisa.page = NULL;
+}
+
+static inline void pisa_task_frag_release(struct task_struct *tsk)
+{
+	if (pisa_enabled() && tsk->task_frag_pisa.page)
+		put_page(tsk->task_frag_pisa.page);
+}
+
 #else /* !CONFIG_PISA */
 
 static inline int pisa_meta_init(void) { return -EOPNOTSUPP; }
@@ -240,6 +252,9 @@ static inline bool pisa_page_can_recycle(const struct page *page)
 	return true;
 }
 
+static inline void pisa_task_frag_init(struct task_struct *tsk) {}
+static inline void pisa_task_frag_release(struct task_struct *tsk) {}
+
 static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
 					   struct page **pages, size_t size,
 					   size_t offset,
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d78555..a8edeeb2661f4 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1455,6 +1455,10 @@ struct task_struct {
 	struct pipe_inode_info		*splice_pipe;
 
 	struct page_frag		task_frag;
+#ifdef CONFIG_PISA
+	/* TX frag that may hold PISA pages, see sk_page_frag_dst(). */
+	struct page_frag		task_frag_pisa;
+#endif
 
 #ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE
 	struct lazy_mmu_state		lazy_mmu_state;
diff --git a/include/net/pisa.h b/include/net/pisa.h
index 1f95e67e99bf8..8e23d8b01e73d 100644
--- a/include/net/pisa.h
+++ b/include/net/pisa.h
@@ -4,6 +4,7 @@
 #define _NET_PISA_H
 
 #include <linux/pisa.h>
+#include <linux/tcp.h>
 #include <net/sock.h>
 
 /*
@@ -72,10 +73,42 @@ static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb)
 	       __pisa_tx_gfp(skb_dst(skb), PISA_TX_FLAG_BUFS | PISA_TX_XFRM_OK) : 0;
 }
 
+/* Hook for skb_page_frag_refill() and __ip*_append_data() (@net_pisa_tx_bufs_key). */
+static inline bool is_pisa_pfrag(const struct page_frag *pfrag)
+{
+	return static_branch_unlikely(&net_pisa_tx_bufs_key) &&
+	       pfrag == &current->task_frag_pisa;
+}
+
+/* Hook for task/socket TX page_frag selection (@net_pisa_tx_bufs_key). */
+static inline struct page_frag *task_page_frag(const struct dst_entry *dst)
+{
+	return use_pisa_tx_bufs(dst) ? &current->task_frag_pisa : &current->task_frag;
+}
+
+static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
+						 const struct dst_entry *dst)
+{
+	return static_branch_unlikely(&net_pisa_tx_bufs_key) &&
+	       sk->sk_use_task_frag ? task_page_frag(dst) : sk_page_frag(sk);
+}
+
 #else /* !CONFIG_PISA */
 
 static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) { return false; }
 static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) { return 0; }
+static inline bool is_pisa_pfrag(const struct page_frag *pfrag) { return false; }
+
+static inline struct page_frag *task_page_frag(const struct dst_entry *dst)
+{
+	return &current->task_frag;
+}
+
+static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
+						 const struct dst_entry *dst)
+{
+	return sk_page_frag(sk);
+}
 
 #endif /* CONFIG_PISA */
 #endif /* _NET_PISA_H */
diff --git a/kernel/exit.c b/kernel/exit.c
index d5e7bc2af804a..ac7ca1fb180c3 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -6,6 +6,7 @@
  */
 
 #include <linux/mm.h>
+#include <linux/pisa.h>
 #include <linux/slab.h>
 #include <linux/sched/autogroup.h>
 #include <linux/sched/mm.h>
@@ -1009,6 +1010,7 @@ void __noreturn do_exit(long code)
 
 	if (tsk->task_frag.page)
 		put_page(tsk->task_frag.page);
+	pisa_task_frag_release(tsk);
 
 	exit_task_stack_account(tsk);
 
diff --git a/kernel/fork.c b/kernel/fork.c
index 5efceb8d58a78..d235d7ab04687 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -44,6 +44,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/fs.h>
 #include <linux/mm.h>
+#include <linux/pisa.h>
 #include <linux/mm_inline.h>
 #include <linux/memblock.h>
 #include <linux/nsproxy.h>
@@ -982,6 +983,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 #endif
 	tsk->splice_pipe = NULL;
 	tsk->task_frag.page = NULL;
+	pisa_task_frag_init(tsk);
 	tsk->wake_q.next = NULL;
 	tsk->worker_private = NULL;
 
diff --git a/net/core/sock.c b/net/core/sock.c
index ad414878dc5ba..56dea49baaec2 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -115,6 +115,7 @@
 #include <linux/memcontrol.h>
 #include <linux/prefetch.h>
 #include <linux/compat.h>
+#include <linux/pisa.h>
 #include <linux/mroute.h>
 #include <linux/mroute6.h>
 #include <linux/icmpv6.h>
@@ -147,6 +148,7 @@
 #include <net/tcp.h>
 #include <net/busy_poll.h>
 #include <net/phonet/phonet.h>
+#include <net/pisa.h>
 
 #include <linux/ethtool.h>
 
@@ -3257,13 +3259,21 @@ DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key);
  */
 bool skb_page_frag_refill(unsigned int sz, struct page_frag *pfrag, gfp_t gfp)
 {
+	/* PISA pages only go into current->task_frag_pisa, see sk_page_frag_dst(). */
+	if (is_pisa_pfrag(pfrag))
+		gfp |= __GFP_PISA;
+
 	if (pfrag->page) {
 		if (page_ref_count(pfrag->page) == 1) {
+			if (!!(gfp & __GFP_PISA) != is_pisa_page(pfrag->page) ||
+			    ((gfp & __GFP_PISA) && !pisa_page_can_recycle(pfrag->page)))
+				goto drop_page;
 			pfrag->offset = 0;
 			return true;
 		}
 		if (pfrag->offset + sz <= pfrag->size)
 			return true;
+drop_page:
 		put_page(pfrag->page);
 	}
 
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index ae701561d9f12..8ed713772e4ee 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -20,6 +20,7 @@
 #include <net/udp.h>
 #include <net/tcp.h>
 #include <net/espintcp.h>
+#include <net/pisa.h>
 #include <linux/skbuff_ref.h>
 
 #include <linux/highmem.h>
@@ -451,7 +452,8 @@ int esp_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
 
 			spin_lock_bh(&x->lock);
 
-			if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+			if (unlikely(!skb_page_frag_refill(allocsize, pfrag,
+							   GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) {
 				spin_unlock_bh(&x->lock);
 				goto cow;
 			}
@@ -560,7 +562,8 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
 		allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
 
 		spin_lock_bh(&x->lock);
-		if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+		if (unlikely(!skb_page_frag_refill(allocsize, pfrag,
+						   GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) {
 			spin_unlock_bh(&x->lock);
 			goto error_free;
 		}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index d42d776af273e..50b8c92c3cd5e 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -71,6 +71,7 @@
 #include <net/xfrm.h>
 #include <linux/skbuff.h>
 #include <net/sock.h>
+#include <net/pisa.h>
 #include <net/arp.h>
 #include <net/icmp.h>
 #include <net/checksum.h>
@@ -986,7 +987,10 @@ static int __ip_append_data(struct sock *sk,
 
 	exthdrlen = !skb ? rt->dst.header_len : 0;
 	mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize;
-	paged = !!cork->gso_size;
+	paged = !!cork->gso_size ||
+		(getfrag == ip_generic_getfrag &&
+		 (rt->dst.dev->features & NETIF_F_SG) &&
+		 is_pisa_pfrag(pfrag));
 
 	hh_len = LL_RESERVED_SPACE(rt->dst.dev);
 
@@ -1383,8 +1387,8 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4,
 	}
 
 	return __ip_append_data(sk, fl4, &sk->sk_write_queue, &inet->cork.base,
-				sk_page_frag(sk), getfrag,
-				from, length, transhdrlen, flags);
+				sk_page_frag_dst(sk, inet->cork.base.dst),
+				getfrag, from, length, transhdrlen, flags);
 }
 
 static void ip_cork_release(struct inet_cork *cork)
@@ -1592,7 +1596,7 @@ struct sk_buff *ip_make_skb(struct sock *sk,
 		return ERR_PTR(err);
 
 	err = __ip_append_data(sk, fl4, &queue, cork,
-			       &current->task_frag, getfrag,
+			       sk_page_frag_dst(sk, cork->dst), getfrag,
 			       from, length, transhdrlen, flags);
 	if (err) {
 		__ip_flush_pending_frames(sk, &queue, cork);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 832b25ddd3c2e..3c3edc48705d7 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -273,6 +273,7 @@
 #include <net/inet_common.h>
 #include <net/inet_ecn.h>
 #include <net/tcp.h>
+#include <net/pisa.h>
 #include <net/tcp_ecn.h>
 #include <net/mptcp.h>
 #include <net/proto_memory.h>
@@ -1295,7 +1296,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 		if (zc == 0) {
 			bool merge = true;
 			int i = skb_shinfo(skb)->nr_frags;
-			struct page_frag *pfrag = sk_page_frag(sk);
+			struct page_frag *pfrag = sk_page_frag_dst(sk, __sk_dst_get(sk));
 
 			if (!sk_page_frag_refill(sk, pfrag))
 				goto wait_for_space;
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index d654f0f442388..98ed9120ae7b4 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -36,6 +36,7 @@
 #include <net/tcp.h>
 #include <net/espintcp.h>
 #include <net/inet6_hashtables.h>
+#include <net/pisa.h>
 #include <linux/skbuff_ref.h>
 
 #include <linux/highmem.h>
@@ -481,7 +482,8 @@ int esp6_output_head(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
 
 			spin_lock_bh(&x->lock);
 
-			if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+			if (unlikely(!skb_page_frag_refill(allocsize, pfrag,
+							   GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) {
 				spin_unlock_bh(&x->lock);
 				goto cow;
 			}
@@ -592,7 +594,8 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
 		allocsize = ALIGN(skb->data_len, L1_CACHE_BYTES);
 
 		spin_lock_bh(&x->lock);
-		if (unlikely(!skb_page_frag_refill(allocsize, pfrag, GFP_ATOMIC))) {
+		if (unlikely(!skb_page_frag_refill(allocsize, pfrag,
+						   GFP_ATOMIC | pisa_tx_bufs_gfp(skb)))) {
 			spin_unlock_bh(&x->lock);
 			goto error_free;
 		}
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index cb648c8394310..8f6f0bed15499 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -40,6 +40,7 @@
 #include <linux/netfilter_ipv6.h>
 
 #include <net/sock.h>
+#include <net/pisa.h>
 #include <net/snmp.h>
 
 #include <net/gso.h>
@@ -1489,7 +1490,10 @@ static int __ip6_append_data(struct sock *sk,
 		dst_exthdrlen = rt->dst.header_len - rt->rt6i_nfheader_len;
 	}
 
-	paged = !!cork->gso_size;
+	paged = !!cork->gso_size ||
+		(getfrag == ip_generic_getfrag &&
+		 (rt->dst.dev->features & NETIF_F_SG) &&
+		 is_pisa_pfrag(pfrag));
 	mtu = cork->gso_size ? IP6_MAX_MTU : cork->fragsize;
 	orig_mtu = mtu;
 
@@ -1892,8 +1896,8 @@ int ip6_append_data(struct sock *sk,
 	}
 
 	return __ip6_append_data(sk, &sk->sk_write_queue, &inet->cork,
-				 sk_page_frag(sk), getfrag,
-				 from, length, transhdrlen, flags);
+				 sk_page_frag_dst(sk, inet->cork.base.dst),
+				 getfrag, from, length, transhdrlen, flags);
 }
 EXPORT_SYMBOL_GPL(ip6_append_data);
 
@@ -2095,7 +2099,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk,
 	}
 
 	err = __ip6_append_data(sk, &queue, cork,
-				&current->task_frag, getfrag, from,
+				sk_page_frag_dst(sk, cork->base.dst), getfrag, from,
 				length + exthdrlen, transhdrlen + exthdrlen,
 				flags);
 	if (err) {
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 15/25] net: Use PISA for page_pool memory
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (13 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
                     ` (11 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

When a page_pool performs DMA mapping (PP_FLAG_DMA_MAP) on a device with
pisa_rx_bufs enabled, pass __GFP_PISA_RW to use PISA memory.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 net/core/page_pool.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index d7c88c0b67e53..91cae5ba78373 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -16,6 +16,7 @@
 #include <net/page_pool/helpers.h>
 #include <net/page_pool/memory_provider.h>
 #include <net/xdp.h>
+#include <linux/pisa.h>
 
 #include <linux/dma-direction.h>
 #include <linux/dma-mapping.h>
@@ -622,6 +623,8 @@ static noinline netmem_ref __page_pool_alloc_netmems_slow(struct page_pool *pool
 	 */
 	if ((gfp & GFP_ATOMIC) == GFP_ATOMIC)
 		gfp |= __GFP_NOWARN;
+	if (dma_map)
+		gfp |= pisa_rx_gfp(pool->p.dev);
 
 	/* Don't support bulk alloc for high-order pages */
 	if (unlikely(pp_order))
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (14 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 15/25] net: Use PISA for page_pool memory Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
                     ` (10 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

skb heads are provided by a slab allocator that is fed by alloc_pages().

Add SLAB_CACHE_PISA_RO so that a slab cache can back its slabs with
zeroed, read-only PISA folios (__GFP_PISA | __GFP_ZERO) via
pisa_alloc_folio().
Slabs are still released with free_frozen_pages(), so pages from PISA
return to their allocator via __free_pages_prepare(). Caches with
SLAB_CACHE_PISA_RO skip KFENCE and are never merged.

Existing caches are unaffected.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa-mm.h | 20 ++++++++++++++
 include/linux/slab.h    | 10 +++++++
 mm/pisa-alloc.c         | 60 +++++++++++++++++++++++++++++++++++++++++
 mm/pisa-kunit.c         | 22 +++++++++++++++
 mm/pisa-priv.h          |  1 +
 mm/slab.h               |  3 ++-
 mm/slab_common.c        |  2 +-
 mm/slub.c               | 23 ++++++++++++++--
 8 files changed, 137 insertions(+), 4 deletions(-)

diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
index 8caa2afded6ff..fd5e7234e9423 100644
--- a/include/linux/pisa-mm.h
+++ b/include/linux/pisa-mm.h
@@ -14,6 +14,8 @@ struct folio_batch;
 unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
 				      int preferred_nid, int nr_pages,
 				      struct page **page_array, bool frozen);
+struct folio *__pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid);
+int pisa_slab_cache_init(unsigned int order);
 bool __pisa_free_page(struct page *page, unsigned int order);
 bool __pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter,
 			      enum dma_data_direction dir);
@@ -63,6 +65,13 @@ pisa_alloc_pages(gfp_t *gfp, unsigned int order, int preferred_nid,
 	return page;
 }
 
+/* Hook for SLAB_CACHE_PISA_RO in alloc_slab_page() (guarded by @pisa_key). */
+static inline struct folio *
+pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+	return pisa_enabled() ? __pisa_alloc_folio(gfp, order, nid) : NULL;
+}
+
 /* Hook for __free_pages_prepare() (guarded by @pisa_key). */
 static inline bool pisa_free_page(struct page *page, unsigned int order)
 {
@@ -129,6 +138,17 @@ pisa_alloc_pages(gfp_t *gfp, unsigned int order, int preferred_nid,
 	return NULL;
 }
 
+static inline struct folio *
+pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+	return NULL;
+}
+
+static inline int pisa_slab_cache_init(unsigned int order)
+{
+	return 0;
+}
+
 static inline bool pisa_free_page(struct page *page, unsigned int order)
 {
 	return false;
diff --git a/include/linux/slab.h b/include/linux/slab.h
index cda126def67af..4b6205540279d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -66,6 +66,9 @@ enum _slab_flag_bits {
 #endif
 #endif
 	_SLAB_NO_SHEAVES,
+#ifdef CONFIG_PISA
+	_SLAB_CACHE_PISA_RO,
+#endif
 	_SLAB_FLAGS_LAST_BIT
 };
 
@@ -260,6 +263,13 @@ enum _slab_flag_bits {
 #define SLAB_OBJ_EXT_IN_OBJ	__SLAB_FLAG_UNUSED
 #endif
 
+/* Back slabs with zeroed read-only PISA folios (__GFP_PISA | __GFP_ZERO) */
+#ifdef CONFIG_PISA
+#define SLAB_CACHE_PISA_RO	__SLAB_FLAG_BIT(_SLAB_CACHE_PISA_RO)
+#else
+#define SLAB_CACHE_PISA_RO	__SLAB_FLAG_UNUSED
+#endif
+
 /*
  * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
  *
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 8c4c0f1f1cf50..4fc3b4d29de67 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -42,6 +42,8 @@
 #include <linux/workqueue.h>
 #include <net/sock.h>
 
+#include "internal.h"
+#include "page_alloc.h"
 #include "pisa-priv.h"
 
 /* Idle PISA pages over the allocator watermark queued locklessly for release. */
@@ -710,6 +712,64 @@ unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
 	return pisa_alloc_bulk(pa, gfp & ~__GFP_PISA_RW, nr_pages, page_array, frozen);
 }
 
+static struct pisa_alloc *pisa_slab_pa_ensure(int nid, unsigned int order)
+{
+	return pisa_alloc_ensure(&pisa_nodes[nid].slab_pa[order], nid, order,
+				 DMA_TO_DEVICE, false, "pisa_slab_o%u_n%d", order, nid);
+}
+
+/**
+ * pisa_slab_cache_init - Create the allocators behind a SLAB_CACHE_PISA_RO cache
+ * @order: Slab page order of the cache
+ *
+ * Called at cache creation, so that the first allocations, which may be atomic
+ * or node-bound, find the allocator of every memory node.
+ *
+ * Return: 0, -EINVAL for an unsupported @order, or -ENOMEM.
+ */
+int pisa_slab_cache_init(unsigned int order)
+{
+	int nid;
+
+	if (order > PISA_MAX_ALLOC_ORDER)
+		return -EINVAL;
+	for_each_node_state(nid, N_MEMORY) {
+		if (!pisa_slab_pa_ensure(nid, order))
+			return -ENOMEM;
+	}
+	return 0;
+}
+
+/* Allocate a zeroed read-only PISA folio for SLAB_CACHE_PISA_RO. */
+struct folio *__pisa_alloc_folio(gfp_t gfp, unsigned int order, int nid)
+{
+	int target_nid = pisa_sanitize_nid(nid);
+	struct page *page = NULL;
+	struct pisa_alloc *pa;
+
+	if (order > PISA_MAX_ALLOC_ORDER)
+		return NULL;
+	/* Pairs with smp_store_release() in pisa_alloc_ensure(). */
+	pa = smp_load_acquire(&pisa_nodes[target_nid].slab_pa[order]);
+	if (unlikely(!pa)) {
+		gfp_t ctx = current_gfp_context(gfp);
+
+		/* Only for a node that got memory after pisa_slab_cache_init(). */
+		if (!gfpflags_allow_blocking(ctx) ||
+		    (ctx & (__GFP_FS | __GFP_IO)) != (__GFP_FS | __GFP_IO))
+			return NULL;
+		pa = pisa_slab_pa_ensure(target_nid, order);
+	}
+	pisa_alloc_bulk(pa, gfp & ~__GFP_THISNODE, 1, &page, true);
+	if (page && (gfp & __GFP_THISNODE) && nid != NUMA_NO_NODE &&
+	    page_to_nid(page) != nid) {
+		/* @page is frozen (refcount 0): __free_pages() would underflow it. */
+		free_frozen_pages(page, order);
+		return NULL;
+	}
+	return page ? page_folio(page) : NULL;
+}
+
 /**
  * __pisa_has_free_blocks_rx - Check if per-node RX allocator of @order on @nid has free blocks
  * @order: Block order
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index d7670c76e98c5..ec7af4313915a 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -13,6 +13,7 @@
 #include <linux/mm.h>
 #include <linux/shrinker.h>
 #include <linux/skbuff.h>
+#include <linux/slab.h>
 #include <linux/swiotlb.h>
 #include <net/sock.h>
 
@@ -716,6 +717,26 @@ static void test_reservoir_and_scrubber(struct kunit *test)
 	WRITE_ONCE(pisa_scrub_max, old_scrub);
 }
 
+static void test_slub_page_alloc(struct kunit *test)
+{
+	struct kmem_cache *s;
+	void *obj;
+
+	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
+	s = kmem_cache_create("pisa_kunit_slab", 1024, NULL,
+			      SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+	KUNIT_ASSERT_NOT_NULL(test, s);
+
+	obj = kmem_cache_alloc(s, GFP_KERNEL);
+	KUNIT_EXPECT_NOT_NULL(test, obj);
+	if (obj) {
+		KUNIT_EXPECT_NOT_NULL(test, pisa_virt_to_page(obj));
+		kmem_cache_free(s, obj);
+	}
+
+	kmem_cache_destroy(s);
+}
+
 static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_meta_init_and_roundtrip),
 	KUNIT_CASE(test_meta_invalid_phys),
@@ -731,6 +752,7 @@ static struct kunit_case pisa_meta_test_cases[] = {
 	KUNIT_CASE(test_pcp_cache),
 	KUNIT_CASE(test_gfp_pisa),
 	KUNIT_CASE(test_reservoir_and_scrubber),
+	KUNIT_CASE(test_slub_page_alloc),
 	{}
 };
 
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index c7f4e30579070..08bf1b64bcf2c 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -134,6 +134,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
 struct pisa_node {
 	unsigned long		next_alloc;	/* atomic alloc backoff jiffies */
 	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
+	struct pisa_alloc	*slab_pa[PISA_MAX_ALLOC_ORDER + 1]; /* SLAB_CACHE_PISA_RO */
 	struct pisa_alloc	*user_pa[3];	/* user/page-cache allocators by dir */
 
 	spinlock_t		lock;		/* protects @pages and @nr_pages */
diff --git a/mm/slab.h b/mm/slab.h
index e94ac63d10dc0..53620e40b216b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -473,7 +473,8 @@ void flush_rcu_sheaves_on_cache(struct kmem_cache *s);
 			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
 			 SLAB_NOLEAKTRACE | SLAB_RECLAIM_ACCOUNT | \
 			 SLAB_TEMPORARY | SLAB_ACCOUNT | \
-			 SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE)
+			 SLAB_NO_USER_FLAGS | SLAB_KMALLOC | SLAB_NO_MERGE | \
+			 SLAB_CACHE_PISA_RO)
 
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
 			  SLAB_TRACE | SLAB_CONSISTENCY_CHECKS)
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 2befd24b4d099..5dae3aa524221 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -49,7 +49,7 @@ struct kmem_cache *kmem_cache;
  */
 #define SLAB_NEVER_MERGE (SLAB_DEBUG_FLAGS | SLAB_TYPESAFE_BY_RCU | \
 		SLAB_NOLEAKTRACE | SLAB_FAILSLAB | SLAB_NO_MERGE | \
-		SLAB_OBJ_EXT_IN_OBJ)
+		SLAB_OBJ_EXT_IN_OBJ | SLAB_CACHE_PISA_RO)
 
 #define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_ACCOUNT | SLAB_MAY_ACCOUNT)
diff --git a/mm/slub.c b/mm/slub.c
index cd671b9347191..0380869fb3301 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -46,6 +46,7 @@
 #include <linux/prandom.h>
 #include <kunit/test.h>
 #include <kunit/test-bug.h>
+#include <linux/pisa-mm.h>
 #include <linux/sort.h>
 #include <linux/irq_work.h>
 #include <linux/kprobes.h>
@@ -2112,7 +2113,8 @@ static inline void dec_slabs_node(struct kmem_cache *s, int node,
  */
 #define OBJCGS_CLEAR_MASK	(__GFP_DMA | __GFP_RECLAIMABLE | \
 				__GFP_ACCOUNT | __GFP_NOFAIL | \
-				__GFP_THISNODE | __GFP_COMP)
+				__GFP_THISNODE | __GFP_COMP | \
+				__GFP_PISA_RW)
 
 #ifdef CONFIG_SLAB_OBJ_EXT
 
@@ -3378,6 +3380,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node,
 	if (unlikely(!allow_spin))
 		page = alloc_frozen_pages_nolock(0/* __GFP_COMP is implied */,
 								  node, order);
+	else if (unlikely(flags & __GFP_PISA))
+		page = (struct page *)pisa_alloc_folio(flags, order, node);
 	else if (node == NUMA_NO_NODE)
 		page = alloc_frozen_pages(flags, order);
 	else
@@ -3484,7 +3488,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
 	gfp_t alloc_gfp;
 	void *start;
 
-	flags &= gfp_allowed_mask;
+	flags &= gfp_allowed_mask & ~__GFP_PISA_RW;
 
 	flags |= s->allocflags;
 
@@ -3495,9 +3499,13 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags,
 	alloc_gfp = (flags | __GFP_NOWARN | __GFP_NORETRY) & ~__GFP_NOFAIL;
 	if ((alloc_gfp & __GFP_DIRECT_RECLAIM) && oo_order(oo) > oo_order(s->min))
 		alloc_gfp = (alloc_gfp | __GFP_NOMEMALLOC) & ~__GFP_RECLAIM;
+	if (unlikely(s->flags & SLAB_CACHE_PISA_RO))
+		alloc_gfp = (flags | __GFP_NOWARN) & ~__GFP_NOFAIL;
 
 	slab = alloc_slab_page(alloc_gfp, node, oo, allow_spin);
 	if (unlikely(!slab)) {
+		if (s->flags & SLAB_CACHE_PISA_RO)
+			return NULL;
 		oo = s->min;
 		alloc_gfp = flags;
 		/*
@@ -8182,6 +8190,11 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 	if (s->flags & SLAB_RECLAIM_ACCOUNT)
 		s->allocflags |= __GFP_RECLAIMABLE;
 
+	if (s->flags & SLAB_CACHE_PISA_RO) {
+		s->flags |= SLAB_SKIP_KFENCE;
+		s->allocflags |= __GFP_PISA | __GFP_ZERO;
+	}
+
 	/*
 	 * For kmalloc caches we enable sheaves later by
 	 * bootstrap_kmalloc_sheaves() to avoid recursion.
@@ -8914,6 +8927,12 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,
 	if (err)
 		goto out;
 
+	if (s->flags & SLAB_CACHE_PISA_RO) {
+		err = pisa_slab_cache_init(oo_order(s->oo));
+		if (err)
+			goto out;
+	}
+
 	err = 0;
 
 	/* Mutex is not taken during early boot */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (15 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY Luigi Rizzo
                     ` (9 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

The linear part of most transmitted skbs is a small head from the
skbuff_small_head cache. To avoid per-packet IOMMU map/unmap operations
or bounce-buffer copies for headers, allow backing small TX skb heads
with PISA memory.

When per-device pisa_tx_skb_heads is enabled, create a parallel
skbuff_small_head_pisa slab cache backed by zeroed, read-only PISA
folios (SLAB_CACHE_PISA_RO). Eligible TX paths passing __GFP_PISA
allocate small heads from this cache and fall back to skbuff_small_head
on failure.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/base/core.c    |  3 +-
 include/linux/device.h |  6 +++
 include/net/pisa.h     | 64 +++++++++++++++++++++++++++++
 mm/pisa-kunit.c        |  5 ++-
 net/core/pisa-net.c    | 93 +++++++++++++++++++++++++++++++++++++-----
 net/core/skbuff.c      | 30 +++++++++++---
 net/core/sock.c        | 21 ++++++++--
 net/ipv4/ip_output.c   |  8 ++--
 net/ipv4/raw.c         |  6 +--
 net/ipv4/tcp.c         |  2 +-
 net/ipv4/tcp_output.c  | 14 ++++---
 net/ipv6/ip6_output.c  |  8 ++--
 net/ipv6/raw.c         |  6 +--
 13 files changed, 224 insertions(+), 42 deletions(-)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index 5f1291e644c5e..92eec7d3d4cf1 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2951,7 +2951,7 @@ static DEVICE_ATTR_RW(name)
 PISA_BOOL_ATTR(pisa_coherent);
 PISA_BOOL_ATTR(pisa_rx_bufs);
 PISA_SET_ATTR(pisa_tx_bufs, net_pisa_tx_bufs_set);
-PISA_BOOL_ATTR(pisa_tx_skb_heads);
+PISA_SET_ATTR(pisa_tx_skb_heads, net_pisa_tx_skb_heads_set);
 PISA_BOOL_ATTR(pisa_tx_zerocopy);
 
 static struct attribute *dev_attr_pisa[] = {
@@ -3114,6 +3114,7 @@ static void device_remove_attrs(struct device *dev)
 	if (IS_ENABLED(CONFIG_PISA) && dev->dma_mask) {
 		device_remove_group(dev, &dev_attr_pisa_group);
 		net_pisa_tx_bufs_set(dev, false);
+		net_pisa_tx_skb_heads_set(dev, false);
 	}
 
 	if (dev->physical_location) {
diff --git a/include/linux/device.h b/include/linux/device.h
index 0d867d8255ba5..6776048d05ca8 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -819,6 +819,7 @@ struct device {
 
 #if defined(CONFIG_NET) && defined(CONFIG_PISA)
 int net_pisa_tx_bufs_set(struct device *dev, bool val);
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val);
 #else
 /* Without CONFIG_NET the flag still gates block-device writes (see pisa_bdev_enabled()). */
 static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
@@ -826,6 +827,11 @@ static inline int net_pisa_tx_bufs_set(struct device *dev, bool val)
 	WRITE_ONCE(dev->pisa_tx_bufs, val);
 	return 0;
 }
+
+static inline int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 #define __create_dev_flag_accessors(accessor_name, flag_name) \
diff --git a/include/net/pisa.h b/include/net/pisa.h
index 8e23d8b01e73d..7fe2e7d5aa97e 100644
--- a/include/net/pisa.h
+++ b/include/net/pisa.h
@@ -7,6 +7,11 @@
 #include <linux/tcp.h>
 #include <net/sock.h>
 
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+				       unsigned long data_len, int noblock,
+				       int *errcode, int max_page_order,
+				       gfp_t gfp);
+
 /*
  * struct net_device caches the PISA enable flags for tx_bufs and tx_heads used
  * by the leaf device, refreshed every 256 jiffies. The helpers below decide
@@ -16,6 +21,7 @@
 
 enum pisa_tx_flags {
 	PISA_TX_FLAG_BUFS	= BIT(0),
+	PISA_TX_FLAG_HEADS	= BIT(1),
 	PISA_TX_XFRM_OK		= BIT(2),
 	PISA_TX_TSTAMP_MASK	= ~0xffU,
 };
@@ -23,10 +29,13 @@ enum pisa_tx_flags {
 /*
  * Static keys guarding networking hooks:
  * - @net_pisa_tx_bufs_key: enabled while any device has pisa_tx_bufs set.
+ * - @net_pisa_tx_skb_heads_key: enabled while any device has pisa_tx_skb_heads set.
  */
 DECLARE_STATIC_KEY_FALSE(net_pisa_tx_bufs_key);
+DECLARE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
 
 u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now);
+void *__pisa_skb_head_alloc_node(gfp_t flags, int node);
 
 /*
  * May TX data routed through @dst use PISA pages? Those pages stay readable by
@@ -59,6 +68,48 @@ static inline gfp_t __pisa_tx_gfp(const struct dst_entry *dst,
  * ============================================================================
  */
 
+/* Hook for kmalloc_reserve() small skb head allocation (@net_pisa_tx_skb_heads_key). */
+static inline void *pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+	return static_branch_unlikely(&net_pisa_tx_skb_heads_key) ?
+	       __pisa_skb_head_alloc_node(flags, node) : NULL;
+}
+
+/* Hook for route-based skb head allocation (@net_pisa_tx_skb_heads_key). */
+static inline gfp_t pisa_tx_head_gfp(const struct dst_entry *dst)
+{
+	return static_branch_unlikely(&net_pisa_tx_skb_heads_key) ?
+	       __pisa_tx_gfp(dst, PISA_TX_FLAG_HEADS) : 0;
+}
+
+/* Hook for socket-based TCP skb head allocation (@net_pisa_tx_skb_heads_key). */
+static inline gfp_t pisa_tx_head_sk_gfp(const struct sock *sk)
+{
+	if (!static_branch_unlikely(&net_pisa_tx_skb_heads_key) || !sk)
+		return 0;
+	guard(rcu)();
+	return __pisa_tx_gfp(rcu_dereference(sk->sk_dst_cache), PISA_TX_FLAG_HEADS);
+}
+
+/* Hook for skb_copy*() / pskb_expand_head() (@net_pisa_tx_skb_heads_key). */
+static inline gfp_t skb_head_pisa_gfp(const struct sk_buff *skb)
+{
+	if (static_branch_unlikely(&net_pisa_tx_skb_heads_key) &&
+	    !skb->head_frag && pisa_virt_to_page(skb->head))
+		return __GFP_PISA;
+	return 0;
+}
+
+/* Hook for UDP/raw sock_alloc_send_skb() with @dst (@net_pisa_tx_skb_heads_key). */
+static inline struct sk_buff *sock_alloc_send_skb_dst(struct sock *sk,
+						      unsigned long size,
+						      int noblock, int *errcode,
+						      const struct dst_entry *dst)
+{
+	return __sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0,
+				      pisa_tx_head_gfp(dst));
+}
+
 /* Hook for route-based TX payload fragment selection (@net_pisa_tx_bufs_key). */
 static inline bool use_pisa_tx_bufs(const struct dst_entry *dst)
 {
@@ -95,6 +146,19 @@ static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
 
 #else /* !CONFIG_PISA */
 
+static inline void *pisa_skb_head_alloc_node(gfp_t flags, int node) { return NULL; }
+static inline gfp_t pisa_tx_head_gfp(const struct dst_entry *dst) { return 0; }
+static inline gfp_t pisa_tx_head_sk_gfp(const struct sock *sk) { return 0; }
+static inline gfp_t skb_head_pisa_gfp(const struct sk_buff *skb) { return 0; }
+
+static inline struct sk_buff *sock_alloc_send_skb_dst(struct sock *sk,
+						      unsigned long size,
+						      int noblock, int *errcode,
+						      const struct dst_entry *dst)
+{
+	return __sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0, 0);
+}
+
 static inline bool use_pisa_tx_bufs(const struct dst_entry *dst) { return false; }
 static inline gfp_t pisa_tx_bufs_gfp(const struct sk_buff *skb) { return 0; }
 static inline bool is_pisa_pfrag(const struct page_frag *pfrag) { return false; }
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index ec7af4313915a..0dbc3ecf44465 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -579,9 +579,10 @@ static void test_dev_pisa_attrs(struct kunit *test)
 	kunit_check_pisa_attr(test, dev, "pisa_coherent", &dev->pisa_coherent);
 	kunit_check_pisa_attr(test, dev, "pisa_rx_bufs", &dev->pisa_rx_bufs);
 	kunit_check_pisa_attr(test, dev, "pisa_tx_zerocopy", &dev->pisa_tx_zerocopy);
-	if (IS_ENABLED(CONFIG_NET))
+	if (IS_ENABLED(CONFIG_NET)) {
 		kunit_check_pisa_attr(test, dev, "pisa_tx_bufs", &dev->pisa_tx_bufs);
-	kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+		kunit_check_pisa_attr(test, dev, "pisa_tx_skb_heads", &dev->pisa_tx_skb_heads);
+	}
 	device_unregister(dev);
 }
 
diff --git a/net/core/pisa-net.c b/net/core/pisa-net.c
index 6d4909c82070c..d0a2c94419187 100644
--- a/net/core/pisa-net.c
+++ b/net/core/pisa-net.c
@@ -10,6 +10,9 @@
 #include <linux/mutex.h>
 #include <linux/netdevice.h>
 #include <linux/pisa.h>
+#include <linux/skbuff.h>
+#include <linux/slab.h>
+#include <net/hotdata.h>
 #include <net/pisa.h>
 
 /* Enabled while at least one device has pisa_tx_bufs set. */
@@ -32,15 +35,79 @@ int net_pisa_tx_bufs_set(struct device *dev, bool val)
 }
 
 /*
- * The sysfs attribute pisa_tx_bufs lives on the device that does DMA: the
- * closest ancestor of a leaf net_device that has a dma_mask (dev->dev.parent,
- * or further up behind bus glue such as virtio). A socket route (dst->dev) may
- * instead point to an upper virtual net_device (bond, VLAN, ipvlan, etc.),
- * whose leaves are its lower devices. Walking lower devices via
- * netdev_walk_all_lower_dev_rcu() on every TX allocation would be too
- * expensive, so dev->pisa_tx_flags_tstamp caches the walk result on the net_device
- * and refreshes it at most once every 256 jiffies:
+ * Enabled while at least one device has pisa_tx_skb_heads set; small TX skb
+ * heads (__GFP_PISA) are allocated from skbuff_small_head_pisa.
+ */
+DEFINE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
+EXPORT_SYMBOL(net_pisa_tx_skb_heads_key);
+
+static struct kmem_cache *pisa_skb_small_head_cache __read_mostly;
+
+/* Lazily initialize skbuff_small_head_pisa on first enable. */
+static int pisa_skb_head_init(void)
+{
+	unsigned int size = kmem_cache_size(net_hotdata.skb_small_head_cache);
+	struct kmem_cache_args args = {
+		/* As skbuff_small_head, see skb_init(). */
+		.useroffset	= 0,
+		.usersize	= SKB_WITH_OVERHEAD(size),
+		.sheaf_capacity	= 32,
+	};
+	static DEFINE_MUTEX(mutex);
+	struct kmem_cache *s;
+
+	guard(mutex)(&mutex);
+	if (pisa_skb_small_head_cache)
+		return 0;
+
+	/* No slab is allocated before the cache is published below. */
+	s = kmem_cache_create("skbuff_small_head_pisa", size, &args,
+			      SLAB_HWCACHE_ALIGN | SLAB_NO_MERGE | SLAB_CACHE_PISA_RO);
+	if (!s)
+		return -ENOMEM;
+	WRITE_ONCE(pisa_skb_small_head_cache, s);
+	return 0;
+}
+
+int net_pisa_tx_skb_heads_set(struct device *dev, bool val)
+{
+	static DEFINE_MUTEX(lock);
+
+	if (val) {
+		int ret = pisa_skb_head_init();
+
+		if (ret)
+			return ret;
+	}
+
+	guard(mutex)(&lock);
+	if (dev->pisa_tx_skb_heads != val) {
+		WRITE_ONCE(dev->pisa_tx_skb_heads, val);
+		if (val)
+			static_branch_inc(&net_pisa_tx_skb_heads_key);
+		else
+			static_branch_dec(&net_pisa_tx_skb_heads_key);
+	}
+	return 0;
+}
+
+void *__pisa_skb_head_alloc_node(gfp_t flags, int node)
+{
+	return kmem_cache_alloc_node(pisa_skb_small_head_cache,
+				     flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node);
+}
+
+/*
+ * The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the
+ * device that does DMA: the closest ancestor of a leaf net_device that has a
+ * dma_mask (dev->dev.parent, or further up behind bus glue such as virtio).
+ * A socket route (dst->dev) may instead point to an upper virtual net_device
+ * (bond, VLAN, ipvlan, etc.), whose leaves are its lower devices.
+ * Walking lower devices via netdev_walk_all_lower_dev_rcu() on every TX
+ * allocation would be too expensive, so dev->pisa_tx_flags_tstamp caches the walk
+ * result on the net_device and refreshes it at most once every 256 jiffies:
  *   bit 0 (PISA_TX_FLAG_BUFS):       pisa_tx_bufs enabled on lower device(s)
+ *   bit 1 (PISA_TX_FLAG_HEADS):      pisa_tx_skb_heads enabled on lower device(s)
  *   bits 31..8 (PISA_TX_TSTAMP_MASK):  upper 24 bits of (u32)jiffies
  */
 static u32 pisa_tx_flags(const struct net_device *dev)
@@ -50,8 +117,12 @@ static u32 pisa_tx_flags(const struct net_device *dev)
 
 	while (parent && !parent->dma_mask)
 		parent = parent->parent;
-	if (parent && READ_ONCE(parent->pisa_tx_bufs))
-		flags |= PISA_TX_FLAG_BUFS;
+	if (parent) {
+		if (READ_ONCE(parent->pisa_tx_bufs))
+			flags |= PISA_TX_FLAG_BUFS;
+		if (READ_ONCE(parent->pisa_tx_skb_heads))
+			flags |= PISA_TX_FLAG_HEADS;
+	}
 	return flags;
 }
 
@@ -89,7 +160,7 @@ u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now)
 		now |= pisa_tx_flags(dev);
 	} else {
 		struct netdev_nested_priv priv = {
-			.data = (void *)(uintptr_t)PISA_TX_FLAG_BUFS,
+			.data = (void *)(uintptr_t)(PISA_TX_FLAG_BUFS | PISA_TX_FLAG_HEADS),
 		};
 
 		rcu_read_lock();
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 1ac1e327ac4c6..4f487e0450d4b 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -70,6 +70,7 @@
 #include <net/protocol.h>
 #include <net/dst.h>
 #include <net/sock.h>
+#include <net/pisa.h>
 #include <net/checksum.h>
 #include <net/gro.h>
 #include <net/gso.h>
@@ -93,6 +94,7 @@
 #include <linux/user_namespace.h>
 #include <linux/indirect_call_wrapper.h>
 #include <linux/textsearch.h>
+#include <linux/pisa.h>
 
 #include "dev.h"
 #include "devmem.h"
@@ -607,16 +609,23 @@ static void *kmalloc_pfmemalloc(size_t obj_size, gfp_t flags, int node)
 static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node,
 			     struct sk_buff *skb)
 {
+	bool pisa = flags & __GFP_PISA;
 	size_t obj_size;
 	void *obj;
 
+	flags &= ~__GFP_PISA_RW;
 	obj_size = SKB_HEAD_ALIGN(*size);
 	if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE &&
 	    !(flags & KMALLOC_NOT_NORMAL_BITS)) {
+		*size = SKB_SMALL_HEAD_CACHE_SIZE;
+		if (unlikely(pisa)) {
+			obj = pisa_skb_head_alloc_node(flags, node);
+			if (likely(obj))
+				goto out;
+		}
 		obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache,
 				flags | __GFP_NOMEMALLOC | __GFP_NOWARN,
 				node);
-		*size = SKB_SMALL_HEAD_CACHE_SIZE;
 		if (likely(obj))
 			goto out;
 		/* Try again but now we are using pfmemalloc reserves */
@@ -2194,6 +2203,7 @@ struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
 
 	headerlen = skb_headroom(skb);
 	size = skb_end_offset(skb) + skb->data_len;
+	gfp_mask |= skb_head_pisa_gfp(skb);
 	n = __alloc_skb(size, gfp_mask,
 			skb_alloc_rx_flag(skb), NUMA_NO_NODE);
 	if (!n)
@@ -2238,7 +2248,10 @@ struct sk_buff *__pskb_copy_fclone(struct sk_buff *skb, int headroom,
 {
 	unsigned int size = skb_headlen(skb) + headroom;
 	int flags = skb_alloc_rx_flag(skb) | (fclone ? SKB_ALLOC_FCLONE : 0);
-	struct sk_buff *n = __alloc_skb(size, gfp_mask, flags, NUMA_NO_NODE);
+	struct sk_buff *n;
+
+	/* For the head only: @gfp_mask is also used to orphan frags below. */
+	n = __alloc_skb(size, gfp_mask | skb_head_pisa_gfp(skb), flags, NUMA_NO_NODE);
 
 	if (!n)
 		goto out;
@@ -2327,7 +2340,8 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
 	if (skb_pfmemalloc(skb))
 		gfp_mask |= __GFP_MEMALLOC;
 
-	data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+	/* For the head only: @gfp_mask is also used to orphan frags below. */
+	data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb), NUMA_NO_NODE, NULL);
 	if (!data)
 		goto nodata;
 	size = SKB_WITH_OVERHEAD(size);
@@ -2535,6 +2549,7 @@ struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
 		return NULL;
 
 	oldheadroom = skb_headroom(skb);
+	gfp_mask |= skb_head_pisa_gfp(skb);
 	n = __alloc_skb(newheadroom + skb->len + newtailroom,
 			gfp_mask, skb_alloc_rx_flag(skb),
 			NUMA_NO_NODE);
@@ -5037,7 +5052,8 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
 				hsize = len;
 
 			nskb = __alloc_skb(hsize + doffset + headroom,
-					   GFP_ATOMIC, skb_alloc_rx_flag(head_skb),
+					   GFP_ATOMIC | skb_head_pisa_gfp(head_skb),
+					   skb_alloc_rx_flag(head_skb),
 					   NUMA_NO_NODE);
 
 			if (unlikely(!nskb))
@@ -7012,7 +7028,8 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
 	if (skb_pfmemalloc(skb))
 		gfp_mask |= __GFP_MEMALLOC;
 
-	data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+	/* For the head only: @gfp_mask is also used to orphan frags below. */
+	data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb), NUMA_NO_NODE, NULL);
 	if (!data)
 		return -ENOMEM;
 	size = SKB_WITH_OVERHEAD(size);
@@ -7134,7 +7151,8 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off,
 	if (skb_pfmemalloc(skb))
 		gfp_mask |= __GFP_MEMALLOC;
 
-	data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL);
+	/* For the head only: @gfp_mask is also used to orphan frags below. */
+	data = kmalloc_reserve(&size, gfp_mask | skb_head_pisa_gfp(skb), NUMA_NO_NODE, NULL);
 	if (!data)
 		return -ENOMEM;
 	size = SKB_WITH_OVERHEAD(size);
diff --git a/net/core/sock.c b/net/core/sock.c
index 56dea49baaec2..0599c7d1aa6d0 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -3062,9 +3062,10 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
  *	Generic send/receive buffer handlers
  */
 
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
-				     unsigned long data_len, int noblock,
-				     int *errcode, int max_page_order)
+struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+				       unsigned long data_len, int noblock,
+				       int *errcode, int max_page_order,
+				       gfp_t gfp)
 {
 	struct sk_buff *skb;
 	long timeo;
@@ -3093,7 +3094,8 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 		timeo = sock_wait_for_wmem(sk, timeo);
 	}
 	skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
-				   errcode, sk->sk_allocation);
+				   errcode,
+				   sk->sk_allocation | gfp);
 	if (skb)
 		skb_set_owner_w(skb, sk);
 	return skb;
@@ -3104,6 +3106,17 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 	*errcode = err;
 	return NULL;
 }
+EXPORT_SYMBOL(__sock_alloc_send_pskb);
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+				     unsigned long data_len, int noblock,
+				     int *errcode, int max_page_order)
+{
+	/* Callers with data_len != 0 (unix, packet, tun/tap) have no sk_dst_cache, so gfp is 0. */
+	return __sock_alloc_send_pskb(sk, header_len, data_len, noblock,
+				      errcode, max_page_order,
+				      pisa_tx_head_sk_gfp(sk));
+}
 EXPORT_SYMBOL(sock_alloc_send_pskb);
 
 int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index 50b8c92c3cd5e..f710cf959124f 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -1135,14 +1135,16 @@ static int __ip_append_data(struct sock *sk,
 			alloclen += alloc_extra;
 
 			if (transhdrlen) {
-				skb = sock_alloc_send_skb(sk, alloclen,
-						(flags & MSG_DONTWAIT), &err);
+				skb = sock_alloc_send_skb_dst(sk, alloclen,
+							      flags & MSG_DONTWAIT,
+							      &err, &rt->dst);
 			} else {
 				skb = NULL;
 				if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
 				    2 * sk->sk_sndbuf)
 					skb = alloc_skb(alloclen,
-							sk->sk_allocation);
+							sk->sk_allocation |
+							pisa_tx_head_gfp(&rt->dst));
 				if (unlikely(!skb))
 					err = -ENOBUFS;
 			}
diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c
index f33a9dfc53099..87622fc1e2849 100644
--- a/net/ipv4/raw.c
+++ b/net/ipv4/raw.c
@@ -60,6 +60,7 @@
 #include <linux/ip.h>
 #include <linux/net.h>
 #include <net/ip.h>
+#include <net/pisa.h>
 #include <net/icmp.h>
 #include <net/udp.h>
 #include <net/raw.h>
@@ -358,9 +359,8 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4,
 
 	hlen = LL_RESERVED_SPACE(rt->dst.dev);
 	tlen = rt->dst.dev->needed_tailroom;
-	skb = sock_alloc_send_skb(sk,
-				  length + hlen + tlen + 15,
-				  flags & MSG_DONTWAIT, &err);
+	skb = sock_alloc_send_skb_dst(sk, length + hlen + tlen + 15,
+				      flags & MSG_DONTWAIT, &err, &rt->dst);
 	if (!skb)
 		goto error;
 	skb_reserve(skb, hlen);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 3c3edc48705d7..83f8ed2aa9a7e 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -932,7 +932,7 @@ struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp,
 {
 	struct sk_buff *skb;
 
-	skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp);
+	skb = alloc_skb_fclone(MAX_TCP_HEADER, gfp | pisa_tx_head_sk_gfp(sk));
 	if (likely(skb)) {
 		bool mem_scheduled;
 
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index cf3ddc055d48d..24629788295b6 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -43,6 +43,7 @@
 #include <net/smc.h>
 #include <net/proto_memory.h>
 #include <net/psp.h>
+#include <net/pisa.h>
 
 #include <linux/compiler.h>
 #include <linux/gfp.h>
@@ -3893,7 +3894,8 @@ void tcp_send_fin(struct sock *sk)
 	} else {
 		skb = alloc_skb_fclone(MAX_TCP_HEADER,
 				       sk_gfp_mask(sk, GFP_ATOMIC |
-						       __GFP_NOWARN));
+						       __GFP_NOWARN) |
+				       pisa_tx_head_sk_gfp(sk));
 		if (unlikely(!skb))
 			return;
 
@@ -3921,7 +3923,7 @@ void tcp_send_active_reset(struct sock *sk, enum sk_rst_reason reason)
 	TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS);
 
 	/* NOTE: No TCP options attached and we never retransmit this. */
-	skb = alloc_skb(MAX_TCP_HEADER, priority);
+	skb = alloc_skb(MAX_TCP_HEADER, priority | pisa_tx_head_sk_gfp(sk));
 	if (!skb) {
 		NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED);
 		return;
@@ -4011,7 +4013,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
 	int mss;
 	u64 now;
 
-	skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC);
+	skb = alloc_skb(MAX_TCP_HEADER, GFP_ATOMIC | pisa_tx_head_gfp(dst));
 	if (unlikely(!skb)) {
 		dst_release(dst);
 		return NULL;
@@ -4548,7 +4550,8 @@ void __tcp_send_ack(struct sock *sk, u32 rcv_nxt, u16 flags)
 	 * sock.
 	 */
 	buff = alloc_skb(MAX_TCP_HEADER,
-			 sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+			 sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) |
+			 pisa_tx_head_sk_gfp(sk));
 	if (unlikely(!buff)) {
 		struct inet_connection_sock *icsk = inet_csk(sk);
 		unsigned long delay;
@@ -4601,7 +4604,8 @@ static int tcp_xmit_probe_skb(struct sock *sk, int urgent, int mib)
 
 	/* We don't queue it, tcp_transmit_skb() sets ownership. */
 	skb = alloc_skb(MAX_TCP_HEADER,
-			sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN));
+			sk_gfp_mask(sk, GFP_ATOMIC | __GFP_NOWARN) |
+			pisa_tx_head_sk_gfp(sk));
 	if (!skb)
 		return -1;
 
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 8f6f0bed15499..9a853655f8558 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1699,14 +1699,16 @@ static int __ip6_append_data(struct sock *sk,
 				goto error;
 			}
 			if (transhdrlen) {
-				skb = sock_alloc_send_skb(sk, alloclen,
-						(flags & MSG_DONTWAIT), &err);
+				skb = sock_alloc_send_skb_dst(sk, alloclen,
+							      flags & MSG_DONTWAIT,
+							      &err, &rt->dst);
 			} else {
 				skb = NULL;
 				if (refcount_read(&sk->sk_wmem_alloc) + wmem_alloc_delta <=
 				    2 * sk->sk_sndbuf)
 					skb = alloc_skb(alloclen,
-							sk->sk_allocation);
+							sk->sk_allocation |
+							pisa_tx_head_gfp(&rt->dst));
 				if (unlikely(!skb))
 					err = -ENOBUFS;
 			}
diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c
index 00d832c7cdae9..aa10e74d0db3f 100644
--- a/net/ipv6/raw.c
+++ b/net/ipv6/raw.c
@@ -33,6 +33,7 @@
 
 #include <net/net_namespace.h>
 #include <net/ip.h>
+#include <net/pisa.h>
 #include <net/sock.h>
 #include <net/snmp.h>
 
@@ -614,9 +615,8 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length,
 	if (flags&MSG_PROBE)
 		goto out;
 
-	skb = sock_alloc_send_skb(sk,
-				  length + hlen + tlen + 15,
-				  flags & MSG_DONTWAIT, &err);
+	skb = sock_alloc_send_skb_dst(sk, length + hlen + tlen + 15,
+				      flags & MSG_DONTWAIT, &err, &rt->dst);
 	if (!skb)
 		goto error;
 	skb_reserve(skb, hlen);
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (16 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages Luigi Rizzo
                     ` (8 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Perform best-effort migration of user pages into PISA memory on
MSG_ZEROCOPY sends in zerocopy_fill_skb_from_iter() when the egress
device has the per-device attribute pisa_tx_zerocopy enabled. For TCP
over a device that also has pisa_tx_bufs enabled, pages that cannot be
migrated fall back to the regular copy path into PISA TX page_frags
without reporting SO_EE_CODE_ZEROCOPY_COPIED.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/linux/pisa.h |  3 ++
 include/net/pisa.h   | 32 +++++++++++++--
 mm/pisa-user.c       |  8 ++++
 net/core/datagram.c  | 17 ++++++--
 net/core/pisa-net.c  | 95 ++++++++++++++++++++++++++++++++++++++++++--
 net/core/skbuff.c    |  3 +-
 net/ipv4/tcp.c       | 12 +++++-
 7 files changed, 158 insertions(+), 12 deletions(-)

diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 84692852299f1..11e3ea9eb263d 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -95,6 +95,7 @@ bool __pisa_page_can_recycle(const struct page *page);
 bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			     size_t size, size_t offset,
 			     enum dma_data_direction dir, bool pinned);
+bool pisa_page_tx_ok(const struct page *page);
 
 static inline bool pisa_enabled(void)
 {
@@ -264,6 +265,8 @@ static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
 	return false;
 }
 
+static inline bool pisa_page_tx_ok(const struct page *page) { return false; }
+
 #endif /* CONFIG_PISA */
 
 #if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
diff --git a/include/net/pisa.h b/include/net/pisa.h
index 7fe2e7d5aa97e..c95e4ab1f344f 100644
--- a/include/net/pisa.h
+++ b/include/net/pisa.h
@@ -13,9 +13,9 @@ struct sk_buff *__sock_alloc_send_pskb(struct sock *sk, unsigned long header_len
 				       gfp_t gfp);
 
 /*
- * struct net_device caches the PISA enable flags for tx_bufs and tx_heads used
- * by the leaf device, refreshed every 256 jiffies. The helpers below decide
- * whether to use PISA memory for a given dst, sk, skb.
+ * struct net_device caches the PISA enable flags for tx_bufs, tx_heads and
+ * tx_zerocopy used by the leaf device, refreshed every 256 jiffies. The helpers
+ * below decide whether to use PISA memory for a given dst, sk, skb.
  */
 #ifdef CONFIG_PISA
 
@@ -23,6 +23,7 @@ enum pisa_tx_flags {
 	PISA_TX_FLAG_BUFS	= BIT(0),
 	PISA_TX_FLAG_HEADS	= BIT(1),
 	PISA_TX_XFRM_OK		= BIT(2),
+	PISA_TX_FLAG_ZC		= BIT(3),
 	PISA_TX_TSTAMP_MASK	= ~0xffU,
 };
 
@@ -36,6 +37,10 @@ DECLARE_STATIC_KEY_FALSE(net_pisa_tx_skb_heads_key);
 
 u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now);
 void *__pisa_skb_head_alloc_node(gfp_t flags, int node);
+bool __pisa_zc_iter_stop(struct sk_buff *skb, struct iov_iter *from,
+			 size_t *length, const struct sock *sk,
+			 struct page **pages, int max_pages,
+			 unsigned int orig_len, size_t *start, ssize_t *copied);
 
 /*
  * May TX data routed through @dst use PISA pages? Those pages stay readable by
@@ -144,6 +149,18 @@ static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
 	       sk->sk_use_task_frag ? task_page_frag(dst) : sk_page_frag(sk);
 }
 
+/* Hook for __zerocopy_fill_skb_from_iter() MSG_ZEROCOPY upgrade (@pisa_user_key). */
+static inline bool pisa_zc_iter_stop(struct sk_buff *skb, struct iov_iter *from,
+				     size_t *length, const struct sock *sk,
+				     struct page **pages, int max_pages,
+				     unsigned int orig_len, size_t *start,
+				     ssize_t *copied)
+{
+	return pisa_user_enabled() &&
+	       __pisa_zc_iter_stop(skb, from, length, sk, pages, max_pages,
+				   orig_len, start, copied);
+}
+
 #else /* !CONFIG_PISA */
 
 static inline void *pisa_skb_head_alloc_node(gfp_t flags, int node) { return NULL; }
@@ -174,5 +191,14 @@ static inline struct page_frag *sk_page_frag_dst(struct sock *sk,
 	return sk_page_frag(sk);
 }
 
+static inline bool pisa_zc_iter_stop(struct sk_buff *skb, struct iov_iter *from,
+				     size_t *length, const struct sock *sk,
+				     struct page **pages, int max_pages,
+				     unsigned int orig_len, size_t *start,
+				     ssize_t *copied)
+{
+	return false;
+}
+
 #endif /* CONFIG_PISA */
 #endif /* _NET_PISA_H */
diff --git a/mm/pisa-user.c b/mm/pisa-user.c
index 5fb1ec7c45204..e4a4e1a606b24 100644
--- a/mm/pisa-user.c
+++ b/mm/pisa-user.c
@@ -439,6 +439,14 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 	return true;
 }
 
+/* Is @page, referenced by the caller, PISA memory that devices can read (usable for TX)? */
+bool pisa_page_tx_ok(const struct page *page)
+{
+	unsigned long pfn = page_to_pfn(page);
+
+	return is_pisa_pfn(pfn) && (READ_ONCE(pisa_meta_of_pfn(pfn)->prot) & IOMMU_READ);
+}
+
 /* Migrate the single-page segments already added to @bio; the caller then extracts them again. */
 bool __pisa_bio_migrate_pages(struct bio *bio, struct iov_iter *iter,
 			      enum dma_data_direction dir)
diff --git a/net/core/datagram.c b/net/core/datagram.c
index ecc30a6f83023..c589a543c3166 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -62,6 +62,7 @@
 #include <net/tcp_states.h>
 #include <trace/events/skb.h>
 #include <net/busy_poll.h>
+#include <net/pisa.h>
 
 #include "devmem.h"
 
@@ -632,10 +633,11 @@ int skb_copy_datagram_from_iter_full(struct sk_buff *skb, int offset,
 }
 EXPORT_SYMBOL(skb_copy_datagram_from_iter_full);
 
-int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
-				struct iov_iter *from, size_t length)
+static int __zerocopy_fill_skb_from_iter(struct sk_buff *skb, struct iov_iter *from,
+					 size_t length, const struct sock *sk)
 {
 	int frag = skb_shinfo(skb)->nr_frags;
+	unsigned int orig_len = skb->len;
 
 	if (!skb_frags_readable(skb))
 		return -EFAULT;
@@ -654,6 +656,9 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
 					    MAX_SKB_FRAGS - frag, &start);
 		if (copied < 0)
 			return -EFAULT;
+		if (pisa_zc_iter_stop(skb, from, &length, sk, pages,
+				      MAX_SKB_FRAGS - frag, orig_len, &start, &copied))
+			return copied;
 
 		length -= copied;
 
@@ -703,6 +708,12 @@ int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
 	return 0;
 }
 
+int zerocopy_fill_skb_from_iter(struct sk_buff *skb,
+				struct iov_iter *from, size_t length)
+{
+	return __zerocopy_fill_skb_from_iter(skb, from, length, NULL);
+}
+
 static int
 zerocopy_fill_skb_from_devmem(struct sk_buff *skb, struct iov_iter *from,
 			      int length,
@@ -760,7 +771,7 @@ int __zerocopy_sg_from_iter(struct msghdr *msg, struct sock *sk,
 	else if (binding)
 		ret = zerocopy_fill_skb_from_devmem(skb, from, length, binding);
 	else
-		ret = zerocopy_fill_skb_from_iter(skb, from, length);
+		ret = __zerocopy_fill_skb_from_iter(skb, from, length, sk);
 
 	truesize = skb->truesize - orig_size;
 	if (sk && sk->sk_type == SOCK_STREAM) {
diff --git a/net/core/pisa-net.c b/net/core/pisa-net.c
index d0a2c94419187..6422c472029a1 100644
--- a/net/core/pisa-net.c
+++ b/net/core/pisa-net.c
@@ -12,6 +12,8 @@
 #include <linux/pisa.h>
 #include <linux/skbuff.h>
 #include <linux/slab.h>
+#include <linux/mm.h>
+#include <linux/uio.h>
 #include <net/hotdata.h>
 #include <net/pisa.h>
 
@@ -98,9 +100,10 @@ void *__pisa_skb_head_alloc_node(gfp_t flags, int node)
 }
 
 /*
- * The sysfs attributes pisa_tx_bufs and pisa_tx_skb_heads live on the
- * device that does DMA: the closest ancestor of a leaf net_device that has a
- * dma_mask (dev->dev.parent, or further up behind bus glue such as virtio).
+ * The sysfs attributes pisa_tx_bufs, pisa_tx_skb_heads and pisa_tx_zerocopy
+ * live on the device that does DMA: the closest ancestor of a leaf net_device
+ * that has a dma_mask (dev->dev.parent, or further up behind bus glue such as
+ * virtio).
  * A socket route (dst->dev) may instead point to an upper virtual net_device
  * (bond, VLAN, ipvlan, etc.), whose leaves are its lower devices.
  * Walking lower devices via netdev_walk_all_lower_dev_rcu() on every TX
@@ -108,6 +111,7 @@ void *__pisa_skb_head_alloc_node(gfp_t flags, int node)
  * result on the net_device and refreshes it at most once every 256 jiffies:
  *   bit 0 (PISA_TX_FLAG_BUFS):       pisa_tx_bufs enabled on lower device(s)
  *   bit 1 (PISA_TX_FLAG_HEADS):      pisa_tx_skb_heads enabled on lower device(s)
+ *   bit 3 (PISA_TX_FLAG_ZC):         pisa_tx_zerocopy enabled on lower device(s)
  *   bits 31..8 (PISA_TX_TSTAMP_MASK):  upper 24 bits of (u32)jiffies
  */
 static u32 pisa_tx_flags(const struct net_device *dev)
@@ -122,6 +126,8 @@ static u32 pisa_tx_flags(const struct net_device *dev)
 			flags |= PISA_TX_FLAG_BUFS;
 		if (READ_ONCE(parent->pisa_tx_skb_heads))
 			flags |= PISA_TX_FLAG_HEADS;
+		if (READ_ONCE(parent->pisa_tx_zerocopy))
+			flags |= PISA_TX_FLAG_ZC;
 	}
 	return flags;
 }
@@ -160,7 +166,8 @@ u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now)
 		now |= pisa_tx_flags(dev);
 	} else {
 		struct netdev_nested_priv priv = {
-			.data = (void *)(uintptr_t)(PISA_TX_FLAG_BUFS | PISA_TX_FLAG_HEADS),
+			.data = (void *)(uintptr_t)(PISA_TX_FLAG_BUFS | PISA_TX_FLAG_HEADS |
+						    PISA_TX_FLAG_ZC),
 		};
 
 		rcu_read_lock();
@@ -173,3 +180,83 @@ u32 pisa_tx_flags_refresh(struct net_device *dev, u32 now)
 	return now;
 }
 EXPORT_SYMBOL(pisa_tx_flags_refresh);
+
+/*
+ * Upgrade user @pages to PISA memory and, for TCP over a pisa_tx_bufs device,
+ * trim non-PISA trailing pages from the batch.
+ *
+ * MSG_ZEROCOPY sends of @sk may migrate user pages into PISA memory when the
+ * egress device has pisa_tx_zerocopy set. *@tx_bufs tells whether it also has
+ * pisa_tx_bufs set, i.e. whether a copy would land in PISA memory.
+ */
+static bool pisa_zc_enabled(const struct sock *sk, bool *tx_bufs)
+{
+	const struct dst_entry *dst;
+
+	if (!sk || !sk->sk_use_task_frag)
+		return false;
+	guard(rcu)();
+	dst = rcu_dereference(sk->sk_dst_cache);
+	if (!__pisa_tx_gfp(dst, PISA_TX_FLAG_ZC))
+		return false;
+	*tx_bufs = use_pisa_tx_bufs(dst);
+	return true;
+}
+
+bool __pisa_zc_iter_stop(struct sk_buff *skb, struct iov_iter *from,
+			 size_t *length, const struct sock *sk,
+			 struct page **pages, int max_pages,
+			 unsigned int orig_len, size_t *start, ssize_t *copied)
+{
+	unsigned int i, nr;
+	bool tx_bufs;
+	ssize_t keep;
+
+	if (!user_backed_iter(from) || !pisa_zc_enabled(sk, &tx_bufs))
+		return false;
+
+	/* Move user pages into PISA memory, then look them up again. */
+	if (pisa_upgrade_user_pages(from, pages, *copied, *start,
+				    DMA_TO_DEVICE, false)) {
+		*copied = iov_iter_get_pages2(from, pages, *length, max_pages, start);
+		if (*copied <= 0) {
+			if (*copied < 0)
+				*copied = -EFAULT;
+			return true;
+		}
+	}
+
+	/*
+	 * Only TCP plain MSG_ZEROCOPY ubufs can fall back to copy, and the copy
+	 * only helps if it goes into PISA memory.
+	 */
+	if (!tx_bufs || skb->sk ||
+	    (skb_zcopy(skb) && skb_zcopy(skb)->ops != &msg_zerocopy_ubuf_ops))
+		return false;
+
+	/*
+	 * Trim @from to the leading pages that are in PISA memory devices can
+	 * read (an RX-only page would not be mapped for TX): drop the
+	 * references to the others, give their bytes back to @from, and cap
+	 * *@length so that the caller ends after attaching what is left.
+	 */
+	nr = DIV_ROUND_UP(*start + *copied, PAGE_SIZE);
+	for (i = 0; i < nr && pisa_page_tx_ok(pages[i]); i++)
+		;
+	if (i == nr)
+		return false;
+
+	keep = i ? i * PAGE_SIZE - *start : 0;
+	while (nr > i)
+		put_page(pages[--nr]);
+	iov_iter_revert(from, *copied - keep);
+	*length = keep;
+	if (keep) {
+		*copied = keep;
+		return false;
+	}
+
+	/* -EXDEV tells tcp_sendmsg_locked() to copy this chunk instead of zerocopy. */
+	*copied = (skb->len != orig_len) ? 0 : -EXDEV;
+	return true;
+}
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 4f487e0450d4b..be6b6f42ee77e 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -1942,7 +1942,8 @@ int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
 
 	err = __zerocopy_sg_from_iter(msg, sk, skb, &msg->msg_iter, len,
 				      binding);
-	if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {
+	if (err == -EFAULT ||
+	    ((err == -EMSGSIZE || err == -EXDEV) && skb->len == orig_len)) {
 		u32 delta_truesize = skb->truesize - orig_truesize;
 		struct sock *save_sk = skb->sk;
 
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 83f8ed2aa9a7e..411714edb90bd 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1132,6 +1132,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 	int process_backlog = 0;
 	int sockc_err = 0;
 	int zc = 0;
+	bool must_copy = false;	/* copy the next chunk of a MSG_ZEROCOPY send */
 	long timeo;
 
 	flags = msg->msg_flags;
@@ -1293,7 +1294,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 		if (copy > msg_data_left(msg))
 			copy = msg_data_left(msg);
 
-		if (zc == 0) {
+		if (zc == 0 || unlikely(must_copy)) {
 			bool merge = true;
 			int i = skb_shinfo(skb)->nr_frags;
 			struct page_frag *pfrag = sk_page_frag_dst(sk, __sk_dst_get(sk));
@@ -1338,6 +1339,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 				page_ref_inc(pfrag->page);
 			}
 			pfrag->offset += copy;
+			must_copy = false;
 		} else if (zc == MSG_ZEROCOPY)  {
 			/* First append to a fragless skb builds initial
 			 * pure zerocopy skb
@@ -1357,6 +1359,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size)
 				tcp_mark_push(tp, skb);
 				goto new_segment;
 			}
+			/*
+			 * The next user page is not in PISA memory: copy this
+			 * chunk, without reporting SO_EE_CODE_ZEROCOPY_COPIED.
+			 */
+			if (unlikely(err == -EXDEV)) {
+				must_copy = true;
+				continue;
+			}
 			if (err < 0)
 				goto do_error;
 			copy = err;
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (17 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms Luigi Rizzo
                     ` (7 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

In Confidential Computing (CoCo) guests (AMD SEV-SNP, Intel TDX, ARM CCA),
guest memory is private/encrypted by default, and DMA buffers must be
converted to shared/decrypted in the host page tables before a device
can access them.

Doing that hypercall per-packet on 4KB pages is prohibitively expensive,
so guests normally bounce DMA through swiotlb. With PISA, 2MB pages are
decrypted once upfront when allocated and re-encrypted only when retired
back to the buddy allocator, avoiding both per-packet hypercalls and
bounce-buffer copies.

Add the pisa_decrypt_key static key (enabled at boot in CoCo guests,
which report CC_ATTR_GUEST_MEM_ENCRYPT) and the PISA_DECRYPTED flag to
manage 2MB page decryption/re-encryption and bypass swiotlb bouncing for
decrypted PISA pages.

PISA refuses to initialize when memory is encrypted but the platform is
neither such a guest nor a bare-metal SME host, rather than hand a device
memory that it cannot access. That covers arm64 Realms until the next
commit.

Pages are decrypted in sleepable context, so atomic callers are only
served from the per-node reservoirs: in a CoCo guest pisa.reservoir_max
cannot be set to 0, and the reservoirs are filled when PISA is first
enabled.

User PTEs that map a decrypted PISA page are built with
pisa_pgprot_decrypted(), which applies pgprot_decrypted().

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/iommu/dma-iommu.c  |   4 +
 drivers/iommu/pisa-map.c   |   8 +-
 include/linux/pisa-iommu.h |   6 ++
 include/linux/pisa-mm.h    |  20 +++++
 include/linux/pisa.h       |  21 +++++
 kernel/dma/direct.c        |   4 +
 mm/ksm.c                   |   3 +-
 mm/memory.c                |   8 +-
 mm/migrate.c               |   3 +-
 mm/pisa-alloc.c            |   9 ++-
 mm/pisa-kunit.c            |  13 ++-
 mm/pisa-meta.c             |  17 ++++
 mm/pisa-node.c             | 159 ++++++++++++++++++++++++++++++++-----
 mm/pisa-priv.h             |   8 +-
 mm/swapfile.c              |   3 +-
 mm/userfaultfd.c           |   7 +-
 16 files changed, 260 insertions(+), 33 deletions(-)

diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 45e9432d2a020..5e4d9803645b4 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -37,6 +37,7 @@
 
 #include "dma-iommu.h"
 #include <linux/pisa.h>
+#include <linux/pisa-mm.h>
 #include "iommu-pages.h"
 #include "iommu-priv.h"
 
@@ -1780,6 +1781,9 @@ int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma,
 		pfn = vmalloc_to_pfn(cpu_addr);
 	} else {
 		pfn = page_to_pfn(virt_to_page(cpu_addr));
+		/* A decrypted PISA page (CoCo guest) needs a decrypted user mapping too. */
+		vma->vm_page_prot = pisa_pgprot_decrypted(virt_to_page(cpu_addr),
+							  vma->vm_page_prot);
 	}
 
 	return remap_pfn_range(vma, vma->vm_start, pfn + off,
diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index bbf330877f824..08ac91fafcf20 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -7,6 +7,7 @@
 #include <linux/atomic.h>
 #include <linux/bitops.h>
 #include <linux/cache.h>
+#include <linux/dma-direct.h>
 #include <linux/dma-map-ops.h>
 #include <linux/dma-mapping.h>
 #include <linux/iommu.h>
@@ -354,7 +355,10 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 
 	/* Every mapping of the page carries IOMMU_CACHE | @meta->prot. */
 	block_prot = IOMMU_CACHE | meta->prot;
-	if (unlikely(prot & ~block_prot))
+	/* In a CoCo guest @dev can only reach decrypted pages; an SME host needs no check. */
+	if (unlikely((prot & ~block_prot) ||
+		     (pisa_decrypt_enabled() && force_dma_unencrypted(dev) &&
+		      !(meta->flags & PISA_DECRYPTED))))
 		return false;
 
 	/* Acquire pairs with the .base release in pisa_domain_init(). */
@@ -388,7 +392,7 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 		phys_addr_t pisa_phys = phys & PISA_MASK;
 
 		ret = iommu_map(domain, pisa_phys_to_iova(pi, meta, pisa_phys),
-				pisa_phys, PISA_SIZE, block_prot, GFP_ATOMIC);
+				pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
 		if (unlikely(ret))
 			return false;
 		/* Ensure the mapping is visible before setting the domain bit. */
diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index 7dd91ff47288a..fed44c2050f9a 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -81,6 +81,12 @@ struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn);
 bool pisa_quarantine_page(struct pisa_meta *meta);
 void pisa_schedule_reclaim(void);
 
+/* Physical address for IOMMU PTE of @meta at @phys (shared alias if decrypted). */
+static inline phys_addr_t pisa_pte_phys(const struct pisa_meta *meta, phys_addr_t phys)
+{
+	return (meta->flags & PISA_DECRYPTED) ? dma_addr_unencrypted(phys) : phys;
+}
+
 /* IOVA of @phys inside active RAM page @meta in @pi. */
 static inline dma_addr_t pisa_phys_to_iova(const struct pisa_iova *pi,
 					   const struct pisa_meta *meta, phys_addr_t phys)
diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
index fd5e7234e9423..8034850d2c87b 100644
--- a/include/linux/pisa-mm.h
+++ b/include/linux/pisa-mm.h
@@ -121,6 +121,21 @@ static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
 	       __pisa_writeback_upgrade(mapping, folio, index, fbatch);
 }
 
+/*
+ * Hook for the paths that build a user PTE for a page that may be a decrypted
+ * PISA page (CoCo guests, @pisa_decrypt_key). Not tied to @pisa_user_key: RX
+ * buffers and coherent memory are mapped to user space too, and decrypted user
+ * pages outlive pisa.user_max_pages=0.
+ */
+static inline pgprot_t pisa_pgprot_decrypted(const struct page *page, pgprot_t prot)
+{
+	if (pisa_decrypt_enabled() &&
+	    (pisa_pfn_flags(page_to_pfn(page)) & (PISA_ACTIVE | PISA_DECRYPTED)) ==
+	    (PISA_ACTIVE | PISA_DECRYPTED))
+		return pgprot_decrypted(prot);
+	return prot;
+}
+
 #else /* !CONFIG_PISA */
 
 static inline unsigned long
@@ -175,6 +190,11 @@ static inline bool pisa_writeback_upgrade(const struct address_space *mapping,
 	return false;
 }
 
+static inline pgprot_t pisa_pgprot_decrypted(const struct page *page, pgprot_t prot)
+{
+	return prot;
+}
+
 static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
 			       unsigned long end, bool to_pisa)
 {
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 11e3ea9eb263d..92a4c529d1605 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -62,20 +62,24 @@ static inline struct pisa_iova *pisa_dma_iova(struct iommu_domain *domain) { ret
 enum {
 	_PISA_ACTIVE,
 	_PISA_QUARANTINED,
+	_PISA_DECRYPTED,
 };
 
 #define PISA_ACTIVE		BIT(_PISA_ACTIVE)	/* set up as a PISA page, not retired yet */
 #define PISA_QUARANTINED	BIT(_PISA_QUARANTINED)	/* IOMMU unmap failed: never reuse */
+#define PISA_DECRYPTED		BIT(_PISA_DECRYPTED)	/* page was set_memory_decrypted() */
 
 /*
  * Static keys guarding all kernel hooks:
  * - @pisa_key: enabled on first device opt-in (pisa_meta_init()).
  * - @pisa_recycle_check_key: enabled when pisa.compact or pisa.target_usage_percent > 0.
  * - @pisa_user_key: enabled when pisa.user_max_pages > 0.
+ * - @pisa_decrypt_key: enabled at boot in CoCo guests, where PISA pages are decrypted.
  */
 DECLARE_STATIC_KEY_FALSE(pisa_key);
 DECLARE_STATIC_KEY_FALSE(pisa_recycle_check_key);
 DECLARE_STATIC_KEY_FALSE(pisa_user_key);
+DECLARE_STATIC_KEY_FALSE(pisa_decrypt_key);
 
 extern u32 *pisa_pfn2slot;
 extern unsigned int pisa_nr_slots;
@@ -92,6 +96,7 @@ void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
 				    unsigned long attrs);
 bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
 bool __pisa_page_can_recycle(const struct page *page);
+bool __is_pisa_direct(phys_addr_t phys, size_t size);
 bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			     size_t size, size_t offset,
 			     enum dma_data_direction dir, bool pinned);
@@ -107,6 +112,11 @@ static inline bool pisa_user_enabled(void)
 	return static_branch_unlikely(&pisa_user_key);
 }
 
+static inline bool pisa_decrypt_enabled(void)
+{
+	return static_branch_unlikely(&pisa_decrypt_key);
+}
+
 static inline u32 pisa_pfn_flags(unsigned long pfn)
 {
 	unsigned long frame = pfn >> PISA_ORDER;
@@ -181,6 +191,12 @@ static inline bool pisa_free_coherent(const void *cpu_addr, size_t size)
 	return true;
 }
 
+/* Hook for dma_direct_map_phys() CoCo swiotlb bypass (guarded by @pisa_key). */
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+	return pisa_enabled() && __is_pisa_direct(phys, size);
+}
+
 /* Hook for __page_pool_alloc_netmems_slow() (guarded by @pisa_key). */
 static inline gfp_t pisa_rx_gfp(const struct device *dev)
 {
@@ -237,6 +253,11 @@ static inline bool pisa_free_coherent(const void *cpu_addr, size_t size)
 	return false;
 }
 
+static inline bool is_pisa_direct(phys_addr_t phys, size_t size)
+{
+	return false;
+}
+
 static inline gfp_t pisa_rx_gfp(const struct device *dev)
 {
 	return 0;
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 9ac4912e4a0a1..545f2fdb6088d 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -709,6 +709,10 @@ dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys,
 		return swiotlb_map(dev, phys, size, dir, attrs);
 	}
 
+	if (force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev) &&
+	    is_pisa_direct(phys, size))
+		attrs |= DMA_ATTR_CC_SHARED;
+
 	if (attrs & DMA_ATTR_CC_SHARED)
 		dma_addr = phys_to_dma_unencrypted(dev, phys);
 	else
diff --git a/mm/ksm.c b/mm/ksm.c
index d24b129932bf7..f5763b0bdb4fb 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -23,6 +23,7 @@
 #include <linux/sched/cputime.h>
 #include <linux/rwsem.h>
 #include <linux/pagemap.h>
+#include <linux/pisa-mm.h>
 #include <linux/rmap.h>
 #include <linux/spinlock.h>
 #include <linux/xxhash.h>
@@ -1447,7 +1448,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,
 	if (!is_zero_pfn(page_to_pfn(kpage))) {
 		folio_get(kfolio);
 		folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE);
-		newpte = mk_pte(kpage, vma->vm_page_prot);
+		newpte = mk_pte(kpage, pisa_pgprot_decrypted(kpage, vma->vm_page_prot));
 	} else {
 		/*
 		 * Use pte_mkdirty to mark the zero page mapped by KSM, and then
diff --git a/mm/memory.c b/mm/memory.c
index 7f019afc35ca0..90aad56b97b6d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -50,6 +50,7 @@
 #include <linux/swap.h>
 #include <linux/highmem.h>
 #include <linux/pagemap.h>
+#include <linux/pisa-mm.h>
 #include <linux/memremap.h>
 #include <linux/kmsan.h>
 #include <linux/ksm.h>
@@ -915,7 +916,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma,
 
 	VM_WARN_ON_FOLIO(!folio_test_locked(folio), folio);
 
-	pte = pte_mkold(mk_pte(page, READ_ONCE(vma->vm_page_prot)));
+	pte = pte_mkold(mk_pte(page, pisa_pgprot_decrypted(page, READ_ONCE(vma->vm_page_prot))));
 	if (pte_swp_soft_dirty(orig_pte))
 		pte = pte_mksoft_dirty(pte);
 
@@ -2487,6 +2488,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte,
 	}
 
 	/* Ok, finally just insert the thing.. */
+	prot = pisa_pgprot_decrypted(page, prot);
 	pteval = mk_pte(page, prot);
 	if (unlikely(is_zero_folio(folio))) {
 		pteval = pte_mkspecial(pteval);
@@ -5285,7 +5287,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 
 	add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages);
 	add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages);
-	pte = mk_pte(page, vma->vm_page_prot);
+	pte = mk_pte(page, pisa_pgprot_decrypted(page, vma->vm_page_prot));
 	if (pte_swp_soft_dirty(vmf->orig_pte))
 		pte = pte_mksoft_dirty(pte);
 	if (pte_swp_uffd(vmf->orig_pte))
@@ -5836,7 +5838,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,
 	pte_t entry;
 
 	flush_icache_pages(vma, page, nr);
-	entry = mk_pte(page, vma->vm_page_prot);
+	entry = mk_pte(page, pisa_pgprot_decrypted(page, vma->vm_page_prot));
 
 	if (prefault && arch_wants_old_prefaulted_pte())
 		entry = pte_mkold(entry);
diff --git a/mm/migrate.c b/mm/migrate.c
index e6cd83cf32aa1..9b1d40710ff0f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -37,6 +37,7 @@
 #include <linux/gfp.h>
 #include <linux/page_idle.h>
 #include <linux/page_owner.h>
+#include <linux/pisa-mm.h>
 #include <linux/sched/mm.h>
 #include <linux/ptrace.h>
 #include <linux/memory.h>
@@ -398,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		folio_get(folio);
 		new = folio_page(folio, idx);
-		pte = mk_pte(new, READ_ONCE(vma->vm_page_prot));
+		pte = mk_pte(new, pisa_pgprot_decrypted(new, READ_ONCE(vma->vm_page_prot)));
 		if (!softleaf_is_migration_young(entry))
 			pte = pte_mkold(pte);
 		if (folio_test_dirty(folio) && softleaf_is_migration_dirty(entry))
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index 4fc3b4d29de67..edfa7cf424451 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -9,6 +9,7 @@
 #include <linux/bitmap.h>
 #include <linux/bitops.h>
 #include <linux/cache.h>
+#include <linux/cc_platform.h>
 #include <linux/cpuhotplug.h>
 #include <linux/device.h>
 #include <linux/dma-direct.h>
@@ -178,7 +179,7 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
 
 	node = llist_del_all(&pisa_free_list);
 	llist_for_each_entry_safe(meta, tmp, node, llnode)
-		pisa_release_page(meta);
+		__pisa_release_page(meta);
 
 	if (next_delay != MAX_JIFFY_OFFSET)
 		queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
@@ -833,7 +834,7 @@ static struct pisa_meta *pisa_alloc_split_page(unsigned int order, gfp_t gfp, in
 	int err;
 
 	/* __GFP_DMA/__GFP_DMA32 were already declined by pisa_alloc_bulk(). */
-	meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, &clean);
+	meta = pisa_reservoir_pop(nid, gfp & ~__GFP_ZERO, pisa_want_decrypt(), &clean);
 	if (IS_ERR(meta))
 		return meta;
 
@@ -1268,7 +1269,7 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 		    !pisa_coherent_dev_fits(dev, pi))
 			return NULL;
 	} else if (is_swiotlb_for_alloc(dev) ||
-		   force_dma_unencrypted(dev) ||
+		   (force_dma_unencrypted(dev) && !pisa_want_decrypt()) ||
 		   !dma_coherent_ok(dev, PFN_PHYS(max_pfn) - 1, 1)) {
 		return NULL;
 	}
@@ -1331,6 +1332,8 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 			pisa_free_coherent(page_address(page), size);
 			return NULL;
 		}
+	} else if (force_dma_unencrypted(dev)) {
+		*dma_handle = phys_to_dma_unencrypted(dev, page_to_phys(page));
 	} else {
 		*dma_handle = phys_to_dma(dev, page_to_phys(page));
 	}
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index 0dbc3ecf44465..c195fdfdbd9fa 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -66,7 +66,7 @@ static void pisa_meta_set_active(struct pisa_meta *meta, bool active)
 	if (active)
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
 	else
-		pisa_meta_clear_flags(meta, PISA_ACTIVE);
+		pisa_meta_clear_flags(meta, PISA_ACTIVE | PISA_DECRYPTED);
 }
 
 static void test_meta_active_toggle(struct kunit *test)
@@ -74,6 +74,7 @@ static void test_meta_active_toggle(struct kunit *test)
 	unsigned long pfn, base_pfn;
 	struct pisa_meta *meta;
 	struct page *page;
+	phys_addr_t pa;
 
 	KUNIT_ASSERT_EQ(test, pisa_meta_init(), 0);
 
@@ -91,16 +92,26 @@ static void test_meta_active_toggle(struct kunit *test)
 	}
 	KUNIT_EXPECT_EQ(test, pisa_meta_ensure_pfn(pfn, false), 0);
 	meta = pisa_meta_of_pfn(pfn);
+	pa = page_to_phys(page);
 
 	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+	KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
 
 	pisa_meta_set_active(meta, true);
 	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn));
 	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(pfn));
 	KUNIT_EXPECT_TRUE(test, is_pisa_pfn(base_pfn + (1UL << PISA_ORDER) - 1));
+	KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
+	KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), pa);
+
+	pisa_meta_set_flags(meta, PISA_ACTIVE | PISA_DECRYPTED);
+	KUNIT_EXPECT_TRUE(test, is_pisa_direct(pa, PAGE_SIZE));
+	KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PISA_SIZE + PAGE_SIZE));
+	KUNIT_EXPECT_EQ(test, pisa_pte_phys(meta, pa), dma_addr_unencrypted(pa));
 
 	pisa_meta_set_active(meta, false);
 	KUNIT_EXPECT_FALSE(test, is_pisa_pfn(pfn));
+	KUNIT_EXPECT_FALSE(test, is_pisa_direct(pa, PAGE_SIZE));
 
 	pisa_slot_free(meta);
 	__free_pages(page, PISA_ORDER);
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 25d3add7a4891..99c5bc42eed10 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -176,6 +176,19 @@ struct pisa_meta *pisa_meta_of_pfn(unsigned long pfn)
 	return &pisa_meta_array[slot - 1];
 }
 
+bool __is_pisa_direct(phys_addr_t phys, size_t size)
+{
+	u32 flags = pisa_pfn_flags(PHYS_PFN(phys));
+
+	/*
+	 * Only decrypted active pages can bypass swiotlb bouncing, and the
+	 * transfer must not straddle the end of the PISA frame.
+	 */
+	return (flags & (PISA_ACTIVE | PISA_DECRYPTED)) ==
+	       (PISA_ACTIVE | PISA_DECRYPTED) &&
+	       (phys & (PISA_SIZE - 1)) + size <= PISA_SIZE;
+}
+
 /*
  * Update @meta's position on @pa's lists under @pa->lock. When all blocks
  * become available (avail == bpp), either queues @meta to @pisa_free_list (and
@@ -716,6 +729,9 @@ int pisa_meta_init(void)
 
 	if (pisa_enabled())
 		return 0;
+	/* Never hand a device memory that it cannot access. */
+	if (pisa_cc_unsupported())
+		return -EOPNOTSUPP;
 
 	mutex_lock(&pisa_meta_mutex);
 	if (pisa_init_err) {
@@ -790,6 +806,7 @@ int pisa_meta_init(void)
 	mutex_unlock(&pisa_meta_mutex);
 out_enable:
 	static_branch_enable(&pisa_key);
+	pisa_reservoir_prime();
 	pisa_schedule_reservoir();
 	return 0;
 
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index 8d6fa9f36bdcf..12fbb617c342b 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -6,6 +6,7 @@
 
 #include <linux/atomic.h>
 #include <linux/bitops.h>
+#include <linux/cc_platform.h>
 #include <linux/device.h>
 #include <linux/export.h>
 #include <linux/pisa.h>
@@ -18,6 +19,7 @@
 #include <linux/rcupdate.h>
 #include <linux/sched.h>
 #include <linux/sched/mm.h>
+#include <linux/set_memory.h>
 #include <linux/spinlock.h>
 #include <linux/topology.h>
 #include <linux/workqueue.h>
@@ -32,6 +34,22 @@
  */
 unsigned int pisa_reservoir_max __read_mostly = 16;
 
+/* Enabled in CoCo guests by pisa_node_init(): new PISA pages are decrypted for DMA. */
+DEFINE_STATIC_KEY_FALSE(pisa_decrypt_key);
+
+/* A confidential guest: PISA pages are decrypted for DMA. A bare-metal SME host is not one. */
+static bool pisa_coco_guest(void)
+{
+	return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT);
+}
+
+/* Memory encryption that PISA cannot handle: neither an SME host nor a guest it decrypts for. */
+bool pisa_cc_unsupported(void)
+{
+	return cc_platform_has(CC_ATTR_MEM_ENCRYPT) &&
+	       !cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT) && !pisa_coco_guest();
+}
+
 /* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
 struct pisa_node pisa_nodes[MAX_NUMNODES] = {
 	[0 ... MAX_NUMNODES - 1] = {
@@ -45,7 +63,9 @@ struct pisa_node pisa_nodes[MAX_NUMNODES] = {
 
 static int pisa_reservoir_max_set(const char *val, const struct kernel_param *kp)
 {
-	int ret = param_set_uint(val, kp);
+	/* CoCo guests: only the reservoir gives decrypted pages to atomic callers. */
+	unsigned int min = pisa_coco_guest() ? 1 : 0;
+	int ret = param_set_uint_minmax(val, kp, min, UINT_MAX);
 
 	/* Schedule per-node workers to refill or trim once pisa_meta_init() has run. */
 	if (pisa_enabled() && !ret)
@@ -71,9 +91,49 @@ static unsigned int pisa_memalloc_save(gfp_t gfp)
 	return memalloc_flags_save(flags);
 }
 
+/* Return true if a new PISA page should be decrypted: CoCo guests, not bare-metal SME hosts. */
+bool pisa_want_decrypt(void)
+{
+	return pisa_decrypt_enabled();
+}
+
+/* Decrypt and zero a new PISA page. Must be called from sleepable context. */
+static int pisa_page_prepare(struct page *page, struct pisa_meta *meta)
+{
+	void *vaddr = page_address(page);
+	int ret;
+
+	ret = set_memory_decrypted((unsigned long)vaddr, 1 << PISA_ORDER);
+	if (ret) {
+		/* Leak partially shared page rather than returning it to the buddy allocator. */
+		pr_warn_ratelimited("pisa: leaking PISA page that could not be decrypted\n");
+		return ret;
+	}
+	/* Initialize cleartext cachelines after flipping encryption state. */
+	memset(vaddr, 0, PISA_SIZE);
+	pisa_meta_set_flags(meta, PISA_DECRYPTED);
+	return 0;
+}
+
+/* Undo pisa_page_prepare(). */
+static int pisa_page_unprepare(struct pisa_meta *meta)
+{
+	struct page *page;
+
+	if (!(meta->flags & PISA_DECRYPTED))
+		return 0;
+	page = pisa_meta_to_page(meta);
+	if (set_memory_encrypted((unsigned long)page_address(page), 1 << PISA_ORDER)) {
+		pr_warn_ratelimited("pisa: leaking PISA page that could not be re-encrypted\n");
+		return -EIO;
+	}
+	pisa_meta_clear_flags(meta, PISA_DECRYPTED);
+	return 0;
+}
+
 /* Allocate a PISA page from the buddy allocator and initialize its metadata. */
 static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
-					   bool *clean)
+					   bool decrypt, bool *clean)
 {
 	struct pisa_meta *meta;
 	unsigned int memflags;
@@ -97,6 +157,8 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
 	 * must not dip into emergency reserves or trigger OOM/warnings.
 	 */
 	gfp &= ~(__GFP_COMP | __GFP_HIGHMEM | __GFP_MOVABLE);
+	if (decrypt)
+		gfp &= ~__GFP_ZERO;
 	gfp |= __GFP_NOWARN | __GFP_NOMEMALLOC | (can_block ? __GFP_RETRY_MAYFAIL : __GFP_NORETRY);
 
 	if (can_block)
@@ -116,13 +178,21 @@ static struct pisa_meta *__pisa_page_alloc(gfp_t gfp, int nid, bool can_block,
 		__free_pages(page, PISA_ORDER);
 		return ERR_PTR(err);
 	}
-	if (can_block)
-		memalloc_flags_restore(memflags);
 
 	meta = pisa_meta_of_pfn(page_to_pfn(page));
 
 	atomic_long_inc(&pisa_nr_pages);
-	*clean = (gfp & __GFP_ZERO) || want_init_on_alloc(0) || want_init_on_free();
+	/* On decrypt failure the page is leaked and stays charged to budget. */
+	if (decrypt && pisa_page_prepare(page, meta)) {
+		pisa_slot_free(meta);
+		if (can_block)
+			memalloc_flags_restore(memflags);
+		return ERR_PTR(-EIO);
+	}
+	if (can_block)
+		memalloc_flags_restore(memflags);
+	*clean = (gfp & __GFP_ZERO) || (meta->flags & PISA_DECRYPTED) ||
+		 want_init_on_alloc(0) || want_init_on_free();
 
 	return meta;
 }
@@ -150,8 +220,8 @@ static void pisa_release_page_rcu(struct rcu_head *head)
 	/* The page has been unmapped from every domain by pisa_release_page() */
 	pisa_slot_free(meta);
 
-	/* Possibly still mapped by a device: leak it, still charged. */
-	if (flags & PISA_QUARANTINED)
+	/* Still shared with the host, or possibly still mapped by a device. */
+	if (flags & (PISA_DECRYPTED | PISA_QUARANTINED))
 		return;
 	atomic_long_dec(&pisa_nr_pages);
 	for (unsigned int i = 0; i < PISA_BLOCKS(order); i++) {
@@ -162,12 +232,20 @@ static void pisa_release_page_rcu(struct rcu_head *head)
 	}
 }
 
-/* Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release. */
-void pisa_release_page(struct pisa_meta *meta)
+/*
+ * Unmap a retired PISA page, clear PISA_ACTIVE, and queue RCU release.
+ * Re-encrypting a decrypted page may sleep.
+ */
+void __pisa_release_page(struct pisa_meta *meta)
 {
 	if (!meta->pa) {
 		struct page *page = pisa_meta_to_page(meta);
 
+		/* Unbound (reservoir) page: leaked if re-encryption fails. */
+		if (pisa_page_unprepare(meta)) {
+			pisa_slot_free(meta);
+			return;
+		}
 		pisa_slot_free(meta);
 		__free_pages(page, PISA_ORDER);
 		atomic_long_dec(&pisa_nr_pages);
@@ -182,9 +260,26 @@ void pisa_release_page(struct pisa_meta *meta)
 
 	/* Clear PISA_ACTIVE before the RCU grace period so new readers stop entering. */
 	pisa_meta_clear_flags(meta, PISA_ACTIVE);
+	if ((READ_ONCE(meta->flags) & (PISA_DECRYPTED | PISA_QUARANTINED)) == PISA_DECRYPTED) {
+		synchronize_rcu();
+		if (!(READ_ONCE(meta->flags) & PISA_QUARANTINED))
+			pisa_page_unprepare(meta);
+	}
+
 	call_rcu(&meta->rcu, pisa_release_page_rcu);
 }
 
+/* Release @meta from any context: a decrypted page goes through the reclaim worker. */
+void pisa_release_page(struct pisa_meta *meta)
+{
+	if (meta->flags & PISA_DECRYPTED) {
+		llist_add(&meta->llnode, &pisa_free_list);
+		pisa_schedule_reclaim();
+	} else {
+		__pisa_release_page(meta);
+	}
+}
+
 /* Refill node @nid's reservoir up to pisa_reservoir_max pre-zeroed PISA pages. */
 static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
 {
@@ -193,13 +288,14 @@ static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
 		unsigned long flags;
 		bool clean, added;
 
-		meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true, &clean);
+		meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
+					 pisa_want_decrypt(), &clean);
 		if (IS_ERR(meta)) {
 			pisa_alloc_backoff(node, false);
 			break;
 		}
 
-		/* init_on_alloc/init_on_free pages are already zeroed. */
+		/* Decrypted and init_on_alloc/init_on_free pages are already zeroed. */
 		if (!clean)
 			memset(page_address(pisa_meta_to_page(meta)), 0, PISA_SIZE);
 
@@ -212,7 +308,7 @@ static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
 		}
 		spin_unlock_irqrestore(&node->lock, flags);
 		if (!added) {
-			pisa_release_page(meta);
+			__pisa_release_page(meta);
 			break;
 		}
 
@@ -238,7 +334,7 @@ static void pisa_reservoir_trim_node(struct pisa_node *node, unsigned int max)
 		if (!meta)
 			break;
 
-		pisa_release_page(meta);
+		__pisa_release_page(meta);
 		cond_resched();
 	}
 }
@@ -303,8 +399,22 @@ void pisa_schedule_reservoir(void)
 		pisa_schedule_reservoir_node(nid);
 }
 
+/*
+ * CoCo guests: atomic callers only get decrypted pages from the reservoirs, so
+ * fill them before pisa_meta_init() returns instead of waiting for the workers.
+ */
+void pisa_reservoir_prime(void)
+{
+	int nid;
+
+	if (!pisa_want_decrypt())
+		return;
+	for_each_node_state(nid, N_MEMORY)
+		pisa_reservoir_refill_node(&pisa_nodes[nid], nid);
+}
+
 /* Pop one pre-prepared PISA page from node @nid's reservoir under @node->lock. */
-static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
+static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool decrypt)
 {
 	struct pisa_node *node = &pisa_nodes[nid];
 	struct pisa_meta *meta;
@@ -316,10 +426,12 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
 
 	spin_lock_irqsave(&node->lock, flags);
 	meta = list_first_entry_or_null(&node->pages, struct pisa_meta, list);
-	if (meta) {
+	if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
 		list_del_init(&meta->list);
 		WRITE_ONCE(node->nr_pages, node->nr_pages - 1);
 		refill = node->nr_pages <= max / 2;
+	} else {
+		meta = NULL;
 	}
 	spin_unlock_irqrestore(&node->lock, flags);
 
@@ -333,14 +445,14 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max)
  * Pop a pre-prepared PISA page from @nid's reservoir (scheduling an async
  * refill when <= half full or empty), or allocate one inline on miss.
  */
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean)
 {
 	bool can_block = gfpflags_allow_blocking(gfp);
 	unsigned int max = READ_ONCE(pisa_reservoir_max);
 	struct pisa_meta *meta;
 
 	if (max) {
-		meta = pisa_reservoir_pop_node(nid, max);
+		meta = pisa_reservoir_pop_node(nid, max, decrypt);
 		if (meta) {
 			*clean = true;
 			return meta;
@@ -352,8 +464,11 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean)
 		return ERR_PTR(-EAGAIN);
 	if (max)
 		pisa_schedule_reservoir_node(nid);
+	/* Page decryption may sleep; atomic callers must rely on the reservoir. */
+	if (decrypt && !can_block)
+		return ERR_PTR(-EAGAIN);
 
-	meta = __pisa_page_alloc(gfp, nid, can_block, clean);
+	meta = __pisa_page_alloc(gfp, nid, can_block, decrypt, clean);
 	if (IS_ERR(meta))
 		pisa_alloc_backoff(&pisa_nodes[nid], can_block);
 	return meta;
@@ -391,6 +506,14 @@ unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned lon
 
 void __init pisa_node_init(void)
 {
+	/*
+	 * CoCo guests require decrypted pages and cannot decrypt in atomic
+	 * context: those callers rely on the per-node reservoir, which
+	 * pisa_reservoir_max_set() does not let a CoCo guest disable.
+	 */
+	if (pisa_coco_guest())
+		static_branch_enable(&pisa_decrypt_key);
+
 	for (int nid = 0; nid < MAX_NUMNODES; nid++) {
 		spin_lock_init(&pisa_nodes[nid].lock);
 		INIT_LIST_HEAD(&pisa_nodes[nid].pages);
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 08bf1b64bcf2c..865f3d779b84d 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -10,6 +10,7 @@
 #undef MODULE_PARAM_PREFIX
 #define MODULE_PARAM_PREFIX "pisa."
 #include <linux/jiffies.h>
+#include <linux/mem_encrypt.h>
 #include <linux/nodemask.h>
 #include <linux/spinlock.h>
 #include <linux/topology.h>
@@ -204,6 +205,9 @@ static inline void pisa_alloc_backoff(struct pisa_node *node, bool can_block)
 		WRITE_ONCE(node->next_alloc, jiffies + DIV_ROUND_UP(HZ, 100));
 }
 
+bool pisa_want_decrypt(void);
+bool pisa_cc_unsupported(void);
+
 /* pisa-alloc.c, also used by the per-node code in pisa-node.c */
 struct pisa_alloc *pisa_alloc_create(int nid, unsigned int order, enum dma_data_direction dir);
 __printf(2, 3)
@@ -220,6 +224,7 @@ bool pisa_place_meta(struct pisa_meta *meta);
 extern struct llist_head pisa_free_list;
 extern unsigned int pisa_idle_ms;
 void pisa_release_page(struct pisa_meta *meta);
+void __pisa_release_page(struct pisa_meta *meta);
 unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
 			      unsigned long nr_pages, struct page **page_array, bool frozen);
 
@@ -249,9 +254,10 @@ bool pisa_user_scrub_node(int nid);
 extern unsigned int pisa_reservoir_max;
 void pisa_node_init(void);
 void pisa_schedule_reservoir(void);
+void pisa_reservoir_prime(void);
 void pisa_schedule_scrub(int nid);
 void pisa_schedule_scrub_all(void);
-struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
+struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
 unsigned long pisa_reservoir_count(void);
 unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
 void pisa_user_boot_enable(void);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 2d2df53fc51a9..c8952ac306d74 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -17,6 +17,7 @@
 #include <linux/swap.h>
 #include <linux/vmalloc.h>
 #include <linux/pagemap.h>
+#include <linux/pisa-mm.h>
 #include <linux/namei.h>
 #include <linux/shmem_fs.h>
 #include <linux/blk-cgroup.h>
@@ -2587,7 +2588,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,
 		folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
 		folio_add_lru_vma(folio, vma);
 	}
-	new_pte = pte_mkold(mk_pte(page, vma->vm_page_prot));
+	new_pte = pte_mkold(mk_pte(page, pisa_pgprot_decrypted(page, vma->vm_page_prot)));
 	if (pte_swp_soft_dirty(old_pte))
 		new_pte = pte_mksoft_dirty(new_pte);
 	if (pte_swp_uffd(old_pte))
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 14fdbf217f00e..200f56bcba069 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -13,6 +13,7 @@
 #include <linux/mm.h>
 #include <linux/sched/signal.h>
 #include <linux/pagemap.h>
+#include <linux/pisa-mm.h>
 #include <linux/rmap.h>
 #include <linux/swap.h>
 #include <linux/leafops.h>
@@ -341,7 +342,7 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd,
 	bool page_in_cache = folio_mapping(folio);
 	pte_t dst_ptep;
 
-	_dst_pte = mk_pte(page, dst_vma->vm_page_prot);
+	_dst_pte = mk_pte(page, pisa_pgprot_decrypted(page, dst_vma->vm_page_prot));
 	_dst_pte = pte_mkdirty(_dst_pte);
 	if (page_in_cache && !vm_shared)
 		writable = false;
@@ -1303,7 +1304,9 @@ static long move_present_ptes(struct mm_struct *mm,
 		folio_move_anon_rmap(src_folio, dst_vma);
 		src_folio->index = linear_anon_page_index(dst_vma, dst_addr);
 
-		orig_dst_pte = folio_mk_pte(src_folio, dst_vma->vm_page_prot);
+		orig_dst_pte = folio_mk_pte(src_folio,
+					    pisa_pgprot_decrypted(folio_page(src_folio, 0),
+								  dst_vma->vm_page_prot));
 		/* Set soft dirty bit so userspace can notice the pte was moved */
 		if (pgtable_supports_soft_dirty())
 			orig_dst_pte = pte_mksoft_dirty(orig_dst_pte);
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (18 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs Luigi Rizzo
                     ` (6 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

An arm64 Realm (ARM CCA guest) only reports CC_ATTR_MEM_ENCRYPT, so PISA
did not treat it as a CoCo guest and refused to initialize there.

Treat a Realm as a CoCo guest: PISA pages are decrypted (shared with the
host) when they are allocated, as on SEV-SNP and TDX.

A decrypted page is mapped with PROT_NS_SHARED, the top IPA bit, set in
the address field of its PTEs. PISA maps such pages into user space with
ordinary PTEs (page cache, migrated user pages, zerocopy receive), and
for those pte_pfn() must be the PFN of the page, or vm_normal_page() and
GUP see a PFN that has no struct page.

Make __pte_to_phys() strip the bit when what is left is a decrypted PISA
page. The check is behind pisa_decrypt_key, so outside CoCo guests it
costs one static branch, and every other shared mapping (ioremap() of
emulated MMIO, swiotlb, DMA mmap) keeps the address it has today.

pisa_decrypt_key and the lookup are exported because pte_pfn() is used
by modules.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 arch/arm64/include/asm/pgtable.h | 28 ++++++++++++++++++++++++++--
 mm/pisa-meta.c                   |  9 +++++++++
 mm/pisa-node.c                   |  9 +++++++--
 3 files changed, 42 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index a474c9fb485ad..86c60b8f43ae0 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -108,7 +108,7 @@ static inline void arch_leave_lazy_mmu_mode(void)
 			  TLBF_NOBROADCAST | TLBF_NONOTIFY | TLBF_NOWALKCACHE)
 
 #ifdef CONFIG_ARM64_PA_BITS_52
-static inline phys_addr_t __pte_to_phys(pte_t pte)
+static inline phys_addr_t __pte_to_phys_raw(pte_t pte)
 {
 	pte_val(pte) &= ~PTE_MAYBE_SHARED;
 	return (pte_val(pte) & PTE_ADDR_LOW) |
@@ -119,7 +119,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
 	return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK;
 }
 #else
-static inline phys_addr_t __pte_to_phys(pte_t pte)
+static inline phys_addr_t __pte_to_phys_raw(pte_t pte)
 {
 	return pte_val(pte) & PTE_ADDR_LOW;
 }
@@ -130,6 +130,30 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
 }
 #endif
 
+#if defined(CONFIG_PISA) && !defined(__DISABLE_EXPORTS)
+DECLARE_STATIC_KEY_FALSE(pisa_decrypt_key);
+bool __pisa_phys_decrypted(phys_addr_t phys);
+#endif
+
+static inline phys_addr_t __pte_to_phys(pte_t pte)
+{
+#if defined(CONFIG_PISA) && !defined(__DISABLE_EXPORTS)
+	/*
+	 * In a Realm, PISA maps its decrypted pages into user space with ordinary
+	 * PTEs. Those have PROT_NS_SHARED, the top IPA bit, set in their address
+	 * field, but pte_pfn() must be the PFN of the page: strip the bit when
+	 * what is left is a decrypted PISA page. Other shared mappings keep it.
+	 */
+	if (static_branch_unlikely(&pisa_decrypt_key) && (pte_val(pte) & PROT_NS_SHARED)) {
+		phys_addr_t phys = __pte_to_phys_raw(__pte(pte_val(pte) & ~PROT_NS_SHARED));
+
+		if (__pisa_phys_decrypted(phys))
+			return phys;
+	}
+#endif
+	return __pte_to_phys_raw(pte);
+}
+
 #define pte_pfn(pte)		(__pte_to_phys(pte) >> PAGE_SHIFT)
 #define pfn_pte(pfn,prot)	\
 	__pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
diff --git a/mm/pisa-meta.c b/mm/pisa-meta.c
index 99c5bc42eed10..b608ef71152af 100644
--- a/mm/pisa-meta.c
+++ b/mm/pisa-meta.c
@@ -189,6 +189,15 @@ bool __is_pisa_direct(phys_addr_t phys, size_t size)
 	       (phys & (PISA_SIZE - 1)) + size <= PISA_SIZE;
 }
 
+#ifdef CONFIG_ARM64
+/* For __pte_to_phys(): true if @phys is in a decrypted PISA page. */
+bool __pisa_phys_decrypted(phys_addr_t phys)
+{
+	return pisa_pfn_flags(PHYS_PFN(phys)) & PISA_DECRYPTED;
+}
+EXPORT_SYMBOL(__pisa_phys_decrypted);
+#endif
+
 /*
  * Update @meta's position on @pa's lists under @pa->lock. When all blocks
  * become available (avail == bpp), either queues @meta to @pisa_free_list (and
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index 12fbb617c342b..aa4f8759ae5e8 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -36,11 +36,16 @@ unsigned int pisa_reservoir_max __read_mostly = 16;
 
 /* Enabled in CoCo guests by pisa_node_init(): new PISA pages are decrypted for DMA. */
 DEFINE_STATIC_KEY_FALSE(pisa_decrypt_key);
+EXPORT_SYMBOL(pisa_decrypt_key);
 
-/* A confidential guest: PISA pages are decrypted for DMA. A bare-metal SME host is not one. */
+/*
+ * A confidential guest: PISA pages are decrypted for DMA. A bare-metal SME host
+ * is not one. An arm64 Realm only reports CC_ATTR_MEM_ENCRYPT.
+ */
 static bool pisa_coco_guest(void)
 {
-	return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT);
+	return cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) ||
+	       (IS_ENABLED(CONFIG_ARM64) && cc_platform_has(CC_ATTR_MEM_ENCRYPT));
 }
 
 /* Memory encryption that PISA cannot handle: neither an SME host nor a guest it decrypts for. */
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (19 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
                     ` (5 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

PISA allocation and mapping failures silently fall back to standard page
allocation and per-buffer DMA mappings, making performance regressions
hard to diagnose without observability.

Add per-allocator, per-node reservoir, user-page migration, and global
shrinker/reclaim counters, and expose them via /sys/kernel/debug/pisa/allocs.
Also emit a one-shot dmesg warning on the first 2MB buddy allocation
failure.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/iommu/pisa-map.c   |  91 ++++++++++++-
 include/linux/pisa-iommu.h |  20 +++
 include/linux/pisa.h       |   2 +
 mm/pisa-alloc.c            | 270 +++++++++++++++++++++++++++++++++----
 mm/pisa-kunit.c            |   6 +-
 mm/pisa-node.c             |  40 ++++++
 mm/pisa-priv.h             |  57 +++++++-
 mm/pisa-user.c             |  71 +++++++++-
 net/core/pisa-net.c        |   2 +
 9 files changed, 521 insertions(+), 38 deletions(-)

diff --git a/drivers/iommu/pisa-map.c b/drivers/iommu/pisa-map.c
index 08ac91fafcf20..fc6fa3682f045 100644
--- a/drivers/iommu/pisa-map.c
+++ b/drivers/iommu/pisa-map.c
@@ -17,11 +17,21 @@
 #include <linux/mm.h>
 #include <linux/pci.h>
 #include <linux/pisa-iommu.h>
+#include <linux/seq_file.h>
 #include <linux/spinlock.h>
 #include <linux/srcu.h>
 
 #include "iommu-priv.h"
 
+static struct {
+	atomic64_t page_map;
+	atomic64_t unmap_fail;
+	atomic64_t fallback_nowindow;
+	atomic64_t fallback_noidx;
+	atomic64_t fallback_nohuge;
+	atomic64_t fallback_maperr;
+	atomic64_t fallback_prot;
+} pisa_map_stats;
 
 /*
  * Registry of PISA IOMMU domains. Each slot is FREE (!domain && !base),
@@ -32,6 +42,8 @@ static struct {
 	struct iommu_domain *domain;
 	dma_addr_t base;
 	spinlock_t map_lock;	/* serialises iommu_map() in __pisa_map_phys() */
+	atomic_t slots;
+	char name[32];
 } pisa_domains[PISA_MAX_DOMAINS] __read_mostly;
 static DEFINE_SPINLOCK(pisa_domains_lock);
 
@@ -41,12 +53,61 @@ DEFINE_STATIC_SRCU(pisa_srcu);
 /* Bumped under pisa_domains_lock whenever a domain index is released. */
 unsigned int pisa_domains_generation __read_mostly;
 
+void pisa_domains_show(struct seq_file *s)
+{
+	unsigned int active = 0, draining = 0;
+	char bmap[PISA_MAX_DOMAINS];
+
+	for (int idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+		if (READ_ONCE(pisa_domains[idx].domain)) {
+			active++;
+			bmap[idx] = 'A';
+		} else if (READ_ONCE(pisa_domains[idx].base)) {
+			draining++;
+			bmap[idx] = 'd';
+		} else {
+			bmap[idx] = '.';
+		}
+	}
+
+	pisa_seq_kv(s, "domains", { "active", active }, { "draining", draining },
+		    { "max", PISA_MAX_DOMAINS },
+		    { "mapped", atomic64_read(&pisa_map_stats.page_map) },
+		    { "unmap_fail", atomic64_read(&pisa_map_stats.unmap_fail) });
+	pisa_seq_kv(s, "map_fail",
+		    { "no_window", atomic64_read(&pisa_map_stats.fallback_nowindow) },
+		    { "no_idx", atomic64_read(&pisa_map_stats.fallback_noidx) },
+		    { "no_huge", atomic64_read(&pisa_map_stats.fallback_nohuge) },
+		    { "map_err", atomic64_read(&pisa_map_stats.fallback_maperr) },
+		    { "prot", atomic64_read(&pisa_map_stats.fallback_prot) });
+	for (int idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+		if ((idx & 63) == 0)
+			seq_printf(s, "  %-12s", idx ? "" : "domains_map");
+		if ((idx & 7) == 0)
+			seq_putc(s, ' ');
+		seq_putc(s, bmap[idx]);
+		if ((idx & 63) == 63 || idx == PISA_MAX_DOMAINS - 1)
+			seq_putc(s, '\n');
+	}
+
+	for (int idx = 0; idx < PISA_MAX_DOMAINS; idx++) {
+		dma_addr_t base = READ_ONCE(pisa_domains[idx].base);
+
+		if (!READ_ONCE(pisa_domains[idx].domain))
+			continue;
+		seq_printf(s, "  domain_%-5d base %pad size %lluM slots %d name %s\n",
+			   idx, &base, pisa_iova_size() >> 20,
+			   atomic_read(&pisa_domains[idx].slots), pisa_domains[idx].name);
+	}
+}
+
 /* Report a short IOMMU unmap and mark @meta quarantined so it is never reused. */
 static bool pisa_quarantine_meta(struct pisa_meta *meta, struct iommu_domain *domain,
 				 int idx, dma_addr_t iova, size_t unmapped)
 {
 	pr_err_ratelimited("pisa: unmap of IOVA %pad in domain %p (index %d) returned %zu, quarantining PISA page\n",
 			   &iova, domain, idx, unmapped);
+	atomic64_inc(&pisa_map_stats.unmap_fail);
 	return pisa_quarantine_page(meta);
 }
 
@@ -71,6 +132,7 @@ void pisa_unmap_all_domains(struct pisa_meta *meta)
 
 		if (!test_and_clear_bit(idx, meta->domains_mapped))
 			continue;
+		atomic_dec(&pisa_domains[idx].slots);
 
 		spin_lock_irqsave(&pisa_domains_lock, flags);
 		domain = pisa_domains[idx].domain;
@@ -138,6 +200,8 @@ void __pisa_domain_release(struct iommu_domain *domain)
 	spin_lock_irqsave(&pisa_domains_lock, flags);
 	/* Mark slot FREE and bump @pisa_domains_generation so NOIDX domains retry. */
 	pisa_domains[idx].base = 0;
+	atomic_set(&pisa_domains[idx].slots, 0);
+	pisa_domains[idx].name[0] = '\0';
 	WRITE_ONCE(pisa_domains_generation, pisa_domains_generation + 1);
 	spin_unlock_irqrestore(&pisa_domains_lock, flags);
 
@@ -176,6 +240,7 @@ void __pisa_domain_quiesce(struct iommu_domain *domain)
 		if (!test_bit(idx, meta->domains_mapped) ||
 		    !test_and_clear_bit(idx, meta->domains_mapped))
 			continue;
+		atomic_dec(&pisa_domains[idx].slots);
 
 		/* Unmap the page, batching the IOTLB flush into @gather. */
 		iova = base + pisa_meta2iova_offset(meta);
@@ -314,6 +379,9 @@ static int pisa_domain_init(struct device *dev, struct iommu_domain *domain,
 	pisa_domains[idx].domain = domain;
 	pisa_domains[idx].base = base;
 	spin_lock_init(&pisa_domains[idx].map_lock);
+	atomic_set(&pisa_domains[idx].slots, 0);
+	strscpy(pisa_domains[idx].name, dev_name(dev),
+		sizeof(pisa_domains[idx].name));
 	WRITE_ONCE(pi->domain_idx, idx + PISA_IDX_FIRST);
 	/* Publish .domain_idx before .base (paired with is_pisa_iova()). */
 	smp_store_release(&pi->base, base);
@@ -358,8 +426,10 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 	/* In a CoCo guest @dev can only reach decrypted pages; an SME host needs no check. */
 	if (unlikely((prot & ~block_prot) ||
 		     (pisa_decrypt_enabled() && force_dma_unencrypted(dev) &&
-		      !(meta->flags & PISA_DECRYPTED))))
+		      !(meta->flags & PISA_DECRYPTED)))) {
+		atomic64_inc(&pisa_map_stats.fallback_prot);
 		return false;
+	}
 
 	/* Acquire pairs with the .base release in pisa_domain_init(). */
 	if (unlikely(!smp_load_acquire(&pi->base))) {
@@ -368,8 +438,15 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 		if (likely(!ret))
 			ret = pisa_domain_init(dev, domain, pi);
 
-		if (ret)
+		if (ret) {
+			if (ret == -EOPNOTSUPP)
+				atomic64_inc(&pisa_map_stats.fallback_nohuge);
+			else if (ret == -EBUSY)
+				atomic64_inc(&pisa_map_stats.fallback_noidx);
+			else
+				atomic64_inc(&pisa_map_stats.fallback_nowindow);
 			return false;
+		}
 	}
 
 	addr = pisa_phys_to_iova(pi, meta, phys);
@@ -377,8 +454,10 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 	/* Check @dev's own DMA mask and bus limit against the window IOVA. */
 	limit = min_not_zero(dma_get_mask(dev), dev->bus_dma_limit);
 	limit = min_not_zero(limit, dev->coherent_dma_mask);
-	if (unlikely(addr + size - 1 > limit))
+	if (unlikely(addr + size - 1 > limit)) {
+		atomic64_inc(&pisa_map_stats.fallback_nowindow);
 		return false;
+	}
 
 	domain_idx = pi->domain_idx - PISA_IDX_FIRST;
 	if (likely(test_bit(domain_idx, meta->domains_mapped))) {
@@ -393,11 +472,15 @@ bool __pisa_map_phys(dma_addr_t *iova, struct device *dev,
 
 		ret = iommu_map(domain, pisa_phys_to_iova(pi, meta, pisa_phys),
 				pisa_pte_phys(meta, pisa_phys), PISA_SIZE, block_prot, GFP_ATOMIC);
-		if (unlikely(ret))
+		if (unlikely(ret)) {
+			atomic64_inc(&pisa_map_stats.fallback_maperr);
 			return false;
+		}
 		/* Ensure the mapping is visible before setting the domain bit. */
 		smp_mb__before_atomic();
 		set_bit(domain_idx, meta->domains_mapped);
+		atomic_inc(&pisa_domains[domain_idx].slots);
+		atomic64_inc(&pisa_map_stats.page_map);
 	}
 
 	*iova = addr;
diff --git a/include/linux/pisa-iommu.h b/include/linux/pisa-iommu.h
index fed44c2050f9a..aa91d3cbaa75c 100644
--- a/include/linux/pisa-iommu.h
+++ b/include/linux/pisa-iommu.h
@@ -11,6 +11,7 @@
 #include <linux/pisa.h>
 
 struct pisa_alloc;
+struct seq_file;
 
 #ifdef CONFIG_PISA
 
@@ -95,6 +96,23 @@ static inline dma_addr_t pisa_phys_to_iova(const struct pisa_iova *pi,
 	       (phys & (PISA_SIZE - 1));
 }
 
+/* One name/value pair of a debugfs statistics line. */
+struct pisa_kv {
+	const char *name;
+	s64 val;
+};
+
+void __pisa_seq_kv(struct seq_file *s, const char *title,
+		   const struct pisa_kv *st, unsigned int n);
+
+/* Print @title and fixed-width name/value pairs to debugfs seq_file @s. */
+#define pisa_seq_kv(s, title, ...)					\
+do {									\
+	const struct pisa_kv __st[] = { __VA_ARGS__ };			\
+									\
+	__pisa_seq_kv(s, title, __st, ARRAY_SIZE(__st));		\
+} while (0)
+
 #endif /* CONFIG_PISA */
 
 #if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
@@ -104,11 +122,13 @@ extern unsigned int pisa_domains_generation;
 
 void pisa_unmap_all_domains(struct pisa_meta *meta);
 int pisa_dev_dma_domain(struct device *dev, struct iommu_domain **domain);
+void pisa_domains_show(struct seq_file *s);
 int pisa_iova_declined(const struct pisa_iova *pi);
 
 #else /* !CONFIG_PISA || !CONFIG_IOMMU_DMA */
 
 static inline void pisa_unmap_all_domains(struct pisa_meta *meta) {}
+static inline void pisa_domains_show(struct seq_file *s) {}
 static inline int pisa_iova_declined(const struct pisa_iova *pi) { return -EOPNOTSUPP; }
 
 static inline int pisa_dev_dma_domain(struct device *dev, struct iommu_domain **domain)
diff --git a/include/linux/pisa.h b/include/linux/pisa.h
index 92a4c529d1605..8ca1a459a2279 100644
--- a/include/linux/pisa.h
+++ b/include/linux/pisa.h
@@ -101,6 +101,7 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			     size_t size, size_t offset,
 			     enum dma_data_direction dir, bool pinned);
 bool pisa_page_tx_ok(const struct page *page);
+void pisa_stats_inc_user_copy(void);
 
 static inline bool pisa_enabled(void)
 {
@@ -287,6 +288,7 @@ static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
 }
 
 static inline bool pisa_page_tx_ok(const struct page *page) { return false; }
+static inline void pisa_stats_inc_user_copy(void) {}
 
 #endif /* CONFIG_PISA */
 
diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
index edfa7cf424451..f3449c0084e28 100644
--- a/mm/pisa-alloc.c
+++ b/mm/pisa-alloc.c
@@ -11,6 +11,7 @@
 #include <linux/cache.h>
 #include <linux/cc_platform.h>
 #include <linux/cpuhotplug.h>
+#include <linux/debugfs.h>
 #include <linux/device.h>
 #include <linux/dma-direct.h>
 #include <linux/dma-map-ops.h>
@@ -35,6 +36,7 @@
 #include <linux/rcupdate.h>
 #include <linux/sched/clock.h>
 #include <linux/sched/mm.h>
+#include <linux/seq_file.h>
 #include <linux/shrinker.h>
 #include <linux/skbuff.h>
 #include <linux/slab.h>
@@ -88,6 +90,9 @@ module_param_named(idle_ms, pisa_idle_ms, uint, 0644);
 unsigned int pisa_scrub_max __read_mostly;
 #define PISA_SCRUB_MAX_LIMIT	65536U
 
+/* Reclaim statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats pisa_reclaim_stats;
+
 /* All live allocators, protected by pisa_allocs_lock. */
 static LIST_HEAD(pisa_allocs);
 static DEFINE_RWLOCK(pisa_allocs_lock);
@@ -152,6 +157,9 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
 	struct pisa_meta *meta, *tmp;
 	struct llist_node *node;
 	struct pisa_alloc *pa;
+	unsigned int n = 0;
+
+	pisa_stat_inc(pisa_reclaim_stats.reclaim_run);
 
 	read_lock(&pisa_allocs_lock);
 	list_for_each_entry(pa, &pisa_allocs, node) {
@@ -178,8 +186,11 @@ static void pisa_reclaim_work_fn(struct work_struct *work)
 	read_unlock(&pisa_allocs_lock);
 
 	node = llist_del_all(&pisa_free_list);
-	llist_for_each_entry_safe(meta, tmp, node, llnode)
+	llist_for_each_entry_safe(meta, tmp, node, llnode) {
 		__pisa_release_page(meta);
+		n++;
+	}
+	pisa_stat_add(pisa_reclaim_stats.reclaim_pages, n);
 
 	if (next_delay != MAX_JIFFY_OFFSET)
 		queue_delayed_work(pisa_wq ?: system_wq, &pisa_reclaim_work, next_delay);
@@ -203,7 +214,8 @@ bool pisa_quarantine_page(struct pisa_meta *meta)
 	struct pisa_alloc *pa;
 	unsigned long flags;
 
-	set_bit(_PISA_QUARANTINED, &meta->flags);
+	if (!test_and_set_bit(_PISA_QUARANTINED, &meta->flags))
+		pisa_stat_atomic_inc(pisa_reclaim_stats.quarantined);
 	pa = READ_ONCE(meta->pa);
 	if (!pa || READ_ONCE(meta->on_list) == PISA_LIST_NONE)
 		return false;
@@ -361,6 +373,7 @@ static unsigned long pisa_alloc_scrub_blocks(struct pisa_alloc *pa)
 		__set_bit(idxs[idx], meta->clean_bitmap);
 	meta->nr_clean += count;
 	meta->nr_scrub = 0;
+	pisa_stat_add(pa->stats.block_scrub, count);
 	pa->scrub_demand -= min(count, pa->scrub_demand);
 	should_release = pisa_place_meta(meta);
 	spin_unlock_irqrestore(&pa->lock, flags);
@@ -484,6 +497,7 @@ static bool pisa_return_block_locked(struct pisa_alloc *pa, struct page *block)
 		meta->nr_dirty++;
 	}
 	pa->blocks_in_use--;
+	pisa_stat_inc(pa->stats.block_free);
 	return pisa_place_meta(meta);
 }
 
@@ -511,10 +525,19 @@ static void pisa_free_bulk(struct pisa_alloc *pa, unsigned int n, struct page **
  */
 #define PISA_PCP_BATCH	32U	/* blocks moved per refill or flush */
 
+struct pisa_pcp_stats {
+	u64	alloc;
+	u64	alloc_ks;
+	u64	alloc_as;
+	u64	miss;
+	u64	free;
+};
+
 struct pisa_pcp {
-	spinlock_t	lock;	/* by this CPU, or by a drain from any CPU */
-	unsigned int	count;
-	struct page	*blocks[PISA_PCP_HIGH];
+	spinlock_t		lock;	/* by this CPU, or by a drain from any CPU */
+	unsigned int		count;
+	struct pisa_pcp_stats	stats;
+	struct page		*blocks[PISA_PCP_HIGH];
 };
 
 /* Max blocks moved per per-CPU cache refill or flush (1..PISA_PCP_BATCH). */
@@ -537,7 +560,8 @@ static inline unsigned int pisa_pcp_batch_order(const struct pisa_alloc *pa)
 }
 
 /* Cache @n blocks at refcount 0 on this CPU, @n <= PISA_PCP_BATCH. */
-static void pisa_pcp_push(struct pisa_alloc *pa, unsigned int n, struct page **blocks)
+static void pisa_pcp_push(struct pisa_alloc *pa, unsigned int n, struct page **blocks,
+			  bool is_free)
 {
 	unsigned int batch = pisa_pcp_batch_order(pa);
 	unsigned int high = max(2U, PISA_PCP_HIGH >> pa->order);
@@ -547,6 +571,8 @@ static void pisa_pcp_push(struct pisa_alloc *pa, unsigned int n, struct page **b
 	/* @pcp->lock protects the cache: migrating to another CPU here is harmless. */
 	pcp = raw_cpu_ptr(pa->pcp);
 	spin_lock_irqsave(&pcp->lock, flags);
+	if (is_free)
+		pisa_stat_add(pcp->stats.free, n);
 	for (unsigned int i = 0; i < n; i++) {
 		if (pcp->count >= high) {
 			/* Keep the most recently freed, likely still in cache. */
@@ -617,17 +643,29 @@ void pisa_pcp_drain(struct pisa_alloc *pa)
 		pisa_pcp_drain_cpu(pa, cpu);
 }
 
-/* Blocks in @pa's caches, for shrinker and debugfs. */
-static unsigned long pisa_pcp_count(struct pisa_alloc *pa)
+/* Return cached block count across @pa's per-CPU caches, and sum counters into @st. */
+static unsigned long pisa_pcp_stats(struct pisa_alloc *pa, struct pisa_pcp_stats *st)
 {
+	struct pisa_pcp_stats tmp;
 	unsigned long n = 0;
 	int cpu;
 
+	if (!st)
+		st = &tmp;
+	memset(st, 0, sizeof(*st));
 	if (!pa->pcp)
 		return 0;
+
 	for_each_possible_cpu(cpu) {
+		const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+
 		/* A racy sum is good enough for reclaim and statistics. */
-		n += data_race(per_cpu_ptr(pa->pcp, cpu)->count);
+		n += READ_ONCE(pcp->count);
+		st->alloc += READ_ONCE(pcp->stats.alloc);
+		st->alloc_ks += READ_ONCE(pcp->stats.alloc_ks);
+		st->alloc_as += READ_ONCE(pcp->stats.alloc_as);
+		st->miss += READ_ONCE(pcp->stats.miss);
+		st->free += READ_ONCE(pcp->stats.free);
 	}
 	return n;
 }
@@ -900,6 +938,7 @@ static unsigned int pisa_scan_bitmap(struct pisa_meta *meta, bool dirty,
 	}
 	*countp -= used;
 	pa->blocks_in_use += used;
+	pisa_stat_add(pa->stats.block_alloc, used);
 	return got;
 }
 
@@ -948,7 +987,8 @@ static struct pisa_meta *pisa_pick_meta(struct pisa_alloc *pa, bool zero)
 
 /* Extract up to @want blocks from @meta under @pa->lock. */
 static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
-				struct page **out, bool zero, bool frozen)
+				struct page **out, bool can_block, bool zero,
+				bool frozen)
 {
 	unsigned int avail_before = pisa_meta_avail(meta), got = 0, used;
 	struct pisa_alloc *pa = meta->pa;
@@ -963,12 +1003,23 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
 		dirty = pisa_scan_bitmap(meta, true, want - got, out + got, false);
 		for (unsigned int i = got; i < got + dirty; i++)
 			set_page_private(out[i], 1);
+		if (can_block) {
+			pisa_stat_add(pa->stats.alloc_kc, got);
+			pisa_stat_add(pa->stats.alloc_ks, dirty);
+		} else {
+			pisa_stat_add(pa->stats.alloc_ac, got);
+			pisa_stat_add(pa->stats.alloc_as, dirty);
+		}
 		got += dirty;
 		pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
 	} else {
 		/* First choice are dirty blocks, followed by clean */
 		got += pisa_scan_bitmap(meta, true, want, out, !frozen);
 		got += pisa_scan_bitmap(meta, false, want - got, out + got, !frozen);
+		if (can_block)
+			pisa_stat_add(pa->stats.alloc_kd, got);
+		else
+			pisa_stat_add(pa->stats.alloc_ad, got);
 	}
 
 	used = avail_before - pisa_meta_avail(meta);
@@ -980,14 +1031,35 @@ static unsigned int take_blocks(struct pisa_meta *meta, unsigned int want,
 	if (READ_ONCE(meta->drain))
 		WRITE_ONCE(meta->drain, false);
 	pisa_place_meta(meta);
+	if (page_to_nid(pisa_meta_to_page(meta)) != pa->nid)
+		pisa_stat_add(pa->stats.numa_mismatch, used);
 
 	return got;
 }
 
+/* Count a failed pisa_alloc_split_page(); @none if the caller got no block at all. */
+static void pisa_alloc_count_failure(struct pisa_alloc *pa, long err, bool none)
+{
+	if (err == -EAGAIN) {
+		pisa_stat_atomic_inc(pa->stats.fail_backoff);
+	} else if (err == -ENOSPC) {
+		pisa_stat_atomic_inc(pa->stats.fail_budget);
+	} else if (err == -ENOMEM) {
+		pisa_stat_atomic_inc(pa->stats.fail_nomem);
+		pr_warn_once("pisa: order-%u allocation failed for allocator %p; further failures are counted in debugfs only\n",
+			     PISA_ORDER, pa);
+	} else if (err == -EBUSY) {
+		pisa_stat_atomic_inc(pa->stats.fail_split);
+	}
+	if (none)
+		pisa_stat_atomic_inc(pa->stats.block_alloc_fail);
+}
+
 /* Allocate @nr_pages blocks from @pa. */
 static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigned long nr_pages,
 				       struct page **pages, bool zero, bool frozen)
 {
+	bool can_block = gfpflags_allow_blocking(gfp);
 	bool should_scrub = false;
 	unsigned long n, flags;
 	struct pisa_meta *meta;
@@ -995,7 +1067,8 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 
 	/* Held across bitmap scans; dropped around slow-path pisa_alloc_split_page(). */
 	spin_lock_irqsave(&pa->lock, flags);
-	for (n = 0; n < nr_pages; n += take_blocks(meta, want, &pages[n], zero, frozen)) {
+	for (n = 0; n < nr_pages;
+	     n += take_blocks(meta, want, &pages[n], can_block, zero, frozen)) {
 		want = min_t(unsigned long, nr_pages - n, PISA_BLOCKS(pa->order));
 		meta = pisa_pick_meta(pa, zero);
 		if (meta)
@@ -1006,15 +1079,17 @@ static unsigned long __pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp, unsigne
 		spin_unlock_irqrestore(&pa->lock, flags);
 		/* With @zero, fresh dirty blocks are zeroed below. */
 		meta = pisa_alloc_split_page(pa->order, gfp, pa->nid);
-		if (IS_ERR(meta))
+		if (IS_ERR(meta)) {
+			pisa_alloc_count_failure(pa, PTR_ERR(meta), !n);
 			goto out;
+		}
 		spin_lock_irqsave(&pa->lock, flags);
 		meta->pa = pa;
 		meta->prot = pa->prot;
 		WRITE_ONCE(meta->alloc_jiffies, jiffies);
 		/* Set PISA_ACTIVE after @meta is fully set up. */
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
-		pa->stats.page_alloc++;
+		pisa_stat_inc(pa->stats.page_alloc);
 	}
 	/* Queue the scrubber when the inline zeroing crosses the trigger. */
 	should_scrub |= pa->scrub_demand >= PISA_SCRUB_TRIGGER;
@@ -1042,9 +1117,11 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
 				    struct page **page_array, bool frozen)
 {
 	unsigned int pcp_batch = pisa_pcp_batch_order(pa);
-	unsigned long got, j = 0;
+	bool zero = want_init_on_alloc(gfp);
+	unsigned long got, local, j = 0;
 
 	got = pisa_pcp_pop(pa, nr_pages, page_array);
+	local = got;
 	if (got < nr_pages) {
 		if (nr_pages - got >= pcp_batch) {
 			got += __pisa_alloc_bulk(pa, gfp, nr_pages - got,
@@ -1058,7 +1135,7 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
 				page_array[got++] = batch[i];
 			if (i < n) {
 				if (pa->nid == numa_mem_id())
-					pisa_pcp_push(pa, n - i, batch + i);
+					pisa_pcp_push(pa, n - i, batch + i, false);
 				else
 					pisa_free_bulk(pa, n - i, batch + i);
 			}
@@ -1070,10 +1147,13 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
 
 		page_array[i] = NULL;
 		/* Poisoned while cached: leak it, as pisa_scan_bitmap() does. */
-		if (is_pisa_block_poisoned(pa, block))
+		if (is_pisa_block_poisoned(pa, block)) {
+			if (i < local)
+				local--;
 			continue;
+		}
 		pisa_block_prep(block);
-		if (want_init_on_alloc(gfp)) {
+		if (zero) {
 			memset(page_address(block), 0, PAGE_SIZE << pa->order);
 			if (frozen) /* Order zeroing before caller publishes the page. */
 				smp_wmb();
@@ -1084,8 +1164,16 @@ static unsigned long pisa_pcp_alloc(struct pisa_alloc *pa, gfp_t gfp, unsigned l
 	}
 	/* Replace blocks dropped as poisoned, so that callers do not see a short allocation. */
 	if (unlikely(j < got))
-		j += __pisa_alloc_bulk(pa, gfp, nr_pages - j, page_array + j,
-				       want_init_on_alloc(gfp), frozen);
+		j += __pisa_alloc_bulk(pa, gfp, nr_pages - j, page_array + j, zero, frozen);
+	this_cpu_add(pa->pcp->stats.alloc, j);
+	if (zero) {
+		if (gfpflags_allow_blocking(gfp))
+			this_cpu_add(pa->pcp->stats.alloc_ks, j);
+		else
+			this_cpu_add(pa->pcp->stats.alloc_as, j);
+	}
+	if (j > local)
+		this_cpu_inc(pa->pcp->stats.miss);
 	return j;
 }
 
@@ -1108,9 +1196,13 @@ unsigned long pisa_alloc_bulk(struct pisa_alloc *pa, gfp_t gfp,
 	/* Skip any already populated prefix, as alloc_pages_bulk_node() does. */
 	while (i < nr_pages && page_array[i])
 		i++;
-	if (unlikely(!pa || i == nr_pages ||
-		     (gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))))
+	if (unlikely(!pa || i == nr_pages))
+		return i;
+
+	if (unlikely(gfp & (__GFP_DMA | __GFP_DMA32 | __GFP_THISNODE | __GFP_ACCOUNT))) {
+		pisa_stat_atomic_inc(pa->stats.fail_gfp);
 		return i;
+	}
 	gfp = current_gfp_context(gfp);
 
 	while (i < nr_pages) {
@@ -1211,6 +1303,11 @@ static struct page *pisa_take_contig(struct pisa_meta *meta, unsigned int nblock
 		page_ref_unfreeze(page + i, 1);
 	}
 	pa->blocks_in_use += nblocks;
+	pisa_stat_add(pa->stats.block_alloc, nblocks);
+	pisa_stat_add(pa->stats.alloc_kc, nblocks - dirty);
+	pisa_stat_add(pa->stats.alloc_ks, dirty);
+	if (page_to_nid(pisa_meta_to_page(meta)) != pa->nid)
+		pisa_stat_add(pa->stats.numa_mismatch, nblocks);
 	pa->scrub_demand = min(pa->scrub_demand + dirty, PISA_SCRUB_DEMAND_MAX);
 	*clean = !dirty;
 	pisa_place_meta(meta);
@@ -1298,6 +1395,7 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 		spin_unlock_irqrestore(&pa->lock, flags);
 		meta = pisa_alloc_split_page(0, gfp, pa->nid);
 		if (IS_ERR(meta)) {
+			pisa_alloc_count_failure(pa, PTR_ERR(meta), true);
 			if (should_scrub)
 				pisa_schedule_scrub(pa->nid);
 			return NULL;
@@ -1307,7 +1405,7 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 		meta->prot = pa->prot;
 		WRITE_ONCE(meta->alloc_jiffies, jiffies);
 		pisa_meta_set_flags(meta, PISA_ACTIVE);
-		pa->stats.page_alloc++;
+		pisa_stat_inc(pa->stats.page_alloc);
 		page = pisa_take_contig(meta, nblocks, align_mask, &clean);
 		if (unlikely(!page))
 			should_release = pisa_place_meta(meta);
@@ -1337,6 +1435,11 @@ __pisa_alloc_map_coherent(struct device *dev, struct iommu_domain *domain,
 	} else {
 		*dma_handle = phys_to_dma(dev, page_to_phys(page));
 	}
+	/* Pairs with cmpxchg_release() in pisa_enable_pcp(). */
+	if (smp_load_acquire(&pa->pcp)) {
+		this_cpu_add(pa->pcp->stats.alloc, nblocks);
+		this_cpu_inc(pa->pcp->stats.miss);
+	}
 	if (!clean)
 		memset(page_address(page), 0, size);
 	return page_address(page);
@@ -1390,9 +1493,10 @@ bool __pisa_free_page(struct page *page, unsigned int order)
 	if (smp_load_acquire(&pa->pcp) && pa->nid == numa_mem_id() &&
 	    page_to_nid(page) == numa_mem_id()) {
 		if (pisa_page_can_recycle(page)) {
-			pisa_pcp_push(pa, 1, &page);
+			pisa_pcp_push(pa, 1, &page, true);
 			return true;
 		}
+		this_cpu_inc(pa->pcp->stats.free);
 		/* A compaction target drains the caches once, when it is done. */
 		if (!(READ_ONCE(meta->on_list) & PISA_LIST_HOLD))
 			pisa_pcp_drain_cpu(pa, raw_smp_processor_id());
@@ -1417,7 +1521,7 @@ static unsigned long pisa_shrink_count(struct shrinker *shrink, struct shrink_co
 	guard(read_lock)(&pisa_allocs_lock);
 	list_for_each_entry(pa, &pisa_allocs, node) {
 		nr += (unsigned long)READ_ONCE(pa->num_idle_pages) << PISA_ORDER;
-		nr += pisa_pcp_count(pa) << pa->order;
+		nr += pisa_pcp_stats(pa, NULL) << pa->order;
 	}
 
 	return nr;
@@ -1447,11 +1551,14 @@ unsigned long pisa_shrink_list(spinlock_t *lock, struct list_head *list,
 VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct pisa_meta *meta, *tmp;
+	unsigned long freed, res_freed;
 	struct pisa_alloc *pa;
-	unsigned long freed;
 	LIST_HEAD(release_list);
 
 	freed = pisa_reservoir_shrink(&release_list, sc->nr_to_scan);
+	res_freed = freed;
+	pisa_stat_atomic_inc(pisa_reclaim_stats.shrink_scan);
+	pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_res, res_freed);
 
 	if (freed < sc->nr_to_scan) {
 		read_lock(&pisa_allocs_lock);
@@ -1466,6 +1573,7 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
 				break;
 		}
 		read_unlock(&pisa_allocs_lock);
+		pisa_stat_atomic_add(pisa_reclaim_stats.pte_pages_idle, freed - res_freed);
 	}
 
 	/* Retiring must not run under @pa->lock. */
@@ -1480,6 +1588,114 @@ VISIBLE_IF_KUNIT unsigned long pisa_shrink_scan(struct shrinker *shrink, struct
 	return freed ?: SHRINK_STOP;
 }
 
+/* See pisa_seq_kv(). */
+void __pisa_seq_kv(struct seq_file *s, const char *title, const struct pisa_kv *st, unsigned int n)
+{
+	seq_printf(s, "  %-12s", title);
+	for (; n; n--, st++)
+		seq_printf(s, " %-10s%11lld", st->name, st->val);
+	seq_putc(s, '\n');
+}
+
+/* Output global and active per-allocator statistics in /sys/kernel/debug/pisa/allocs. */
+static int pisa_allocs_show(struct seq_file *s, void *unused)
+{
+	struct pisa_reclaim_stats *rs = &pisa_reclaim_stats;
+	char no_pages[16], *pages = no_pages + 3;
+	struct pisa_alloc *pa;
+
+	snprintf(no_pages, sizeof(no_pages), "no_pages_%luM", PISA_SIZE >> 20);
+
+	seq_puts(s, "global (allocators with no activity are omitted)\n");
+	pisa_seq_kv(s, pages, { "in_use", atomic_long_read(&pisa_nr_pages) },
+		    { "max", pisa_max_pages },
+		    { "slots", pisa_nr_slots });
+	pisa_domains_show(s);
+	pisa_seq_kv(s, "reservoir", { pages, pisa_reservoir_count() },
+		    { "max_node", pisa_reservoir_max });
+	pisa_seq_kv(s, "shrinker",
+		    { "scan", pisa_stat_atomic_read(rs->shrink_scan) },
+		    { "from_res", pisa_stat_atomic_read(rs->pte_pages_res) },
+		    { "from_idle", pisa_stat_atomic_read(rs->pte_pages_idle) });
+	pisa_seq_kv(s, "reclaim",
+		    { "runs", pisa_stat_read(rs->reclaim_run) },
+		    { pages, pisa_stat_read(rs->reclaim_pages) },
+		    { "quarantined", pisa_stat_atomic_read(rs->quarantined) });
+	pisa_reservoir_show(s);
+	pisa_user_show(s);
+
+	guard(read_lock)(&pisa_allocs_lock);
+	list_for_each_entry(pa, &pisa_allocs, node) {
+		u64 pages_in_use, blocks_in_use, used_pct = 0, cached;
+		struct pisa_alloc_stats st = data_race(pa->stats); /* debugfs snapshot */
+		struct pisa_pcp_stats pcp;
+
+		cached = pisa_pcp_stats(pa, &pcp);
+		/* Omit allocators that have never allocated or failed an allocation. */
+		if (!st.page_alloc && !st.block_alloc &&
+		    !pisa_stat_atomic_read(st.block_alloc_fail) &&
+		    !pisa_stat_atomic_read(st.fail_gfp))
+			continue;
+
+		u64 page_free = atomic64_read(&st.page_free);
+
+		pages_in_use = st.page_alloc > page_free ?
+			       st.page_alloc - page_free : 0;
+		blocks_in_use = st.block_alloc > st.block_free + cached ?
+				st.block_alloc - st.block_free - cached : 0;
+		if (pages_in_use)
+			used_pct = div64_u64((blocks_in_use << pa->order) * 100,
+					     pages_in_use << PISA_ORDER);
+
+		seq_printf(s, "alloc %s order %u nid %d%s\n", pa->name, pa->order,
+			   pa->nid, pa->prot == IOMMU_READ ? " tx-only" :
+			   pa->prot == IOMMU_WRITE ? " rx-only" : "");
+		pisa_seq_kv(s, pages, { "got", st.page_alloc },
+			    { "put", page_free }, { "in_use", pages_in_use },
+			    { "idle", READ_ONCE(pa->num_idle_pages) },
+			    { "idle_max", READ_ONCE(pa->max_idle_pages) });
+		pisa_seq_kv(s, no_pages,
+			    { "backoff", pisa_stat_atomic_read(st.fail_backoff) },
+			    { "budget", pisa_stat_atomic_read(st.fail_budget) },
+			    { "nomem", pisa_stat_atomic_read(st.fail_nomem) },
+			    { "split", pisa_stat_atomic_read(st.fail_split) });
+		pisa_seq_kv(s, "blocks", { "alloc", st.block_alloc },
+			    { "free", st.block_free }, { "in_use", blocks_in_use },
+			    { "used_pct", used_pct }, { "scrub", st.block_scrub });
+		pisa_seq_kv(s, "blocks",
+			    { "failed", pisa_stat_atomic_read(st.block_alloc_fail) },
+			    { "gfp", pisa_stat_atomic_read(st.fail_gfp) },
+			    { "numa_miss", st.numa_mismatch });
+		pisa_seq_kv(s, "allocations",
+			    { "KD", st.alloc_kd }, { "KC", st.alloc_kc }, { "KS", st.alloc_ks },
+			    { "AD", st.alloc_ad }, { "AC", st.alloc_ac }, { "AS", st.alloc_as });
+		if (pa->pcp) {
+			int cpu;
+
+			pisa_seq_kv(s, "pcpu", { "alloc", pcp.alloc },
+				    { "miss", pcp.miss },
+				    { "free", pcp.free }, { "cached", cached },
+				    { "KS", pcp.alloc_ks }, { "AS", pcp.alloc_as });
+			for_each_possible_cpu(cpu) {
+				const struct pisa_pcp *pcp = per_cpu_ptr(pa->pcp, cpu);
+				struct pisa_pcp_stats stats = data_race(pcp->stats); /* debugfs snapshot */
+				unsigned int c = READ_ONCE(pcp->count);
+				char title[16];
+
+				if (!stats.alloc && !stats.free && !c)
+					continue;
+				snprintf(title, sizeof(title), "pcpu_%u", cpu);
+				pisa_seq_kv(s, title, { "alloc", stats.alloc },
+					    { "miss", stats.miss },
+					    { "free", stats.free }, { "cached", c });
+			}
+		}
+	}
+
+	return 0;
+}
+DEFINE_SHOW_ATTRIBUTE(pisa_allocs);
+
 /* Return the blocks a dead CPU cached, which nothing else would. */
 static int pisa_cpu_dead(unsigned int cpu)
 {
@@ -1522,6 +1738,10 @@ static int __init pisa_init(void)
 				      pisa_cpu_dead) < 0)
 		pr_warn("pisa: no CPU hotplug callback, offline CPUs keep cached blocks\n");
 
+	/* Built-in and never unloaded; debugfs stubs are no-ops when disabled. */
+	debugfs_create_file("allocs", 0444, debugfs_create_dir("pisa", NULL),
+			    NULL, &pisa_allocs_fops);
+
 	shrink = shrinker_alloc(0, "pisa");
 	if (!shrink) {
 		pr_warn("pisa: shrinker registration failed\n");
diff --git a/mm/pisa-kunit.c b/mm/pisa-kunit.c
index c195fdfdbd9fa..98f06a7f2db7f 100644
--- a/mm/pisa-kunit.c
+++ b/mm/pisa-kunit.c
@@ -389,9 +389,9 @@ static int fake_domain_setup(struct kunit *test)
 
 static void test_map_and_domain_lifecycle(struct kunit *test)
 {
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
 	struct device dev = { .init_name = "pisa_map_kunit", .numa_node = NUMA_NO_NODE,
 			      .coherent_dma_mask = DMA_BIT_MASK(64) };
-	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
 	struct pisa_iova *pi;
 	struct pisa_meta *meta;
 	struct pisa_alloc *pa;
@@ -676,9 +676,9 @@ static void test_gfp_pisa(struct kunit *test)
 
 static void test_reservoir_and_scrubber(struct kunit *test)
 {
+	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
 	unsigned int old_res = READ_ONCE(pisa_reservoir_max);
 	unsigned int old_scrub = READ_ONCE(pisa_scrub_max);
-	struct shrink_control sc = { .nr_to_scan = ULONG_MAX };
 	struct pisa_meta *meta;
 	struct pisa_alloc *pa;
 	struct page *p;
@@ -711,9 +711,9 @@ static void test_reservoir_and_scrubber(struct kunit *test)
 			WRITE_ONCE(pisa_scrub_max, 512);
 			pisa_scrub_node(pa->nid);
 			KUNIT_EXPECT_EQ(test, meta->nr_dirty, 0U);
+			KUNIT_EXPECT_GE(test, pa->stats.block_scrub, 1ULL);
 		}
 	}
-	pisa_shrink_scan(NULL, &sc);
 
 	/* Trim reservoirs back to 0. */
 	for_each_node_state(nid, N_MEMORY)
diff --git a/mm/pisa-node.c b/mm/pisa-node.c
index aa4f8759ae5e8..e1ae834d0f9b6 100644
--- a/mm/pisa-node.c
+++ b/mm/pisa-node.c
@@ -19,6 +19,7 @@
 #include <linux/rcupdate.h>
 #include <linux/sched.h>
 #include <linux/sched/mm.h>
+#include <linux/seq_file.h>
 #include <linux/set_memory.h>
 #include <linux/spinlock.h>
 #include <linux/topology.h>
@@ -228,6 +229,7 @@ static void pisa_release_page_rcu(struct rcu_head *head)
 	/* Still shared with the host, or possibly still mapped by a device. */
 	if (flags & (PISA_DECRYPTED | PISA_QUARANTINED))
 		return;
+
 	atomic_long_dec(&pisa_nr_pages);
 	for (unsigned int i = 0; i < PISA_BLOCKS(order); i++) {
 		struct page *block = page + (i << order);
@@ -296,6 +298,7 @@ static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
 		meta = __pisa_page_alloc(GFP_KERNEL | __GFP_THISNODE, nid, true,
 					 pisa_want_decrypt(), &clean);
 		if (IS_ERR(meta)) {
+			pisa_stat_inc(node->stats.refill_err);
 			pisa_alloc_backoff(node, false);
 			break;
 		}
@@ -310,6 +313,7 @@ static void pisa_reservoir_refill_node(struct pisa_node *node, int nid)
 		if (added) {
 			list_add(&meta->list, &node->pages);
 			WRITE_ONCE(node->nr_pages, node->nr_pages + 1);
+			pisa_stat_inc(node->stats.refill);
 		}
 		spin_unlock_irqrestore(&node->lock, flags);
 		if (!added) {
@@ -334,6 +338,7 @@ static void pisa_reservoir_trim_node(struct pisa_node *node, unsigned int max)
 		if (meta) {
 			list_del_init(&meta->list);
 			WRITE_ONCE(node->nr_pages, node->nr_pages - 1);
+			pisa_stat_inc(node->stats.trim);
 		}
 		spin_unlock_irqrestore(&node->lock, flags);
 		if (!meta)
@@ -434,6 +439,7 @@ static struct pisa_meta *pisa_reservoir_pop_node(int nid, unsigned int max, bool
 	if (meta && !!(meta->flags & PISA_DECRYPTED) == decrypt) {
 		list_del_init(&meta->list);
 		WRITE_ONCE(node->nr_pages, node->nr_pages - 1);
+		pisa_stat_inc(node->stats.hit);
 		refill = node->nr_pages <= max / 2;
 	} else {
 		meta = NULL;
@@ -462,6 +468,7 @@ struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *cle
 			*clean = true;
 			return meta;
 		}
+		pisa_stat_atomic_inc(pisa_nodes[nid].stats.miss);
 	}
 
 	/* Back off briefly on GFP_ATOMIC if previous allocation on @nid failed. */
@@ -525,3 +532,36 @@ void __init pisa_node_init(void)
 		INIT_WORK(&pisa_nodes[nid].work, pisa_node_work_fn);
 	}
 }
+
+/* Print the reservoir of each node that has seen any activity. */
+void pisa_reservoir_show(struct seq_file *s)
+{
+	bool header = false;
+	char pages[16];
+	int nid;
+
+	for_each_node_state(nid, N_MEMORY) {
+		struct pisa_node *node = &pisa_nodes[nid];
+		unsigned int nr = READ_ONCE(node->nr_pages);
+		s64 refill = pisa_stat_read(node->stats.refill);
+		s64 refill_err = pisa_stat_read(node->stats.refill_err);
+		s64 miss = pisa_stat_atomic_read(node->stats.miss);
+		char title[16];
+
+		/* Pages served or trimmed were refilled first. */
+		if (!nr && !refill && !refill_err && !miss)
+			continue;
+		if (!header) {
+			snprintf(pages, sizeof(pages), "pages_%luM", PISA_SIZE >> 20);
+			seq_puts(s, "reservoir\n");
+			header = true;
+		}
+		snprintf(title, sizeof(title), "node_%d", nid);
+		pisa_seq_kv(s, title, { pages, nr },
+			    { "hit", pisa_stat_read(node->stats.hit) },
+			    { "miss", miss });
+		pisa_seq_kv(s, title, { "refill", refill },
+			    { "refill_err", refill_err },
+			    { "trim", pisa_stat_read(node->stats.trim) });
+	}
+}
diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
index 865f3d779b84d..dc073a36cbc83 100644
--- a/mm/pisa-priv.h
+++ b/mm/pisa-priv.h
@@ -91,10 +91,38 @@ static inline unsigned int pisa_meta_avail(const struct pisa_meta *meta)
 	return meta->nr_clean + meta->nr_dirty;
 }
 
+#define pisa_stat_add(var, val)		((var) += (val))
+#define pisa_stat_inc(var)		pisa_stat_add(var, 1)
+#define pisa_stat_dec(var)		pisa_stat_add(var, -1)
+#define pisa_stat_read(var)		READ_ONCE(var)
+
+#define pisa_stat_atomic_add(var, val)	atomic64_add(val, &(var))
+#define pisa_stat_atomic_inc(var)	atomic64_inc(&(var))
+#define pisa_stat_atomic_read(var)	atomic64_read(&(var))
+
 /* Per-allocator statistics, see pisa_allocs_show(). */
 struct pisa_alloc_stats {
-	u64		page_alloc;
-	atomic64_t	page_free;
+	/* Updated under @pa->lock. */
+	u64		page_alloc;	/* PISA pages allocated */
+	u64		block_alloc;	/* block allocations satisfied */
+	u64		block_free;	/* block frees recycled */
+	u64		block_scrub;	/* dirty blocks zeroed by scrubber */
+	u64		numa_mismatch;	/* blocks from non-target NUMA node */
+	u64		alloc_kd;	/* sleepable non-zeroing */
+	u64		alloc_kc;	/* sleepable zeroing from clean_bitmap */
+	u64		alloc_ks;	/* sleepable zeroing inline */
+	u64		alloc_ad;	/* atomic non-zeroing */
+	u64		alloc_ac;	/* atomic zeroing from clean_bitmap */
+	u64		alloc_as;	/* atomic zeroing inline */
+
+	/* Updated without @pa->lock (e.g. pisa_alloc_split_page()). */
+	atomic64_t	page_free;		/* PISA pages released to buddy */
+	atomic64_t	fail_backoff;		/* skipped due to atomic backoff */
+	atomic64_t	fail_budget;		/* refused by pisa_max_pages */
+	atomic64_t	fail_nomem;		/* buddy allocation failed */
+	atomic64_t	fail_split;		/* split_page_compound() failed */
+	atomic64_t	block_alloc_fail;	/* failed on empty allocator */
+	atomic64_t	fail_gfp;		/* declined unsupported GFP */
 };
 
 /* Allocator managing blocks of a fixed order, carved from PISA pages. */
@@ -144,8 +172,30 @@ struct pisa_node {
 	struct list_head	pages;		/* reservoir */
 	unsigned long		flags;		/* PISA_NODE_* requests */
 	struct work_struct	work;		/* refill/trim/scrub, on this node */
+	struct {
+		u64		hit;		/* requests for this node it served */
+		u64		refill;		/* PISA pages added by the refill */
+		u64		refill_err;	/* refill allocations that failed */
+		u64		trim;		/* PISA pages released by the trim */
+		atomic64_t	miss;		/* requests that found it empty */
+	} stats;
 } ____cacheline_aligned_in_smp;
 
+/* Global reclaim and shrinker statistics, see pisa_allocs_show(). */
+struct pisa_reclaim_stats {
+	/* Updated by singleton pisa_reclaim_work. */
+	u64		reclaim_run;	/* pisa_reclaim_work_fn() runs */
+	u64		reclaim_pages;	/* PISA pages it released */
+
+	/* Updated locklessly. */
+	atomic64_t	shrink_scan;	/* pisa_shrink_scan() calls */
+	atomic64_t	pte_pages_res;	/* PTE pages taken from the reservoirs */
+	atomic64_t	pte_pages_idle;	/* PTE pages taken from idle allocators */
+	atomic64_t	quarantined;	/* PISA pages quarantined */
+};
+
+extern struct pisa_reclaim_stats pisa_reclaim_stats;
+
 /* Per-NUMA-node PISA state (ring/TX/user allocators, reservoir). */
 extern struct pisa_node pisa_nodes[MAX_NUMNODES];
 
@@ -260,6 +310,9 @@ void pisa_schedule_scrub_all(void);
 struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool decrypt, bool *clean);
 unsigned long pisa_reservoir_count(void);
 unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
+
+void pisa_reservoir_show(struct seq_file *s);
+void pisa_user_show(struct seq_file *s);
 void pisa_user_boot_enable(void);
 
 #endif /* CONFIG_PISA */
diff --git a/mm/pisa-user.c b/mm/pisa-user.c
index e4a4e1a606b24..5020269476845 100644
--- a/mm/pisa-user.c
+++ b/mm/pisa-user.c
@@ -19,6 +19,7 @@
 #include <linux/pagewalk.h>
 #include <linux/folio_batch.h>
 #include <linux/sched/clock.h>
+#include <linux/seq_file.h>
 #include <linux/swap.h>
 #include <linux/uio.h>
 
@@ -49,6 +50,26 @@ static const struct kernel_param_ops pisa_scrub_migrate_ops = {
 };
 module_param_cb(scrub_migrate, &pisa_scrub_migrate_ops, &pisa_scrub_migrate, 0644);
 
+/* Statistics, see pisa_user_show(). */
+struct pisa_user_stats {
+	atomic64_t	in;		/* pages migrated into PISA blocks */
+	atomic64_t	in_fail;	/* ... isolated but not migrated */
+	atomic64_t	out;		/* pages migrated back to ordinary pages */
+	atomic64_t	out_fail;	/* ... isolated but not migrated */
+	atomic64_t	no_budget;	/* pages not moved in: cap reached */
+	atomic64_t	busy;		/* pages skipped: referenced elsewhere */
+	atomic64_t	large;		/* pages skipped: large anon folio not split */
+	atomic64_t	copy;		/* TCP chunks copied: page not in PISA */
+};
+
+static struct pisa_user_stats pisa_user_stats;
+
+/* Called by zerocopy_fill_skb_from_iter() when it asks TCP to copy. */
+void pisa_stats_inc_user_copy(void)
+{
+	pisa_stat_atomic_inc(pisa_user_stats.copy);
+}
+
 /* Create per-node user allocators and enable pisa_user_key (serialized by param_lock/init). */
 static int pisa_user_init(void)
 {
@@ -269,7 +290,8 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
 		.same_alloc = same_alloc,
 	};
 	enum migrate_mode mode = MIGRATE_SYNC;
-	unsigned int done = 0;
+	unsigned int nr = 0, done = 0;
+	struct folio *folio;
 
 	if (list_empty(list))
 		return 0;
@@ -278,10 +300,19 @@ static unsigned int pisa_user_migrate_dir(struct list_head *list, bool to_pisa,
 		if (!gfpflags_allow_blocking(gfp))
 			mode = MIGRATE_ASYNC;
 	}
+	list_for_each_entry(folio, list, lru)
+		nr += folio_nr_pages(folio);
 	if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
 			  to_pisa ? pisa_user_put_folio : NULL, (unsigned long)&mtc,
 			  mode, MR_SYSCALL, &done))
 		putback_movable_pages(list);
+	if (to_pisa) {
+		pisa_stat_atomic_add(pisa_user_stats.in, done);
+		pisa_stat_atomic_add(pisa_user_stats.in_fail, nr - done);
+	} else {
+		pisa_stat_atomic_add(pisa_user_stats.out, done);
+		pisa_stat_atomic_add(pisa_user_stats.out_fail, nr - done);
+	}
 	return done;
 }
 
@@ -313,7 +344,7 @@ static unsigned int pisa_user_split_isolated(struct folio *folio, struct list_he
 					     struct page **pages, unsigned int nr, bool split)
 {
 	const struct page *lo = NULL, *hi = NULL;
-	unsigned int kept = 0;
+	unsigned int kept = 0, skipped = 0;
 	struct folio *f, *next;
 	LIST_HEAD(pieces);
 
@@ -325,9 +356,14 @@ static unsigned int pisa_user_split_isolated(struct folio *folio, struct list_he
 			lo = pages[i];
 		if (!hi || pages[i] > hi)
 			hi = pages[i];
+		skipped++;
 	}
 	list_move(&folio->lru, &pieces);
-	if (split && pisa_user_split(folio, &pieces, false)) {
+	if (!split) {
+		pisa_stat_atomic_add(pisa_user_stats.no_budget, skipped);
+	} else if (!pisa_user_split(folio, &pieces, false)) {
+		pisa_stat_atomic_add(pisa_user_stats.large, skipped);
+	} else {
 		list_for_each_entry_safe(f, next, &pieces, lru) {
 			if (!folio_test_large(f) && &f->page >= lo && &f->page <= hi) {
 				list_move_tail(&f->lru, out);
@@ -380,6 +416,7 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 	if (!want)
 		return false;
 	if (need_budget && pisa_user_budget() <= 0) {
+		pisa_stat_atomic_add(pisa_user_stats.no_budget, need_budget);
 		if (want == need_budget)
 			return false;
 		no_budget = true;
@@ -402,8 +439,10 @@ bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
 			continue;
 		for (unsigned int j = 0; j < nr; j++)
 			refs += page_folio(pages[j]) == folio;
-		if (pisa_user_batch_busy(folio, refs, pinned))
+		if (pisa_user_batch_busy(folio, refs, pinned)) {
+			pisa_stat_atomic_inc(pisa_user_stats.busy);
 			continue;
+		}
 		if (isolate_folio_to_list(folio, &list))
 			list_move_tail(&folio->lru, &list);
 	}
@@ -789,6 +828,9 @@ static int pisa_madvise_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long
 				/* Look at @addr again: it now maps an order-0 folio. */
 				if (split)
 					continue;
+				pisa_stat_atomic_add(pisa_user_stats.large, nr);
+			} else if (busy) {
+				pisa_stat_atomic_add(pisa_user_stats.busy, nr);
 			}
 		}
 		if (w->n >= PISA_USER_BATCH) {
@@ -833,3 +875,24 @@ int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long
 	pisa_user_migrate_dir(&w.list, to_pisa, DMA_TO_DEVICE, false, PISA_MADVISE_GFP);
 	return ret < 0 ? ret : 0;
 }
+
+/* Print user page state and migration statistics to debugfs seq_file @s. */
+void pisa_user_show(struct seq_file *s)
+{
+	char pages[16];
+
+	snprintf(pages, sizeof(pages), "pages_%luK", PAGE_SIZE >> 10);
+	seq_puts(s, "user\n");
+	pisa_seq_kv(s, pages, { "in_use", pisa_user_in_use(true) },
+		    { "max", pisa_user_max_pages << PISA_ORDER });
+	pisa_seq_kv(s, "migrate",
+		    { "in", pisa_stat_atomic_read(pisa_user_stats.in) },
+		    { "in_fail", pisa_stat_atomic_read(pisa_user_stats.in_fail) },
+		    { "out", pisa_stat_atomic_read(pisa_user_stats.out) },
+		    { "out_fail", pisa_stat_atomic_read(pisa_user_stats.out_fail) });
+	pisa_seq_kv(s, "skipped",
+		    { "no_budget", pisa_stat_atomic_read(pisa_user_stats.no_budget) },
+		    { "busy", pisa_stat_atomic_read(pisa_user_stats.busy) },
+		    { "large", pisa_stat_atomic_read(pisa_user_stats.large) },
+		    { "copy", pisa_stat_atomic_read(pisa_user_stats.copy) });
+}
diff --git a/net/core/pisa-net.c b/net/core/pisa-net.c
index 6422c472029a1..2ff90511cb7ae 100644
--- a/net/core/pisa-net.c
+++ b/net/core/pisa-net.c
@@ -256,6 +256,8 @@ bool __pisa_zc_iter_stop(struct sk_buff *skb, struct iov_iter *from,
 		return false;
 	}
 
+	if (skb->len == orig_len)
+		pisa_stats_inc_user_copy();
 	/* -EXDEV tells tcp_sendmsg_locked() to copy this chunk instead of zerocopy. */
 	*copied = (skb->len != orig_len) ? 0 : -EXDEV;
 	return true;
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (20 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 23/25] net/gve: " Luigi Rizzo
                     ` (4 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

libeth_rx_alloc() is called both when pre-filling RX fill queues in
process context (for indices 0 .. fq->count - 2, when buf->truesize is
still 0) and when refilling RX buffers from NAPI softirq.

Use GFP_KERNEL during the initial process-context ring fill so page_pool
and PISA page allocations (including set_memory_decrypted() under CoCo)
are not forced into GFP_ATOMIC context.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 include/net/libeth/rx.h | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/include/net/libeth/rx.h b/include/net/libeth/rx.h
index 0e736846c5e8a..4361dbcf03cb1 100644
--- a/include/net/libeth/rx.h
+++ b/include/net/libeth/rx.h
@@ -106,10 +106,17 @@ void libeth_rx_fq_destroy(struct libeth_fq *fq);
 static inline dma_addr_t libeth_rx_alloc(const struct libeth_fq_fp *fq, u32 i)
 {
 	struct libeth_fqe *buf = &fq->fqes[i];
+	/* fq->fqes is zero-allocated and pre-filled in process context for
+	 * indices [0 .. fq->count - 2]; slot fq->count - 1 is first allocated
+	 * in NAPI. Use GFP_KERNEL for initial pre-fill and GFP_ATOMIC after.
+	 */
+	gfp_t gfp = (buf->truesize || i == fq->count - 1 || in_softirq()) ?
+		    GFP_ATOMIC : GFP_KERNEL;
 
 	buf->truesize = fq->truesize;
-	buf->netmem = page_pool_dev_alloc_netmem(fq->pp, &buf->offset,
-						 &buf->truesize);
+	buf->netmem = page_pool_alloc_netmem(fq->pp, &buf->offset,
+					     &buf->truesize,
+					     gfp | __GFP_NOWARN);
 	if (unlikely(!buf->netmem))
 		return DMA_MAPPING_ERROR;
 
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 23/25] net/gve: Use GFP_KERNEL when pre-filling RX rings
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (21 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
                     ` (3 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

When pre-filling DQO and GQI RX rings in process context before NAPI is
enabled (e.g. gve_open), use GFP_KERNEL instead of GFP_ATOMIC so page
allocation (and CoCo page decryption) can sleep instead of dipping into
atomic reserves.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c |  8 +++++++-
 drivers/net/ethernet/google/gve/gve_rx.c              | 10 +++++-----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
index 6880d1531ba5e..a3fe684ca087a 100644
--- a/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
+++ b/drivers/net/ethernet/google/gve/gve_buffer_mgmt_dqo.c
@@ -223,13 +223,19 @@ void gve_free_to_page_pool(struct gve_rx_ring *rx,
 static int gve_alloc_from_page_pool(struct gve_rx_ring *rx,
 				    struct gve_rx_buf_state_dqo *buf_state)
 {
+	/*
+	 * Use GFP_KERNEL when pre-filling RX rings in process context (e.g.
+	 * gve_open) so PISA metadata init and CoCo page decryption can sleep.
+	 */
+	gfp_t gfp = (gve_get_napi_enabled(rx->gve) || in_softirq()) ?
+		    GFP_ATOMIC : GFP_KERNEL;
 	netmem_ref netmem;
 
 	buf_state->page_info.buf_size = rx->packet_buffer_truesize;
 	netmem = page_pool_alloc_netmem(rx->dqo.page_pool,
 					&buf_state->page_info.page_offset,
 					&buf_state->page_info.buf_size,
-					GFP_ATOMIC);
+					gfp);
 
 	if (!netmem)
 		return -ENOMEM;
diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 81ea800e66e94..7c201fd513ee3 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -159,14 +159,13 @@ static void gve_setup_rx_buffer(struct gve_rx_ring *rx,
 static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
 			       struct gve_rx_slot_page_info *page_info,
 			       union gve_rx_data_slot *data_slot,
-			       struct gve_rx_ring *rx)
+			       struct gve_rx_ring *rx, gfp_t gfp)
 {
 	struct page *page;
 	dma_addr_t dma;
 	int err;
 
-	err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE,
-			     GFP_ATOMIC);
+	err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
 	if (err) {
 		u64_stats_update_begin(&rx->statss);
 		rx->rx_buf_alloc_fail++;
@@ -209,7 +208,7 @@ static int gve_rx_prefill_pages(struct gve_rx_ring *rx,
 		}
 		err = gve_rx_alloc_buffer(priv, &priv->pdev->dev,
 					  &rx->data.page_info[i],
-					  &rx->data.data_ring[i], rx);
+					  &rx->data.data_ring[i], rx, GFP_KERNEL);
 		if (err)
 			goto alloc_err_rda;
 	}
@@ -986,7 +985,8 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
 				gve_rx_free_buffer(dev, page_info, data_slot);
 				page_info->page = NULL;
 				if (gve_rx_alloc_buffer(priv, dev, page_info,
-							data_slot, rx)) {
+							data_slot, rx,
+							GFP_ATOMIC)) {
 					break;
 				}
 			}
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (22 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 23/25] net/gve: " Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20  1:06   ` [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation Luigi Rizzo
                     ` (2 subsequent siblings)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

When dev->pisa_rx_bufs is enabled, allocate GQI-RDA RX packet buffers
with pisa_rx_gfp(dev) so they come from the per-node PISA
allocator, and evict plain pages when a free PISA RX block is available;
DQO-RDA uses page_pool, which checks dev->pisa_rx_bufs directly.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 drivers/net/ethernet/google/gve/gve_rx.c | 34 ++++++++++++++++--------
 1 file changed, 23 insertions(+), 11 deletions(-)

diff --git a/drivers/net/ethernet/google/gve/gve_rx.c b/drivers/net/ethernet/google/gve/gve_rx.c
index 7c201fd513ee3..562c11f5bdd24 100644
--- a/drivers/net/ethernet/google/gve/gve_rx.c
+++ b/drivers/net/ethernet/google/gve/gve_rx.c
@@ -4,6 +4,7 @@
  * Copyright (C) 2015-2021 Google, Inc.
  */
 
+#include <linux/pisa.h>
 #include "gve.h"
 #include "gve_adminq.h"
 #include "gve_utils.h"
@@ -165,6 +166,8 @@ static int gve_rx_alloc_buffer(struct gve_priv *priv, struct device *dev,
 	dma_addr_t dma;
 	int err;
 
+	if (rx->data.raw_addressing)
+		gfp |= pisa_rx_gfp(dev);
 	err = gve_alloc_page(priv, dev, &page, &dma, DMA_FROM_DEVICE, gfp);
 	if (err) {
 		u64_stats_update_begin(&rx->statss);
@@ -494,13 +497,19 @@ static void gve_rx_flip_buff(struct gve_rx_slot_page_info *page_info, __be64 *sl
 	*(slot_addr) ^= offset;
 }
 
-static int gve_rx_can_recycle_buffer(struct gve_rx_slot_page_info *page_info)
+static int gve_rx_can_recycle_buffer(struct gve_rx_ring *rx,
+				     struct gve_rx_slot_page_info *page_info)
 {
 	int pagecount = page_count(page_info->page);
 
 	/* This page is not being used by any SKBs - reuse */
-	if (pagecount == page_info->pagecnt_bias)
+	if (pagecount == page_info->pagecnt_bias) {
+		if (rx && rx->data.raw_addressing &&
+		    !is_pisa_page(page_info->page) &&
+		    pisa_has_free_blocks_rx(&rx->gve->pdev->dev, 0, rx->gve->numa_node))
+			return 0;
 		return 1;
+	}
 	/* This page is still being used by an SKB - we can't reuse */
 	else if (pagecount > page_info->pagecnt_bias)
 		return 0;
@@ -544,7 +553,7 @@ static struct sk_buff *gve_rx_copy_to_pool(struct gve_rx_ring *rx,
 
 	copy_page_info = &rx->qpl_copy_pool[pool_idx];
 	if (!copy_page_info->can_flip) {
-		int recycle = gve_rx_can_recycle_buffer(copy_page_info);
+		int recycle = gve_rx_can_recycle_buffer(NULL, copy_page_info);
 
 		if (unlikely(recycle < 0)) {
 			gve_schedule_reset(rx->gve);
@@ -664,7 +673,7 @@ static struct sk_buff *gve_rx_skb(struct gve_priv *priv, struct gve_rx_ring *rx,
 			u64_stats_update_end(&rx->statss);
 		}
 	} else {
-		int recycle = gve_rx_can_recycle_buffer(page_info);
+		int recycle = gve_rx_can_recycle_buffer(rx, page_info);
 
 		if (unlikely(recycle < 0)) {
 			gve_schedule_reset(priv);
@@ -970,7 +979,7 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
 			 * owns half the page it is impossible to tell which half. Either
 			 * the whole page is free or it needs to be replaced.
 			 */
-			int recycle = gve_rx_can_recycle_buffer(page_info);
+			int recycle = gve_rx_can_recycle_buffer(rx, page_info);
 
 			if (recycle < 0) {
 				if (!rx->data.raw_addressing)
@@ -982,13 +991,16 @@ static bool gve_rx_refill_buffers(struct gve_priv *priv, struct gve_rx_ring *rx)
 				union gve_rx_data_slot *data_slot =
 						&rx->data.data_ring[idx];
 				struct device *dev = &priv->pdev->dev;
-				gve_rx_free_buffer(dev, page_info, data_slot);
-				page_info->page = NULL;
-				if (gve_rx_alloc_buffer(priv, dev, page_info,
-							data_slot, rx,
-							GFP_ATOMIC)) {
+				struct gve_rx_slot_page_info new_info = {};
+				union gve_rx_data_slot new_slot;
+
+				if (gve_rx_alloc_buffer(priv, dev, &new_info,
+							&new_slot, rx,
+							GFP_ATOMIC))
 					break;
-				}
+				gve_rx_free_buffer(dev, page_info, data_slot);
+				*page_info = new_info;
+				*data_slot = new_slot;
 			}
 		}
 		fill_cnt++;
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (23 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
@ 2026-11-20  1:06   ` Luigi Rizzo
  2026-11-20 13:50   ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
  2026-11-20 14:54   ` Lorenzo Stoakes (ARM)
  26 siblings, 0 replies; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20  1:06 UTC (permalink / raw)
  To: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo
  Cc: Lorenzo Stoakes, Liam R . Howlett, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Brendan Jackman, Zi Yan,
	Harry Yoo, Christoph Lameter, David Rientjes, Roman Gushchin,
	Hao Li, Matthew Wilcox, Jan Kara, Jann Horn, Catalin Marinas,
	Greg Kroah-Hartman, Rafael J . Wysocki, Danilo Krummrich,
	Jonathan Corbet, Simon Horman, Jesper Dangaard Brouer,
	Ilias Apalodimas, Willem de Bruijn, Kuniyuki Iwashima,
	Joshua Washington, Harshitha Ramamurthy, Alexander Lobakin,
	Christoph Hellwig, Jason Gunthorpe, Dragos Tatulea, Naman Gulati,
	Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap, linux-mm, iommu,
	netdev, linux-block, linux-fsdevel, driver-core, linux-arm-kernel,
	linux-api, linux-doc, linux-kernel, Luigi Rizzo

Document the PISA IOMMU 2MB-hugepage-backed DMA memory
subsystem in Documentation/core-api/pisa.rst.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 .../admin-guide/kernel-parameters.txt         |   7 +
 Documentation/core-api/index.rst              |   1 +
 Documentation/core-api/pisa.rst               | 485 ++++++++++++++++++
 3 files changed, 493 insertions(+)
 create mode 100644 Documentation/core-api/pisa.rst

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec2..cce75a9abddad 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5345,6 +5345,13 @@ Kernel parameters
 	pirq=		[SMP,APIC] Manual mp-table setup
 			See Documentation/arch/x86/i386/IO-APIC.rst.
 
+	pisa.*=		[ARM64,X86-64] Pre-mapped IOVA Superpage Allocator (PISA)
+			tuning parameters (max_pages, max_idle, idle_ms,
+			reservoir_max, scrub_max, compact,
+			target_usage_percent, old_ms, pcp_batch, user_max_pages,
+			scrub_migrate).
+			See Documentation/core-api/pisa.rst.
+
 	plip=		[PPT,NET] Parallel port network link
 			Format: { parport<nr> | timid | 0 }
 			See also Documentation/admin-guide/parport.rst.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..4955f477ed797 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
    dma-attributes
    dma-isa-lpc
    swiotlb
+   pisa
    mm-api
    cgroup
    genalloc
diff --git a/Documentation/core-api/pisa.rst b/Documentation/core-api/pisa.rst
new file mode 100644
index 0000000000000..f9690b4a9c96a
--- /dev/null
+++ b/Documentation/core-api/pisa.rst
@@ -0,0 +1,485 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=============================================
+PISA: Pre-mapped IOVA Superpage Allocator
+=============================================
+
+Overview
+========
+
+PISA is an extension to the page allocator: requests with ``__GFP_PISA``
+return memory that makes I/O-related operations cheaper when using
+``DMA-FQ``, ``DMA``, or Confidential Computing ``swiotlb``.  I/O devices
+can individually opt in to use PISA for specific buffer classes (e.g. read
+or write data, headers, zerocopy, ...).  Relevant kernel subsystems
+(mm, iommu, dma, disk I/O, page cache, the network stack) are modified
+to make use of PISA memory for opt-in devices.
+
+The design and implementation have the following main goals:
+
+- no changes to individual device drivers (or minimal changes for NIC drivers
+  that use custom RX page allocators)
+- minimize changes to base code: in most cases one or a few lines of code
+- safety: PISA memory is only used for devices that opt-in,
+  and it is returned to the buddy allocator only after IOMMU mappings
+  and IOTLB entries have been flushed
+- no dependency on ``struct page`` or ``struct folio``
+- compile-time option to disable or enable PISA
+- zero runtime overhead when no devices opt in (via static keys and inline hooks)
+- minimal overhead when in use (direct PFN lookup in a flat metadata table)
+- keep memory overhead as low as possible (adaptive shrinking and compaction
+  of memory)
+
+Mechanisms and performance benefits
+-----------------------------------
+PISA returns order-N blocks backed by 2MB physically contiguous pages, so it
+can use 2MB IOMMU leaves and reduce the IOTLB pressure by up to 512x.
+PISA maps memory in pre-allocated per-domain IOVA ranges, and recycles
+blocks and mappings until the 2MB page is returned to the buddy allocator.
+This makes most of dma map/unmap a no-op, also avoiding related VM exits
+in overcommitted VMs.
+In Confidential Computing VMs, PISA pages are also decrypted, removing the need
+for bounce buffers.
+The kernel identifies devices and sockets that use an
+IOMMU or Confidential Computing and have opted into PISA, and allocates (or
+migrates) I/O buffers using PISA memory so that DMA mapping and unmapping are
+mostly no-ops. All the above makes performance comparable to ``identity`` mode,
+with isolation better than ``DMA-FQ`` and comparable to ``DMA``.
+
+Devices that do not opt in continue to use their native DMA/DMA-FQ mode.
+
+Comparison with standard DMA modes
+----------------------------------
+
+The table below summarizes how PISA (layered on top of strict ``DMA``)
+compares against pass-through/identity mapping (``identity``), batched
+deferred IOTLB flush (``DMA-FQ``), and strict per-request IOMMU mapping
+(``DMA``):
+
+======================================= ==================== ================== ===============
+Feature ( > means better )              vs ``identity``      vs ``DMA-FQ``      vs ``DMA``
+======================================= ==================== ================== ===============
+``dma_map`` cost                        ``PISA ~= identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+``dma_unmap`` cost                      ``PISA == identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+IOTLB friendliness                      ``na``               ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Buffer shared across domains            ``na``               ``PISA <= DMA-FQ`` ``PISA <= DMA``
+Memory remains accessible to device     ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA <= DMA``
+Needs bounce buffers                    ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Page alloc/free cost                    ``PISA >= identity`` ``PISA >= DMA-FQ`` ``PISA >= DMA``
+======================================= ==================== ================== ===============
+
+Notes on the comparisons:
+
+- **dma_map / dma_unmap cost:** Once a 2MB PISA page is mapped in a domain's
+  reserved IOVA window, ``dma_map_*()`` is an offset lookup and
+  ``dma_unmap_*()`` is a no-op, avoiding IOVA tree
+  allocation, page-table walks, and synchronous IOTLB shootdowns.
+- **IOTLB friendliness:** PISA maps 2MB IOMMU leaves instead of 4KB leaves,
+  covering 512x more memory per IOTLB entry than ``DMA-FQ`` and ``DMA``.
+  A domain without a 2MB page size maps each PISA page with smaller IOMMU
+  pages: mapping is still done once per PISA page, without this benefit.
+- **Buffer shared across domains:** Even in ``DMA-FQ`` and ``DMA``, IOMMU
+  mappings operate at page granularity, so sub-page buffers from ``task_frag``
+  (shared across all sockets of a task) or ``skb`` head slabs can cause the
+  underlying page to be mapped across multiple domains. PISA exhibits the same
+  behavior, just at 2MB granularity.
+- **Memory remains accessible to device:** In ``identity`` mode, all of RAM is
+  always accessible to devices. In ``DMA-FQ``, pages returned to the buddy
+  allocator remain accessible until the asynchronous flush queue drains. Both
+  PISA and ``DMA`` unmap the page and perform a synchronous IOTLB flush before
+  returning memory to the buddy allocator. In all modes, receive buffers remain mapped
+  and accessible to the device while queued in the rx socket buffer.
+- **Needs bounce buffers:** In Confidential Computing (CoCo) VMs, PISA
+  allocators pre-decrypt 2MB pages in place, avoiding ``swiotlb`` bounce-buffer
+  copies across all DMA modes.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+  (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+  access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+  exceeds the IOTLB size, backpressuring the bus via flow control.
+  This is one of the biggest bottlenecks for high speed devices:
+  typical network devices have very little IOVA locality, and that makes
+  the IOTLB almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA
+   map/unmap costs. This is common practice for command, completion and
+   buffer queues, as well as disk I/O and network receive buffers.  Network
+   TX buffers are a noticeable exception. They normally come from
+   ``alloc_pages()`` calls unaware of the leaf device, and require
+   mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+   very expensive operation (up to several microseconds to wait for
+   completion, necessary for security), but the way it is implemented
+   in DMA-FQ compromises on security: IOTLB flush requests are run
+   periodically in batches, but the underlying memory is recycled without
+   waiting. During that window, the physical memory is still accessible,
+   opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+   blocks, e.g. 2MB large pages instead of the usual 4KB (PTE). This is not
+   currently pursued by the kernel.
+
+PISA addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map,
+  unmap, and free device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB PISA pages and 2MB IOMMU
+  mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused PISA pages
+  to the system
+
+Execution is what makes PISA practical: the implementation intercepts
+a small set of kernel APIs so that devices will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since PISA intercepts allocations of device-accessible
+memory, it can transparently configure unencrypted mode, required for
+confidential computing. Under a paravirtualized IOMMU, the long-lived 2MB
+mappings also make host-side pinning of DMA memory a rare event.
+
+Limitations
+===========
+
+The implementation assumes non-preemptible spinlocks, so PISA is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+On arm64, PISA requires 4KB pages and at least 4 levels of page tables.
+This is enforced at compile time.
+
+PISA can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers follow the direction the allocator was created
+with. Descriptor queues and receive buffers are ``DMA_BIDIRECTIONAL``, so received
+buffers can be retransmitted (XDP_TX, forwarding) without a new mapping.
+Allocators that only hold transmit data, such as disk/network write buffers and socket TX fragments, are
+``DMA_TO_DEVICE``.
+
+Implementation details
+======================
+
+PISA is implemented by a few groups of commits:
+
+- mm: infrastructure to implement per-page metadata (avoid any dependency on struct page),
+  and hook into ``alloc_pages``, PISA optimizations (per-cpu caches, shrinker, background zeroing)
+- driver core: per-device sysfs entries to opt-in
+- iommu/dma: manage PISA IOVA ranges and intercept map/unmap
+- block/, mm/, net/: transparently upgrade pages to PISA for disk I/O and network tx zerocopy
+- net: identify sockets that need PISA pages, hook PISA into page frags, skb heads, pagepool
+
+Selected NIC drivers that do not use pagepool need small changes (a few lines)
+to pass ``__GFP_PISA_RW`` when allocating RX buffers.
+
+The implementation lives in ``mm/pisa-*.c``, ``drivers/iommu/pisa-map.c``,
+``net/core/pisa-net.c``, and ``include/linux/pisa*.h``.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, socket TX page frags, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+- ``__filemap_get_folio_mpol()``, ``ractl_alloc_folio()``,
+  ``writeback_get_folio()``, ``__bio_iov_iter_get_pages()``, ``madvise()``
+
+and most devices will automatically inherit the benefits. Fast-path hooks
+are guarded by static keys: ``pisa_key`` remains disabled (a single NOP)
+until the first device opts in via sysfs.
+Separate static keys gate TX socket fragments, small TX skb heads,
+user/disk page migration, and compaction checks until those features are
+enabled.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of PISA pages, based on either their Physical
+  Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to PISA page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **PISA metadata** (``pisa_pfn2slot`` and ``struct pisa_meta``):
+
+  Each 2MB page used for PISA memory is identified by a compact index (``slot``),
+  up to half of physical RAM (256 per 1GB).
+  Identification of PISA pages and per-page metadata use two flat
+  arrays in virtual address space, with holes mapped read-only
+  to a zero page:
+
+  1. ``pisa_pfn2slot[pfn >> PISA_ORDER]`` (4 bytes per 2MB physical frame)
+     packs the page's status flags and the compact page index. It is populated for
+     system RAM at boot or via hotplug events.
+     A full 57-bit address space requires only 256GB of virtual address space
+     and 12KB for page tables, and expands by 4KB for each 2GB of physical RAM,
+     regardless of RAM layout. With ``CONFIG_KASAN_VMALLOC`` the array covers a
+     49-bit address space (1GB of virtual address space), to bound the KASAN
+     shadow memory.
+  2. ``pisa_meta_array[slot]`` (256B per slot/2MB PISA page)
+     holds ``struct pisa_meta``, tracking the page's base PFN, status flags, free/clean/dirty
+     subpage bitmaps, the bitmap of IOMMU domains with an active 2MB mapping
+     for the page, and list/RCU linkage to manage its lifetime.
+     The slot table is sized at boot for up to half of physical RAM (e.g. 128K
+     slots = 32MB of virtual address space for 512GB of RAM), with 4KB backing
+     pages allocated on demand up to ``pisa.max_pages`` (default 1/8th of RAM).
+
+- **Per-domain reserved PISA IOVA ranges**:
+
+  A PISA page can be mapped in multiple domains (e.g.
+  a network buffer used to route packets between different NICs). To simplify
+  address tracking,
+  PISA reserves on each domain an IOVA range covering the slot table
+  (half of physical RAM; the base address may differ per domain),
+  and the page is mapped on demand at an offset ``slot * PISA_SIZE`` in
+  each domain that requires it.
+  This makes PA-IOVA translations just an addition, uniquely identifies
+  PISA IOVAs, and does not expose physical addresses.
+
+- **alloc_pages() compatible pisa_alloc allocator for I/O buffers**:
+
+  PISA implements a pisa_alloc object for fixed-order page allocations
+  returning objects fully compatible with those returned by the page allocator,
+  and with no dependencies on the struct page.
+  ``alloc_pages()`` and related requests are routed to PISA via a
+  ``__GFP_PISA`` flag. This makes it easy to select PISA pages when needed
+  (eg. only two lines in pagepool, similar small changes in page frag refill
+  and disk I/O buffers).
+
+- **Fine grained control**
+
+  Both for experimentation and production use, it is useful to have
+  some form of control on which devices want to use PISA and for
+  what. PISA exposes the following controls. Keep in mind that full
+  performance can only be achieved if all regions are mapped using PISA.
+
+  - ``/sys/devices/*/*/pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads,tx_zerocopy}``
+    per-device entries that control whether a device uses PISA for coherent
+    allocations, RX/disk-read buffers, TX/disk-write buffers, small TX
+    skb heads, and ``MSG_ZEROCOPY`` user pages
+
+- **Safe release of memory to the system**
+
+  A key feature of PISA is that memory is not released to the system
+  while there are active IOMMU mappings. Release of buffers is managed
+  as follows
+
+  1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to
+     PISA, IOMMU mappings are still active, and they can be recycled
+
+  2. when all blocks in a PISA page are unused, the page is moved to the
+     allocator's "idle" list, still with active mappings and available for reuse
+
+  3. idle pages above the per-allocator watermark (``pisa.max_idle``) are
+     released by a reclaim work item after the ``pisa.idle_ms`` grace period,
+     and under memory pressure the shrinker also releases reservoir and idle pages
+     from all allocators: the iommu mappings are removed and a synchronous IOTLB
+     flush is issued. The pages are no longer usable for allocations but not
+     returned to the buddy allocator yet
+
+  4. once the previous step is complete, the system schedules ``call_rcu()``
+     and waits for an RCU grace period (and re-encrypts the page if it was
+     decrypted) before returning the PISA page to the buddy allocator.
+
+- **Memory compaction**
+
+  Partially allocated 2MB PISA pages are classified into dense (at most 50%
+  free blocks, placed at the head of the partial list and preferred for new
+  allocations) and sparse (at least 75% free blocks, placed at the tail to
+  drain). When ``pisa.compact`` or ``pisa.target_usage_percent`` is enabled and
+  utilization drops below the target, ``pisa_page_can_recycle()`` stops per-CPU
+  and caller-pool recycling for blocks on old sparse pages so they drain to the
+  idle list. For ``user_pa`` pages, ``pisa.scrub_migrate`` also migrates live
+  4KB pages out of old sparse 2MB pages in the background.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+PISA integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of PISA memory is opt-in via per-device sysfs
+attributes or module parameters.
+
+- **Streaming DMA map and unmap**
+
+  - On map, any buffer belonging to a PISA page
+    (``is_pisa_phys(phys)``) is mapped via the domain's reserved
+    PISA IOVA window with a simple addition, typically reusing
+    existing mappings. Under ``dma-direct``, decrypted PISA pages
+    (``is_pisa_direct(phys, size)``) can be mapped directly without
+    ``swiotlb`` bounce.
+
+    On unmap, IOVAs can be identified as PISA buffers with a simple
+    range check and that results in a no-operation.
+
+- **Coherent DMA allocation and free**
+
+  - PISA is enabled per device via ``/sys/devices/.../pisa_coherent``
+    (``dev->pisa_coherent``). When set, sleepable ``GFP_KERNEL`` coherent
+    allocations up to ``PISA_SIZE`` are rounded up to 4KB, carved out of
+    the per-node order-0 ``pisa_alloc`` via ``pisa_alloc_map_coherent()``, and
+    mapped through the domain's PISA IOVA window (or directly at their
+    physical/decrypted DMA address for ``dma-direct``). ``dma_free_attrs()``
+    releases the order-0 pages via ``pisa_free_coherent()``, which recycles
+    them back to the allocator.
+
+- **Page allocator release**
+
+  - ``pisa_free_page()`` checks ``is_pisa_pfn()``
+    and recycles PISA subpages back to their owning ``pisa_alloc``
+    instead of releasing them to the buddy allocator while their 2MB IOMMU
+    mappings remain active.
+
+- **IOMMU domain teardown**
+
+  - A call to ``pisa_domain_release()`` before freeing a
+    domain's IOVA cookie and page tables clears cached domain state across
+    all allocators. When the last DMA API driver unbinds from a group (or
+    ownership is transferred to VFIO/iommufd), ``pisa_domain_quiesce()``
+    unmaps active PISA pages from that domain and flushes the IOTLB while
+    keeping the reserved window for future rebinds.
+
+Memory allocations are handled as follows:
+
+- **Networking receive packet buffers**
+
+  - PISA is enabled per device via ``pisa_rx_bufs``. A two-line change
+    in pagepool adds ``__GFP_PISA_RW`` to the allocation calls,
+    allocating RX pages from the per-node ``DMA_BIDIRECTIONAL`` PISA
+    allocator and falling back to the buddy allocator on exhaustion.
+
+  - small patches are needed to use PISA memory for RX buffers in devices
+    managing their own page rings e.g. see ``gve`` as an example.
+
+- **Socket TX page fragments**
+
+  - PISA is enabled per device via ``pisa_tx_bufs``, which allows
+    socket TX fragments to be allocated from PISA pages, mapped ``DMA_TO_DEVICE``.
+  - Only data routed to an opt-in device with a DMA parent, outside IPsec,
+    uses PISA memory.  Loopback, virtual devices, kTLS, sockmap, KCM
+    and MPTCP use ordinary pages. ESP output uses PISA memory when its
+    egress device allows it.
+
+    The code implementing these constraints touches multiple
+    files because there isn't a single decision point.
+
+- **Small skb heads**
+
+  - PISA is enabled per device via ``pisa_tx_skb_heads``,
+    which allows small skb heads to come from an ``skbuff_small_head_pisa`` slab cache,
+    backed by PISA memory mapped ``DMA_TO_DEVICE``.
+    PISA memory is only used in the cases listed earlier.
+
+    One exception, PISA memory is never used for skb small heads in Confidential
+    Computing because it would expose kernel pointers and callback structures.
+    The fall back to swiotlb causes allocations, mappings and unmappings
+    per packet that can be avoided with ``ethtool --set-tunable <dev> tx-copybreak <bytes>``
+    which uses PISA memory if ``pisa_coherent`` is enabled.
+
+- **Zerocopy network receive**
+
+  - When ``pisa_rx_bufs`` is enabled, receive buffers allocated from
+    ``DMA_BIDIRECTIONAL`` PISA pages can be mapped directly into user space by
+    ``TCP_ZEROCOPY_RECEIVE`` (``vm_insert_pages()``). On Confidential Computing
+    guests, user PTEs mapping decrypted PISA pages are automatically marked
+    decrypted via ``pgprot_decrypted()``.
+
+- **Zerocopy network transmit**
+
+  - PISA is enabled per device via ``pisa_tx_zerocopy`` (and requires
+    ``pisa.user_max_pages``). ``MSG_ZEROCOPY`` sends best-effort migrate eligible user
+    pages into ``DMA_TO_DEVICE`` PISA blocks (skipping pages still referenced
+    by in-flight skbs). Migrated pages stay readable by the device for as
+    long as they remain in PISA, not only during the send.
+  - If ``pisa_tx_bufs`` is also enabled, TCP copies the pages that could not
+    be migrated into PISA socket fragments. Without ``pisa_tx_zerocopy``,
+    ``MSG_ZEROCOPY`` sends use ordinary per-buffer mappings.
+
+- **User and disk I/O pages**
+
+  - ``madvise(MADV_PISA)`` migrates the present private anonymous pages
+    of a range (splitting large anonymous folios to order 0) into blocks of
+    per-node ``user_pa`` allocators, so that zerocopy sends and I/O from them
+    use the PISA window instead of per-request IOMMU mappings or hypervisor
+    pinning. ``MADV_NOPISA`` migrates them back to ordinary pages. Both are
+    allowed through ``process_madvise()``.
+    Freeing the pages (``munmap()``, ``MADV_DONTNEED``, exit, reclaim)
+    returns the blocks to their allocators.
+  - ``pisa.user_max_pages``: Soft cap on user and page-cache pages held in
+    PISA blocks, in 2MB PISA pages like ``pisa.max_pages``. Setting
+    a non-zero value initializes the per-node ``user_pa`` allocators (for
+    ``DMA_BIDIRECTIONAL``, ``DMA_TO_DEVICE``, and ``DMA_FROM_DEVICE``) and
+    enables ``pisa_user_key``; 0 (default) disables it.
+  - For block devices with ``pisa_rx_bufs`` (reads) or ``pisa_tx_bufs``
+    (writes) enabled on a parent device, new order-0 page-cache folios are
+    allocated directly from ``user_pa``, and writeback or direct-I/O bio
+    preparation migrates eligible folios into ``user_pa`` for the I/O's DMA
+    direction, promoting existing read-only or write-only PISA pages to
+    ``DMA_BIDIRECTIONAL`` when accessed in the opposite direction.
+  - On CoCo guests, user PTEs mapping decrypted PISA blocks are marked
+    decrypted via ``pgprot_decrypted()`` so both user space and the device
+    access the shared page in cleartext.
+
+- **Observability**
+
+  - ``/sys/kernel/debug/pisa/allocs``: Debugfs summary of global, per-node,
+    user-page, and per-allocator allocation, mapping, and fallback counters.
+
+- **Global controls:**
+
+  Module parameters are set on the kernel command line as ``pisa.<name>`` and
+  exposed under ``/sys/module/pisa/parameters/<name>``:
+
+  - ``pisa.max_pages``: Global cap on PISA 2MB pages in use
+    (0 defaults to 1/8th of physical RAM, at least 16 pages). Can be changed
+    at runtime, max is half of physical RAM.
+  - ``pisa.reservoir_max``: Per-NUMA-node reservoir target of pre-allocated
+    PISA pages (defaults to 16 per memory node, 0 = disabled). It cannot be 0
+    in Confidential Computing guests: pages are decrypted in sleepable
+    context, so atomic allocations are only served from the reservoirs, which
+    are also filled when PISA is first enabled.
+  - ``pisa.max_idle``: Per-allocator idle PISA page watermark (default 1,
+    minimum 1) before releasing excess idle pages.
+  - ``pisa.idle_ms``: Grace period in ms (default 200, 0 = immediate) before
+    releasing idle PISA pages above ``pisa.max_idle``.
+  - ``pisa.pcp_batch``: Number of blocks moved per per-CPU cache refill or
+    flush (default 32, range 1..32).
+  - ``pisa.compact``: Usage-based compaction threshold in 2MB PISA pages
+    (0 = disabled). When free blocks across an allocator's non-idle 2MB pages
+    exceed ``compact`` 2MB pages, blocks belonging to sparse partial pages
+    bypass per-CPU caches and caller fragment pools so those sparse pages can
+    drain back to the idle list.
+  - ``pisa.target_usage_percent``: Target minimum block utilization percentage
+    across an allocator's non-idle 2MB pages (0 = disabled, 1..100). When usage
+    is below ``target_usage_percent``%, stop recycling blocks from sparse
+    partial pages older than ``old_ms`` ms so they can drain.
+  - ``pisa.old_ms``: Minimum age in ms before a sparse partial page is drained
+    by ``target_usage_percent`` or migrated by ``scrub_migrate`` (0 = disabled).
+  - ``pisa.scrub_max``: Max 4KB pages of dirty blocks scrubbed per background
+    pass (0 disables background scrubbing).
+  - ``pisa.scrub_migrate``: Background migration of old sparse ``user_pa``
+    pages when usage is below ``target_usage_percent`` (0 = disabled,
+    1 = compact into denser PISA pages, 2 = demote back to the buddy allocator).
+  - ``pisa.user_max_pages``: Soft cap in 2MB PISA pages on user and page-cache
+    pages held in PISA blocks (0 = disabled).
-- 
2.56.1.809.ga0b61035b5-goog


^ permalink raw reply related	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (24 preceding siblings ...)
  2026-11-20  1:06   ` [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation Luigi Rizzo
@ 2026-11-20 13:50   ` Jason Gunthorpe
  2026-11-20 14:18     ` Lorenzo Stoakes (ARM)
  2026-11-20 14:19     ` Luigi Rizzo
  2026-11-20 14:54   ` Lorenzo Stoakes (ARM)
  26 siblings, 2 replies; 32+ messages in thread
From: Jason Gunthorpe @ 2026-11-20 13:50 UTC (permalink / raw)
  To: Luigi Rizzo
  Cc: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo, Lorenzo Stoakes,
	Liam R . Howlett, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Brendan Jackman, Zi Yan, Harry Yoo, Christoph Lameter,
	David Rientjes, Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara,
	Jann Horn, Catalin Marinas, Greg Kroah-Hartman,
	Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
	Simon Horman, Jesper Dangaard Brouer, Ilias Apalodimas,
	Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
	Harshitha Ramamurthy, Alexander Lobakin, Christoph Hellwig,
	Dragos Tatulea, Naman Gulati, Pranjal Shrivastava, Mostafa Saleh,
	Randy Dunlap, linux-mm, iommu, netdev, linux-block, linux-fsdevel,
	driver-core, linux-arm-kernel, linux-api, linux-doc, linux-kernel

On Fri, Nov 20, 2026 at 01:06:30AM +0000, Luigi Rizzo wrote:
> This is v3 of the RFC previously posted as DMA_PMD [0], now renamed PISA
> ("Pre-mapped IOVA Superpage Allocator"). The text below follows
> Documentation/core-api/pisa.rst (patch 25/25), which has the details.

I thought the conclusion was to just do this stuff in netdev pagepool,
now you come with an enormous AI series spread across the entire
kernel?

> - Needs bounce buffers: in Confidential Computing VMs, PISA allocators
>   pre-decrypt 2MB pages in place, avoiding swiotlb bounce-buffer copies.

It seems this was ignored too. We want the bounce buffers in CC. They
serve a security purpose. You cannot loose control of the shared
memory buffers have have them leak into SKBs, user mappings and so
on. It is not secure.

I don't see how any of this is valuable or necessary for CC,
especially when we really don't even expect to have an vIOMMU for CC
very often..

Jason

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 11/25] mm: Migrate user pages into PISA
  2026-11-20  1:06   ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
@ 2026-11-20 14:10     ` Lorenzo Stoakes (ARM)
  0 siblings, 0 replies; 32+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-11-20 14:10 UTC (permalink / raw)
  To: Luigi Rizzo
  Cc: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo, Liam R . Howlett,
	Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
	Zi Yan, Harry Yoo, Christoph Lameter, David Rientjes,
	Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara, Jann Horn,
	Catalin Marinas, Greg Kroah-Hartman, Rafael J . Wysocki,
	Danilo Krummrich, Jonathan Corbet, Simon Horman,
	Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
	Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
	Alexander Lobakin, Christoph Hellwig, Jason Gunthorpe,
	Dragos Tatulea, Naman Gulati, Pranjal Shrivastava, Mostafa Saleh,
	Randy Dunlap, linux-mm, iommu, netdev, linux-block, linux-fsdevel,
	driver-core, linux-arm-kernel, linux-api, linux-doc, linux-kernel

On Fri, Nov 20, 2026 at 01:06:41AM +0000, Luigi Rizzo wrote:
> Zero-copy DMA paths (such as O_DIRECT disk I/O and MSG_ZEROCOPY sockets)
> map each user page with a per-buffer IOMMU mapping and IOTLB flush.
> Moving user buffers into PISA blocks once allows subsequent DMA
> operations to reuse pre-mapped 2MB IOMMU entries.
>
> Add mm/pisa-user.c to migrate private anonymous order-0 user pages into
> per-node DMA_TO_DEVICE PISA allocators, controlled by
> pisa.user_max_pages (0 = disabled), via madvise(MADV_PISA) /
> MADV_NOPISA (also supported via process_madvise()) and
> pisa_upgrade_user_pages().

No, this isn't how madvise works. It's not a place to stuff custom hooks for a
feature you want.

>
> Large anonymous folios are split to order 0 first, while budget remains,
> with split_folio_to_list_keep_zero(): a variant of split_folio_to_list()
> that leaves zero-filled subpages mapped instead of remapping them to the
> shared zeropage, so that pages the caller is about to write to (e.g. the
> target of an O_DIRECT read) are still there to migrate.
>
> Signed-off-by: Luigi Rizzo <lrizzo@google.com>

No thanks. This whole approach seems completely wrong.

> ---
>  include/linux/huge_mm.h                |   7 +
>  include/linux/pisa-mm.h                |   8 +
>  include/linux/pisa.h                   |  18 ++
>  include/linux/swap.h                   |   1 +
>  include/uapi/asm-generic/mman-common.h |   2 +
>  mm/Makefile                            |   2 +-
>  mm/huge_memory.c                       |  36 ++-
>  mm/madvise.c                           |  19 ++
>  mm/pisa-alloc.c                        |   2 +
>  mm/pisa-priv.h                         |   2 +
>  mm/pisa-user.c                         | 394 +++++++++++++++++++++++++
>  11 files changed, 483 insertions(+), 8 deletions(-)
>  create mode 100644 mm/pisa-user.c
>
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index fac05120ab774..d679de8cf261d 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -404,6 +404,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
>  int folio_split_unmapped(struct folio *folio, unsigned int new_order);
>  unsigned int min_order_for_split(struct folio *folio);
>  int split_folio_to_list(struct folio *folio, struct list_head *list);
> +int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list);
>  int folio_check_splittable(struct folio *folio, unsigned int new_order,
>  			   enum split_type split_type);
>  int folio_split(struct folio *folio, unsigned int new_order, struct page *page,
> @@ -671,6 +672,12 @@ static inline int split_folio_to_list(struct folio *folio, struct list_head *lis
>  	return -EINVAL;
>  }
>
> +static inline int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list)
> +{
> +	VM_WARN_ON_ONCE_FOLIO(1, folio);
> +	return -EINVAL;
> +}
> +
>  static inline int folio_split(struct folio *folio, unsigned int new_order,
>  		struct page *page, struct list_head *list)
>  {
> diff --git a/include/linux/pisa-mm.h b/include/linux/pisa-mm.h
> index d55a42397c4b3..1e82e081b4004 100644
> --- a/include/linux/pisa-mm.h
> +++ b/include/linux/pisa-mm.h
> @@ -11,6 +11,8 @@ unsigned long __pisa_alloc_pages_bulk(gfp_t gfp, unsigned int order,
>  				      int preferred_nid, int nr_pages,
>  				      struct page **page_array, bool frozen);
>  bool __pisa_free_page(struct page *page, unsigned int order);
> +int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
> +		 unsigned long end, bool to_pisa);
>
>  /*
>   * ============================================================================
> @@ -77,5 +79,11 @@ static inline bool pisa_free_page(struct page *page, unsigned int order)
>  	return false;
>  }
>
> +static inline int pisa_madvise(struct vm_area_struct *vma, unsigned long start,
> +			       unsigned long end, bool to_pisa)
> +{
> +	return -EINVAL;
> +}
> +
>  #endif /* CONFIG_PISA */
>  #endif /* _LINUX_PISA_MM_H */
> diff --git a/include/linux/pisa.h b/include/linux/pisa.h
> index edd34771ad1c4..9321e7ff00042 100644
> --- a/include/linux/pisa.h
> +++ b/include/linux/pisa.h
> @@ -14,6 +14,7 @@
>
>  struct iommu_domain;
>  struct iova_domain;
> +struct iov_iter;
>
>  /**
>   * struct pisa_iova - A domain's IOVA window reserved for PISA pages
> @@ -70,9 +71,11 @@ enum {
>   * Static keys guarding all kernel hooks:
>   * - @pisa_key: enabled on first device opt-in (pisa_meta_init()).
>   * - @pisa_recycle_check_key: enabled when pisa.compact or pisa.target_usage_percent > 0.
> + * - @pisa_user_key: enabled when pisa.user_max_pages > 0.
>   */
>  DECLARE_STATIC_KEY_FALSE(pisa_key);
>  DECLARE_STATIC_KEY_FALSE(pisa_recycle_check_key);
> +DECLARE_STATIC_KEY_FALSE(pisa_user_key);
>
>  extern u32 *pisa_pfn2slot;
>  extern unsigned int pisa_nr_slots;
> @@ -89,12 +92,19 @@ void *__pisa_alloc_map_coherent_dev(struct device *dev, size_t size,
>  				    unsigned long attrs);
>  bool __pisa_has_free_blocks_rx(unsigned int order, int nid);
>  bool __pisa_page_can_recycle(const struct page *page);
> +bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
> +			     size_t size, size_t offset);
>
>  static inline bool pisa_enabled(void)
>  {
>  	return static_branch_unlikely(&pisa_key);
>  }
>
> +static inline bool pisa_user_enabled(void)
> +{
> +	return static_branch_unlikely(&pisa_user_key);
> +}
> +
>  static inline u32 pisa_pfn_flags(unsigned long pfn)
>  {
>  	unsigned long frame = pfn >> PISA_ORDER;
> @@ -195,6 +205,7 @@ static inline bool pisa_page_can_recycle(const struct page *page)
>
>  static inline int pisa_meta_init(void) { return -EOPNOTSUPP; }
>  static inline bool pisa_enabled(void) { return false; }
> +static inline bool pisa_user_enabled(void) { return false; }
>  static inline bool is_pisa_pfn(unsigned long pfn) { return false; }
>  static inline bool is_pisa_page(const struct page *page) { return false; }
>  static inline bool is_pisa_phys(phys_addr_t phys) { return false; }
> @@ -228,6 +239,13 @@ static inline bool pisa_page_can_recycle(const struct page *page)
>  	return true;
>  }
>
> +static inline bool pisa_upgrade_user_pages(struct iov_iter *iter,
> +					   struct page **pages, size_t size,
> +					   size_t offset)
> +{
> +	return false;
> +}
> +
>  #endif /* CONFIG_PISA */
>
>  #if defined(CONFIG_PISA) && defined(CONFIG_IOMMU_DMA)
> diff --git a/include/linux/swap.h b/include/linux/swap.h
> index 4f686709d71f0..3736786ccc5a3 100644
> --- a/include/linux/swap.h
> +++ b/include/linux/swap.h
> @@ -314,6 +314,7 @@ extern unsigned long totalreserve_pages;
>  /* linux/mm/folio.c */
>  void folio_add_lru(struct folio *folio);
>  void folio_mark_accessed(struct folio *folio);
> +void lru_add_drain(void);
>  void lru_add_drain_all(void);
>
>  enum lru_cache_drained {
> diff --git a/include/uapi/asm-generic/mman-common.h b/include/uapi/asm-generic/mman-common.h
> index ef1c27fa3c570..c4332db8e56ff 100644
> --- a/include/uapi/asm-generic/mman-common.h
> +++ b/include/uapi/asm-generic/mman-common.h
> @@ -81,6 +81,8 @@
>
>  #define MADV_GUARD_INSTALL 102		/* fatal signal on access to range */
>  #define MADV_GUARD_REMOVE 103		/* unguard range */
> +#define MADV_PISA		210	/* move pages into PISA memory */
> +#define MADV_NOPISA		211	/* move pages out of PISA memory */
>
>  /* compatibility flags */
>  #define MAP_FILE	0
> diff --git a/mm/Makefile b/mm/Makefile
> index 7d624e4bad456..d6b54a6eb3dc0 100644
> --- a/mm/Makefile
> +++ b/mm/Makefile
> @@ -147,7 +147,7 @@ obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o
>  obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
>  obj-$(CONFIG_EXECMEM) += execmem.o
>  obj-$(CONFIG_PISA) += pisa.o
> -pisa-y := pisa-meta.o pisa-alloc.o pisa-node.o
> +pisa-y := pisa-meta.o pisa-alloc.o pisa-node.o pisa-user.o
>  obj-$(CONFIG_PISA_META_KUNIT_TEST) += pisa-kunit.o
>  obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
>  obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 7791872f14af7..ffa3912f2418b 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -4113,7 +4113,8 @@ static void folio_reset_partially_mapped(struct folio *folio)
>
>  static int __folio_freeze_split_anon(struct folio *folio,
>  		unsigned int new_order, struct page *split_at, bool do_lru,
> -		struct list_head *list, enum split_type split_type)
> +		struct list_head *list, enum split_type split_type,
> +		bool keep_zero)
>  {
>  	struct folio *end_folio = folio_next(folio);
>  	struct swap_cluster_info *ci = NULL;
> @@ -4207,7 +4208,13 @@ static int __folio_freeze_split_anon(struct folio *folio,
>  out_no_split:
>  	local_irq_enable();
>  	if (anon_vma) {
> -		if (!ret && !folio_is_device_private(folio))
> +		/*
> +		 * Zero-filled anon subpages are normally unmapped, freed and
> +		 * replaced by the shared zeropage. A @keep_zero caller needs
> +		 * the subpages themselves, e.g. to migrate them before a
> +		 * device writes to them.
> +		 */
> +		if (!ret && !folio_is_device_private(folio) && !keep_zero)
>  			ttu_flags = TTU_USE_SHARED_ZEROPAGE;
>  		remap_anon_folio(folio, 1 << old_order, ttu_flags);
>  	}
> @@ -4387,6 +4394,7 @@ static int __folio_freeze_split_file(struct folio *folio,
>   * @lock_at: a page within @folio to be left locked to caller
>   * @list: after-split folios will be put on it if non NULL
>   * @split_type: perform uniform split or not (non-uniform split)
> + * @keep_zero: do not remap zero-filled anon subpages to the shared zeropage
>   *
>   * It calls __split_frozen_folio() to perform uniform and non-uniform split.
>   * It is in charge of checking whether the split is supported or not and
> @@ -4402,7 +4410,8 @@ static int __folio_freeze_split_file(struct folio *folio,
>   */
>  static int __folio_split(struct folio *folio, unsigned int new_order,
>  		struct page *split_at, struct page *lock_at,
> -		struct list_head *list, enum split_type split_type)
> +		struct list_head *list, enum split_type split_type,
> +		bool keep_zero)
>  {
>  	struct folio *end_folio = folio_next(folio);
>  	const bool is_anon = folio_test_anon(folio);
> @@ -4432,7 +4441,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order,
>
>  	if (is_anon)
>  		ret = __folio_freeze_split_anon(folio, new_order, split_at,
> -						true, list, split_type);
> +						true, list, split_type,
> +						keep_zero);
>  	else
>  		ret = __folio_freeze_split_file(folio, new_order, split_at,
>  						list, split_type);
> @@ -4495,7 +4505,7 @@ int folio_split_unmapped(struct folio *folio, unsigned int new_order)
>  	VM_WARN_ON_ONCE_FOLIO(!folio_test_anon(folio), folio);
>
>  	return __folio_freeze_split_anon(folio, new_order, &folio->page,
> -					 false, NULL, SPLIT_TYPE_UNIFORM);
> +					 false, NULL, SPLIT_TYPE_UNIFORM, false);
>  }
>
>  /*
> @@ -4551,7 +4561,7 @@ int __split_huge_page_to_list_to_order(struct page *page, struct list_head *list
>  	struct folio *folio = page_folio(page);
>
>  	return __folio_split(folio, new_order, &folio->page, page, list,
> -			     SPLIT_TYPE_UNIFORM);
> +			     SPLIT_TYPE_UNIFORM, false);
>  }
>
>  /**
> @@ -4582,7 +4592,7 @@ int folio_split(struct folio *folio, unsigned int new_order,
>  		struct page *split_at, struct list_head *list)
>  {
>  	return __folio_split(folio, new_order, split_at, &folio->page, list,
> -			     SPLIT_TYPE_NON_UNIFORM);
> +			     SPLIT_TYPE_NON_UNIFORM, false);
>  }
>
>  /**
> @@ -4617,6 +4627,18 @@ int split_folio_to_list(struct folio *folio, struct list_head *list)
>  	return split_huge_page_to_list_to_order(&folio->page, list, 0);
>  }
>
> +/*
> + * As split_folio_to_list(), but zero-filled subpages of an anon folio stay
> + * mapped instead of being remapped to the shared zeropage. For callers that
> + * are about to take write references on the subpages: PISA uses it to migrate
> + * the pages of an I/O buffer, e.g. the target of an O_DIRECT read.
> + */
> +int split_folio_to_list_keep_zero(struct folio *folio, struct list_head *list)
> +{
> +	return __folio_split(folio, 0, &folio->page, &folio->page, list,
> +			     SPLIT_TYPE_UNIFORM, true);
> +}
> +
>  /*
>   * __folio_unqueue_deferred_split() is not to be called directly:
>   * the folio_unqueue_deferred_split() inline wrapper in mm/internal.h
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 0214efd2a014f..46981ad295aa5 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -6,6 +6,7 @@
>   * Copyright (C) 2002  Christoph Hellwig
>   */
>
> +#include <linux/pisa-mm.h>
>  #include <linux/mman.h>
>  #include <linux/pagemap.h>
>  #include <linux/syscalls.h>
> @@ -1393,6 +1394,11 @@ static int madvise_vma_behavior(struct madvise_behavior *madv_behavior)
>  		return madvise_guard_install(madv_behavior);
>  	case MADV_GUARD_REMOVE:
>  		return madvise_guard_remove(madv_behavior);
> +#ifdef MADV_PISA
> +	case MADV_PISA:
> +	case MADV_NOPISA:
> +		return pisa_madvise(vma, range->start, range->end, behavior == MADV_PISA);
> +#endif
>
>  	/* The below behaviours update VMAs via madvise_update_vma(). */
>
> @@ -1582,6 +1588,11 @@ madvise_behavior_valid(int behavior)
>  	case MADV_HWPOISON:
>  #endif
>  		return true;
> +#ifdef MADV_PISA
> +	case MADV_PISA:
> +	case MADV_NOPISA:
> +		return IS_ENABLED(CONFIG_PISA);
> +#endif
>
>  	default:
>  		return false;
> @@ -1596,6 +1607,10 @@ static bool process_madvise_remote_valid(int behavior)
>  	case MADV_PAGEOUT:
>  	case MADV_WILLNEED:
>  	case MADV_COLLAPSE:
> +#ifdef MADV_PISA
> +	case MADV_PISA:
> +	case MADV_NOPISA:
> +#endif
>  		return true;
>  	default:
>  		return false;
> @@ -1782,6 +1797,10 @@ static enum madvise_lock_mode get_lock_mode(struct madvise_behavior *madv_behavi
>  	case MADV_POPULATE_READ:
>  	case MADV_POPULATE_WRITE:
>  	case MADV_COLLAPSE:
> +#ifdef MADV_PISA
> +	case MADV_PISA:
> +	case MADV_NOPISA:
> +#endif
>  		return MADVISE_MMAP_READ_LOCK;
>  	case MADV_GUARD_INSTALL:
>  	case MADV_GUARD_REMOVE:
> diff --git a/mm/pisa-alloc.c b/mm/pisa-alloc.c
> index ccba2bf01bd59..2213a56de1242 100644
> --- a/mm/pisa-alloc.c
> +++ b/mm/pisa-alloc.c
> @@ -1432,6 +1432,8 @@ static int __init pisa_init(void)
>  				  "pisa_ro_o%u_n%d", SKB_FRAG_PAGE_ORDER, nid);
>  	}
>
> +	pisa_user_boot_enable();
> +
>  	/* Dedicated WQ_MEM_RECLAIM workqueue so reclaim never blocks behind system_wq. */
>  	pisa_wq = alloc_workqueue("pisa", WQ_MEM_RECLAIM, 0);
>  	if (!pisa_wq)
> diff --git a/mm/pisa-priv.h b/mm/pisa-priv.h
> index 9f01173d30f5f..b18d461198ac5 100644
> --- a/mm/pisa-priv.h
> +++ b/mm/pisa-priv.h
> @@ -133,6 +133,7 @@ static inline long pisa_alloc_held_pages(const struct pisa_alloc *pa)
>  struct pisa_node {
>  	unsigned long		next_alloc;	/* atomic alloc backoff jiffies */
>  	struct pisa_alloc	*pa[2][PISA_MAX_ALLOC_ORDER + 1]; /* [0]=RW, [1]=RO */
> +	struct pisa_alloc	*user_pa;	/* user page migration pa */
>
>  	spinlock_t		lock;		/* protects @pages and @nr_pages */
>  	unsigned int		nr_pages;	/* entries in @pages */
> @@ -245,6 +246,7 @@ void pisa_schedule_scrub_all(void);
>  struct pisa_meta *pisa_reservoir_pop(int nid, gfp_t gfp, bool *clean);
>  unsigned long pisa_reservoir_count(void);
>  unsigned long pisa_reservoir_shrink(struct list_head *release_list, unsigned long nr_to_scan);
> +void pisa_user_boot_enable(void);
>
>  #endif /* CONFIG_PISA */
>  #endif /* _MM_PISA_PRIV_H */
> diff --git a/mm/pisa-user.c b/mm/pisa-user.c
> new file mode 100644
> index 0000000000000..6c3efaee4ecda
> --- /dev/null
> +++ b/mm/pisa-user.c
> @@ -0,0 +1,394 @@
> +// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
> +/*
> + * PISA user and page-cache folios: allocate or migrate order-0 folios into
> + * per-node PISA allocators so zero-copy TX and block I/O use the PISA IOMMU mappings.
> + */
> +#include <linux/pisa.h>
> +#include <linux/gfp.h>
> +#include <linux/huge_mm.h>
> +#include <linux/migrate.h>
> +#include <linux/mm.h>
> +#include <linux/moduleparam.h>
> +#include <linux/nodemask.h>
> +#include <linux/pagewalk.h>
> +#include <linux/swap.h>
> +#include <linux/uio.h>
> +
> +#include "pisa-priv.h"
> +
> +/* Soft cap on user (PTE) pages held in PISA blocks, in PISA pages (0 = disabled). */
> +static unsigned long pisa_user_max_pages __read_mostly;
> +DEFINE_STATIC_KEY_FALSE(pisa_user_key);
> +
> +/* Create per-node user allocators and enable pisa_user_key (serialized by param_lock/init). */
> +static int pisa_user_init(void)
> +{
> +	int nid;
> +
> +	if (pisa_user_enabled())
> +		return 0;
> +	if (pisa_meta_init())
> +		return -ENOMEM;
> +	for_each_node_state(nid, N_MEMORY) {
> +		if (!pisa_alloc_ensure(&pisa_nodes[nid].user_pa, nid, 0,
> +				       DMA_TO_DEVICE, false, "user_tx_n%d", nid))
> +			return -ENOMEM;
> +	}
> +	static_branch_enable(&pisa_user_key);
> +	return 0;
> +}
> +
> +static int pisa_user_max_set(const char *val, const struct kernel_param *kp)
> +{
> +	unsigned long max_pages;
> +	int ret;
> +
> +	ret = kstrtoul(val, 0, &max_pages);
> +	if (ret)
> +		return ret;
> +	WRITE_ONCE(pisa_user_max_pages, max_pages);
> +	if (!max_pages) {
> +		static_branch_disable(&pisa_user_key);
> +	} else if (pisa_nodes[first_memory_node].pa[0][0] && pisa_user_init()) {
> +		WRITE_ONCE(pisa_user_max_pages, 0);
> +		return -ENOMEM;
> +	}
> +	return 0;
> +}
> +
> +static const struct kernel_param_ops pisa_user_max_ops = {
> +	.set = pisa_user_max_set,
> +	.get = param_get_ulong,
> +};
> +module_param_cb(user_max_pages, &pisa_user_max_ops, &pisa_user_max_pages, 0644);
> +
> +/*
> + * Initialize per-node user allocators if pisa.user_max_pages was set on
> + * the kernel command line before pisa_init() ran.
> + */
> +void __init pisa_user_boot_enable(void)
> +{
> +	if (READ_ONCE(pisa_user_max_pages))
> +		pisa_user_init();
> +}
> +
> +/* User pages currently held in PISA blocks. */
> +static unsigned long pisa_user_in_use(void)
> +{
> +	unsigned long used = 0;
> +	int nid;
> +
> +	for_each_node_state(nid, N_MEMORY) {
> +		/* Pairs with smp_store_release() in pisa_user_init(). */
> +		const struct pisa_alloc *pa = smp_load_acquire(&pisa_nodes[nid].user_pa);
> +
> +		used += pa ? READ_ONCE(pa->blocks_in_use) : 0;
> +	}
> +	return used;
> +}
> +
> +/* Pages that can still be moved in before reaching pisa.user_max_pages. */
> +static long pisa_user_budget(void)
> +{
> +	return (long)((READ_ONCE(pisa_user_max_pages) << PISA_ORDER) - pisa_user_in_use());
> +}
> +
> +/* Is @folio worth moving into (@to_pisa) or out of PISA memory? */
> +static bool pisa_user_want(struct folio *folio, bool to_pisa)
> +{
> +	if (!folio_test_anon(folio) || folio_is_zone_device(folio) ||
> +	    folio_test_ksm(folio) || is_huge_zero_folio(folio) ||
> +	    (folio_test_large(folio) && !to_pisa) ||
> +	    folio_maybe_mapped_shared(folio))
> +		return false;
> +	return is_pisa_pfn(folio_pfn(folio)) != to_pisa;
> +}
> +
> +/* Return true if @folio has references beyond its mappings, swap cache, and @extra. */
> +static bool pisa_user_busy(struct folio *folio, int extra)
> +{
> +	return folio_ref_count(folio) > folio_expected_ref_count(folio) + extra;
> +}
> +
> +struct pisa_user_mtc {
> +	gfp_t	gfp;
> +	long	budget;
> +};
> +
> +/* new_folio_t for migration into PISA memory. */
> +static struct folio *pisa_user_new_folio(struct folio *src, unsigned long private)
> +{
> +	struct pisa_user_mtc *mtc = (void *)private;
> +	int nid = folio_nid(src);
> +	struct page *page;
> +
> +	/* Callers split large folios first (see pisa_user_split()); none is expected here. */
> +	if (mtc->budget <= 0 || folio_test_large(src))
> +		return NULL;
> +	page = pisa_alloc(pisa_nodes[nid].user_pa, mtc->gfp);
> +	if (!page)
> +		return NULL;
> +	mtc->budget--;
> +	return page_folio(page);
> +}
> +
> +/* free_folio_t: give back an unused destination block and the budget it was charged. */
> +static void pisa_user_put_folio(struct folio *dst, unsigned long private)
> +{
> +	struct pisa_user_mtc *mtc = (void *)private;
> +
> +	mtc->budget++;
> +	folio_put(dst);
> +}
> +
> +/* new_folio_t for migration out of PISA memory: allocators never serve these. */
> +static struct folio *pisa_user_old_folio(struct folio *src, unsigned long private)
> +{
> +	return __folio_alloc_node(GFP_HIGHUSER_MOVABLE | __GFP_NOWARN, 0, folio_nid(src));
> +}
> +
> +/* Migrate the isolated folios on @list; failures are put back. */
> +static void pisa_user_migrate(struct list_head *list, bool to_pisa, gfp_t gfp)
> +{
> +	struct pisa_user_mtc mtc = { .gfp = gfp, .budget = LONG_MAX };
> +	enum migrate_mode mode = MIGRATE_SYNC;
> +
> +	if (list_empty(list))
> +		return;
> +	if (to_pisa) {
> +		mtc.budget = pisa_user_budget();
> +		if (!gfpflags_allow_blocking(gfp))
> +			mode = MIGRATE_ASYNC;
> +	}
> +	if (migrate_pages(list, to_pisa ? pisa_user_new_folio : pisa_user_old_folio,
> +			  to_pisa ? pisa_user_put_folio : NULL, (unsigned long)&mtc,
> +			  mode, MR_SYSCALL, NULL))
> +		putback_movable_pages(list);
> +}
> +
> +/*
> + * Split large anon @folio to order 0, with the caller holding the only extra
> + * reference. Zero-filled subpages stay mapped instead of being replaced by the
> + * shared zeropage, so that a GUP for write that follows (e.g. an O_DIRECT
> + * read) finds them. After-split folios go to @list, or to the LRU if NULL.
> + */
> +static bool pisa_user_split(struct folio *folio, struct list_head *list, bool can_wait)
> +{
> +	int ret;
> +
> +	if (can_wait)
> +		folio_lock(folio);
> +	else if (!folio_trylock(folio))
> +		return false;
> +	ret = split_folio_to_list_keep_zero(folio, list);
> +	folio_unlock(folio);
> +	return !ret;
> +}
> +
> +/*
> + * Split isolated large @folio and move to @out the subpages found in @pages.
> + * The others, or all of @folio if @split is false or the split fails, go back
> + * to the LRU. Returns the number of subpages moved to @out.
> + */
> +static unsigned int pisa_user_split_isolated(struct folio *folio, struct list_head *out,
> +					     struct page **pages, unsigned int nr, bool split)
> +{
> +	const struct page *lo = NULL, *hi = NULL;
> +	unsigned int kept = 0;
> +	struct folio *f, *next;
> +	LIST_HEAD(pieces);
> +
> +	/* @folio is held isolated, so page_folio() of its subpages is stable. */
> +	for (unsigned int i = 0; i < nr; i++) {
> +		if (page_folio(pages[i]) != folio)
> +			continue;
> +		if (!lo || pages[i] < lo)
> +			lo = pages[i];
> +		if (!hi || pages[i] > hi)
> +			hi = pages[i];
> +	}
> +	list_move(&folio->lru, &pieces);
> +	if (split && pisa_user_split(folio, &pieces, false)) {
> +		list_for_each_entry_safe(f, next, &pieces, lru) {
> +			if (!folio_test_large(f) && &f->page >= lo && &f->page <= hi) {
> +				list_move_tail(&f->lru, out);
> +				kept++;
> +			}
> +		}
> +	}
> +	putback_movable_pages(&pieces);
> +	return kept;
> +}
> +
> +/**
> + * pisa_upgrade_user_pages - Migrate extracted user @pages into PISA
> + * @iter: Iterator from which @pages were just extracted
> + * @pages: Extracted page array, one reference each
> + * @size: Number of bytes extracted from @iter
> + * @offset: Byte offset within @pages[0]
> + *
> + * Return: false if no pages were isolated (@pages and @iter are untouched);
> + * true otherwise, so the caller must re-extract @pages from the reverted @iter.
> + */
> +bool pisa_upgrade_user_pages(struct iov_iter *iter, struct page **pages,
> +			     size_t size, size_t offset)
> +{
> +	unsigned int nr = DIV_ROUND_UP(offset + size, PAGE_SIZE);
> +	unsigned int want = 0;
> +	struct folio *f, *next;
> +	bool drained = false;
> +	LIST_HEAD(small);
> +	LIST_HEAD(list);
> +	long budget;
> +
> +	if (!pisa_user_enabled())
> +		return false;
> +	/*
> +	 * First pass: check if any page needs migration and whether budget is
> +	 * available before draining per-CPU LRU caches or isolating folios.
> +	 */
> +	for (unsigned int i = 0; i < nr; i++)
> +		want += pisa_user_want(page_folio(pages[i]), true);
> +	if (!want || pisa_user_budget() <= 0)
> +		return false;
> +	for (unsigned int i = 0; i < nr; i++) {
> +		struct folio *folio = page_folio(pages[i]);
> +		unsigned int refs = 0;
> +
> +		if (!pisa_user_want(folio, true))
> +			continue;
> +		/* Drain per-CPU LRU batches once so freshly faulted folios can be isolated. */
> +		if (!drained && !folio_test_lru(folio)) {
> +			lru_add_drain();
> +			drained = true;
> +		}
> +		if (!folio_test_lru(folio))
> +			continue;
> +		for (unsigned int j = 0; j < nr; j++)
> +			refs += page_folio(pages[j]) == folio;
> +		/* The references from iov_iter_get_pages2() are expected. */
> +		if (folio_maybe_dma_pinned(folio) || pisa_user_busy(folio, refs))
> +			continue;
> +		if (isolate_folio_to_list(folio, &list))
> +			list_move_tail(&folio->lru, &list);
> +	}
> +	if (list_empty(&list))
> +		return false;
> +	/*
> +	 * Release the caller's references on @pages before migrating
> +	 * (migrate_pages() rejects folios with extra references).
> +	 */
> +	for (unsigned int i = 0; i < nr; i++)
> +		put_page(pages[i]);
> +	/*
> +	 * Split large folios while budget remains and keep only their subpages
> +	 * that are in @pages: migrate_pages() would split them regardless of
> +	 * the budget, migrate every subpage and drop the zero-filled ones.
> +	 */
> +	budget = pisa_user_budget();
> +	list_for_each_entry_safe(f, next, &list, lru) {
> +		if (folio_test_large(f))
> +			budget -= pisa_user_split_isolated(f, &small, pages, nr, budget > 0);
> +		else
> +			budget -= !is_pisa_pfn(folio_pfn(f));
> +	}
> +	list_splice_tail(&small, &list);
> +	pisa_user_migrate(&list, true,
> +			  GFP_NOWAIT | __GFP_HARDWALL | __GFP_NOWARN);
> +	/* Rewind @iter so the caller re-extracts the updated page pointers. */
> +	iov_iter_revert(iter, size);
> +	return true;
> +}
> +
> +#define PISA_USER_BATCH	512
> +
> +struct pisa_madvise_walk {
> +	struct list_head	list;	/* isolated folios waiting to be migrated */
> +	unsigned int		n;	/* pages on @list */
> +	bool			to_pisa;
> +};
> +
> +#define PISA_MADVISE_GFP	(GFP_USER | __GFP_NOWARN)
> +
> +/* Isolate the wanted folios mapped in one PMD's range, splitting the large ones first. */
> +static int pisa_madvise_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end,
> +				  struct mm_walk *walk)
> +{
> +	struct pisa_madvise_walk *w = walk->private;
> +	bool to_pisa = w->to_pisa;
> +	struct folio_walk fw;
> +	struct folio *folio;
> +
> +	while (addr < end) {
> +		unsigned long nr = 1;
> +
> +		if (fatal_signal_pending(current))
> +			return -EINTR;
> +		/* Stop the walk, without an error, once the cap is reached. */
> +		if (to_pisa && pisa_user_budget() <= (long)w->n)
> +			return 1;
> +		folio = folio_walk_start(&fw, walk->vma, addr, 0);
> +		if (folio) {
> +			bool want = pisa_user_want(folio, to_pisa) && folio_test_lru(folio);
> +			bool busy = want && pisa_user_busy(folio, 0);
> +			bool split = want && !busy && folio_test_large(folio);
> +
> +			/* A PMD mapping covers the rest of the range, a PTE maps one page. */
> +			if (fw.level != FW_LEVEL_PTE)
> +				nr = (end - addr) >> PAGE_SHIFT;
> +			if (split) {
> +				folio_get(folio);
> +			} else if (want && !busy && isolate_folio_to_list(folio, &w->list)) {
> +				list_move_tail(&folio->lru, &w->list);
> +				w->n++;
> +			}
> +			folio_walk_end(&fw, walk->vma);
> +			if (split) {
> +				split = pisa_user_split(folio, NULL, true);
> +				folio_put(folio);
> +				/* Look at @addr again: it now maps an order-0 folio. */
> +				if (split)
> +					continue;
> +			}
> +		}
> +		if (w->n >= PISA_USER_BATCH) {
> +			pisa_user_migrate(&w->list, to_pisa, PISA_MADVISE_GFP);
> +			w->n = 0;
> +		}
> +		addr += nr << PAGE_SHIFT;
> +		cond_resched();
> +	}
> +	return 0;
> +}
> +
> +static const struct mm_walk_ops pisa_madvise_walk_ops = {
> +	.pmd_entry	= pisa_madvise_pmd_entry,
> +	.walk_lock	= PGWALK_RDLOCK,
> +};
> +
> +/**
> + * pisa_madvise - MADV_PISA / MADV_NOPISA on [@start, @end) of @vma
> + * @vma: VMA, with the mmap lock held for reading
> + * @start: Start address
> + * @end: End address
> + * @to_pisa: true to move present pages into PISA, false to move them out
> + *
> + * Return: 0, -EINVAL for unsupported mappings, or -EOPNOTSUPP if disabled.
> + */
> +int pisa_madvise(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool to_pisa)
> +{
> +	struct pisa_madvise_walk w = { .list = LIST_HEAD_INIT(w.list), .to_pisa = to_pisa };
> +	int ret;
> +
> +	if (vma->vm_flags & (VM_SHARED | VM_IO | VM_PFNMAP | VM_HUGETLB))
> +		return -EINVAL;
> +	if (to_pisa && !pisa_user_enabled())
> +		return -EOPNOTSUPP;
> +	if (!pisa_enabled() && !to_pisa)
> +		return 0;
> +	lru_add_drain();
> +	/* The page walker skips the unpopulated page-table levels of sparse mappings. */
> +	ret = walk_page_range_vma(vma, start, end, &pisa_madvise_walk_ops, &w);
> +	pisa_user_migrate(&w.list, to_pisa, PISA_MADVISE_GFP);
> +	return ret < 0 ? ret : 0;
> +}
> --
> 2.56.1.809.ga0b61035b5-goog
>

--
Cheers, Lorenzo

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
  2026-11-20 13:50   ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
@ 2026-11-20 14:18     ` Lorenzo Stoakes (ARM)
  2026-11-20 14:19     ` Luigi Rizzo
  1 sibling, 0 replies; 32+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-11-20 14:18 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: Luigi Rizzo, Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo, Liam R . Howlett,
	Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
	Zi Yan, Harry Yoo, Christoph Lameter, David Rientjes,
	Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara, Jann Horn,
	Catalin Marinas, Greg Kroah-Hartman, Rafael J . Wysocki,
	Danilo Krummrich, Jonathan Corbet, Simon Horman,
	Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
	Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
	Alexander Lobakin, Christoph Hellwig, Dragos Tatulea,
	Naman Gulati, Pranjal Shrivastava, Mostafa Saleh, Randy Dunlap,
	linux-mm, iommu, netdev, linux-block, linux-fsdevel, driver-core,
	linux-arm-kernel, linux-api, linux-doc, linux-kernel

On Fri, Nov 20, 2026 at 09:50:13AM -0400, Jason Gunthorpe wrote:
> On Fri, Nov 20, 2026 at 01:06:30AM +0000, Luigi Rizzo wrote:
> > This is v3 of the RFC previously posted as DMA_PMD [0], now renamed PISA
> > ("Pre-mapped IOVA Superpage Allocator"). The text below follows
> > Documentation/core-api/pisa.rst (patch 25/25), which has the details.
>
> I thought the conclusion was to just do this stuff in netdev pagepool,
> now you come with an enormous AI series spread across the entire
> kernel?

Welcome to 2026 :) It's so exhausting.

>
> > - Needs bounce buffers: in Confidential Computing VMs, PISA allocators
> >   pre-decrypt 2MB pages in place, avoiding swiotlb bounce-buffer copies.
>
> It seems this was ignored too. We want the bounce buffers in CC. They
> serve a security purpose. You cannot loose control of the shared
> memory buffers have have them leak into SKBs, user mappings and so
> on. It is not secure.
>
> I don't see how any of this is valuable or necessary for CC,
> especially when we really don't even expect to have an vIOMMU for CC
> very often..
>
> Jason

Agreed, this whole thing needs to be rethought.

Will reply separately to the cover I think :)

--
Cheers, Lorenzo

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
  2026-11-20 13:50   ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
  2026-11-20 14:18     ` Lorenzo Stoakes (ARM)
@ 2026-11-20 14:19     ` Luigi Rizzo
  2026-11-20 14:36       ` Lorenzo Stoakes (ARM)
  1 sibling, 1 reply; 32+ messages in thread
From: Luigi Rizzo @ 2026-11-20 14:19 UTC (permalink / raw)
  To: Jason Gunthorpe
  Cc: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo, Lorenzo Stoakes,
	Liam R . Howlett, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Brendan Jackman, Zi Yan, Harry Yoo, Christoph Lameter,
	David Rientjes, Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara,
	Jann Horn, Catalin Marinas, Greg Kroah-Hartman,
	Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
	Simon Horman, Jesper Dangaard Brouer, Ilias Apalodimas,
	Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
	Harshitha Ramamurthy, Alexander Lobakin, Christoph Hellwig,
	Dragos Tatulea, Naman Gulati, Pranjal Shrivastava, Mostafa Saleh,
	Randy Dunlap, linux-mm, iommu, netdev, linux-block, linux-fsdevel,
	driver-core, linux-arm-kernel, linux-api, linux-doc, linux-kernel

On Fri, Nov 20, 2026 at 2:50 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Nov 20, 2026 at 01:06:30AM +0000, Luigi Rizzo wrote:
> > This is v3 of the RFC previously posted as DMA_PMD [0], now renamed PISA
> > ("Pre-mapped IOVA Superpage Allocator"). The text below follows
> > Documentation/core-api/pisa.rst (patch 25/25), which has the details.
>
> I thought the conclusion was to just do this stuff in netdev pagepool,
> now you come with an enormous AI series spread across the entire
> kernel?

There were surely comments claiming nothing outside net/ cares or needs this.

I respectfully disagree, witness is the fact that e.g. there was already an
io_uring proposal which is smaller, but does not address applications
that want to use the standard disk I/O paths (libaio, buffer cache etc)

Nothing has changed from the initial series in terms of size, use cases or
subsystems involved. Large files are placed where they should,
the series covers the same three use cases (disk, network, CoCo), and
touches three parts which it has to touch: mm (most of it; it is a
memory allocator
extension after all), dma (it hooks into map and unmap, not a large change)
and net (the pagepool change is a couple of lines; the tx path needs to
touch more places because the current stack assumes all paths have
the same I/O requirements).

For CoCo, there are very fine grained controls so the bounce buffers
can be enabled only when they don't pose a risk, which in turn
depends on the threat model.

Specifically, for network tx buffers or disk writes, the user-to-kernel copy
is basically a bounce buffer provided it is used correctly and not leaked
elsewhere (which is the design; of course the implementation
needs to be verified and possibly fixed so that it matches the design).

cheers
luigi

> > - Needs bounce buffers: in Confidential Computing VMs, PISA allocators
> >   pre-decrypt 2MB pages in place, avoiding swiotlb bounce-buffer copies.
>
> It seems this was ignored too. We want the bounce buffers in CC. They
> serve a security purpose. You cannot loose control of the shared
> memory buffers have have them leak into SKBs, user mappings and so
> on. It is not secure.
>
> I don't see how any of this is valuable or necessary for CC,
> especially when we really don't even expect to have an vIOMMU for CC
> very often..
>
> Jason

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
  2026-11-20 14:19     ` Luigi Rizzo
@ 2026-11-20 14:36       ` Lorenzo Stoakes (ARM)
  0 siblings, 0 replies; 32+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-11-20 14:36 UTC (permalink / raw)
  To: Luigi Rizzo
  Cc: Jason Gunthorpe, Andrew Morton, David Hildenbrand,
	Johannes Weiner, Vlastimil Babka, Joerg Roedel, Will Deacon,
	Robin Murphy, Marek Szyprowski, Jens Axboe, David S . Miller,
	Eric Dumazet, Jakub Kicinski, Paolo Abeni, Luigi Rizzo,
	Liam R . Howlett, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
	Brendan Jackman, Zi Yan, Harry Yoo, Christoph Lameter,
	David Rientjes, Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara,
	Jann Horn, Catalin Marinas, Greg Kroah-Hartman,
	Rafael J . Wysocki, Danilo Krummrich, Jonathan Corbet,
	Simon Horman, Jesper Dangaard Brouer, Ilias Apalodimas,
	Willem de Bruijn, Kuniyuki Iwashima, Joshua Washington,
	Harshitha Ramamurthy, Alexander Lobakin, Christoph Hellwig,
	Dragos Tatulea, Naman Gulati, Pranjal Shrivastava, Mostafa Saleh,
	Randy Dunlap, linux-mm, iommu, netdev, linux-block, linux-fsdevel,
	driver-core, linux-arm-kernel, linux-api, linux-doc, linux-kernel

On Fri, Nov 20, 2026 at 03:19:34PM +0100, Luigi Rizzo wrote:
> On Fri, Nov 20, 2026 at 2:50 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >
> > On Fri, Nov 20, 2026 at 01:06:30AM +0000, Luigi Rizzo wrote:
> > > This is v3 of the RFC previously posted as DMA_PMD [0], now renamed PISA
> > > ("Pre-mapped IOVA Superpage Allocator"). The text below follows
> > > Documentation/core-api/pisa.rst (patch 25/25), which has the details.
> >
> > I thought the conclusion was to just do this stuff in netdev pagepool,
> > now you come with an enormous AI series spread across the entire
> > kernel?
>
> There were surely comments claiming nothing outside net/ cares or needs this.
>
> I respectfully disagree, witness is the fact that e.g. there was already an
> io_uring proposal which is smaller, but does not address applications
> that want to use the standard disk I/O paths (libaio, buffer cache etc)
>
> Nothing has changed from the initial series in terms of size, use cases or
> subsystems involved. Large files are placed where they should,
> the series covers the same three use cases (disk, network, CoCo), and
> touches three parts which it has to touch: mm (most of it; it is a
> memory allocator
> extension after all), dma (it hooks into map and unmap, not a large change)
> and net (the pagepool change is a couple of lines; the tx path needs to
> touch more places because the current stack assumes all paths have
> the same I/O requirements).
>
> For CoCo, there are very fine grained controls so the bounce buffers
> can be enabled only when they don't pose a risk, which in turn
> depends on the threat model.
>
> Specifically, for network tx buffers or disk writes, the user-to-kernel copy
> is basically a bounce buffer provided it is used correctly and not leaked
> elsewhere (which is the design; of course the implementation
> needs to be verified and possibly fixed so that it matches the design).
>
> cheers
> luigi

NAK.

Cheers, Lorenzo

--
Cheers, Lorenzo

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo
  2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
                     ` (25 preceding siblings ...)
  2026-11-20 13:50   ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
@ 2026-11-20 14:54   ` Lorenzo Stoakes (ARM)
  26 siblings, 0 replies; 32+ messages in thread
From: Lorenzo Stoakes (ARM) @ 2026-11-20 14:54 UTC (permalink / raw)
  To: Luigi Rizzo
  Cc: Andrew Morton, David Hildenbrand, Johannes Weiner,
	Vlastimil Babka, Joerg Roedel, Will Deacon, Robin Murphy,
	Marek Szyprowski, Jens Axboe, David S . Miller, Eric Dumazet,
	Jakub Kicinski, Paolo Abeni, Luigi Rizzo, Liam R . Howlett,
	Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Brendan Jackman,
	Zi Yan, Harry Yoo, Christoph Lameter, David Rientjes,
	Roman Gushchin, Hao Li, Matthew Wilcox, Jan Kara, Jann Horn,
	Catalin Marinas, Greg Kroah-Hartman, Rafael J . Wysocki,
	Danilo Krummrich, Jonathan Corbet, Simon Horman,
	Jesper Dangaard Brouer, Ilias Apalodimas, Willem de Bruijn,
	Kuniyuki Iwashima, Joshua Washington, Harshitha Ramamurthy,
	Alexander Lobakin, Christoph Hellwig, Jason Gunthorpe,
	Dragos Tatulea, Naman Gulati, Pranjal Shrivastava, Mostafa Saleh,
	Randy Dunlap, linux-mm, iommu, netdev, linux-block, linux-fsdevel,
	driver-core, linux-arm-kernel, linux-api, linux-doc, linux-kernel

TL;DR: NAK. All of it. (And yeah, I'm steeling myself for the inevitable 'oh
Lorenzo' :)

Sigh. You know, it's hard being a maintainer, and it's twice as hard lately,
given the endless surge of AI-generated stuff we see every day.

But I am especially frustrated that I have to ask people to do the _absolute
basics_. And from people who are not helping with any of that. Quite the
opposite.

So the basics - you haven't cc'd the right people here and you're sending a
massive new series in-reply-to another. See below for some automatically
generated feedback on that.

In any case, the courteous thing to do would be to familiarise yourself with how
series are sent in mm. And that goes so much more for a series that is so
invasive. And seemingly, AI generated...

And you're sending it when
https://lore.kernel.org/linux-mm/a2f584f6-e382-4876-aa6d-d5a34de6ee59@kernel.org/
is in effect.

Anyway, to the series itself....

This is ludicrous:

>  .../admin-guide/kernel-parameters.txt         |    7 +
>  Documentation/core-api/index.rst              |    1 +
>  Documentation/core-api/pisa.rst               |  485 +++++
>  arch/arm64/include/asm/pgtable.h              |   28 +-
>  block/bio.c                                   |    9 +
>  drivers/base/core.c                           |   78 +
>  drivers/iommu/Makefile                        |    3 +
>  drivers/iommu/dma-iommu.c                     |   49 +-
>  drivers/iommu/iommu.c                         |    7 +-
>  drivers/iommu/pisa-map.c                      |  488 +++++
>  .../ethernet/google/gve/gve_buffer_mgmt_dqo.c |    8 +-
>  drivers/net/ethernet/google/gve/gve_rx.c      |   40 +-
>  include/linux/device.h                        |   32 +
>  include/linux/gfp_types.h                     |   14 +-
>  include/linux/huge_mm.h                       |    7 +
>  include/linux/netdevice.h                     |    4 +
>  include/linux/pisa-iommu.h                    |  141 ++
>  include/linux/pisa-mm.h                       |  205 ++
>  include/linux/pisa.h                          |  356 ++++
>  include/linux/sched.h                         |    4 +
>  include/linux/slab.h                          |   10 +
>  include/linux/swap.h                          |    1 +
>  include/net/libeth/rx.h                       |   11 +-
>  include/net/pisa.h                            |  204 ++
>  include/trace/events/mmflags.h                |   11 +-
>  include/uapi/asm-generic/mman-common.h        |    2 +
>  kernel/dma/direct.c                           |   11 +
>  kernel/exit.c                                 |    2 +
>  kernel/fork.c                                 |    2 +
>  mm/Kconfig                                    |   35 +
>  mm/Makefile                                   |    3 +
>  mm/filemap.c                                  |   12 +-
>  mm/huge_memory.c                              |   36 +-
>  mm/ksm.c                                      |    3 +-
>  mm/madvise.c                                  |   19 +
>  mm/memory.c                                   |    8 +-
>  mm/migrate.c                                  |    3 +-
>  mm/page-writeback.c                           |    9 +
>  mm/page_alloc.c                               |   10 +
>  mm/pisa-alloc.c                               | 1758 +++++++++++++++++
>  mm/pisa-kunit.c                               |  778 ++++++++
>  mm/pisa-meta.c                                |  834 ++++++++
>  mm/pisa-node.c                                |  567 ++++++
>  mm/pisa-priv.h                                |  319 +++
>  mm/pisa-user.c                                |  898 +++++++++
>  mm/readahead.c                                |    3 +-
>  mm/slab.h                                     |    3 +-
>  mm/slab_common.c                              |    2 +-
>  mm/slub.c                                     |   23 +-
>  mm/swapfile.c                                 |    3 +-
>  mm/userfaultfd.c                              |    7 +-
>  net/core/Makefile                             |    1 +
>  net/core/datagram.c                           |   17 +-
>  net/core/page_pool.c                          |    3 +
>  net/core/pisa-net.c                           |  264 +++
>  net/core/skbuff.c                             |   33 +-
>  net/core/sock.c                               |   31 +-
>  net/ipv4/esp4.c                               |    7 +-
>  net/ipv4/ip_output.c                          |   20 +-
>  net/ipv4/raw.c                                |    6 +-
>  net/ipv4/tcp.c                                |   17 +-
>  net/ipv4/tcp_output.c                         |   14 +-
>  net/ipv6/esp6.c                               |    7 +-
>  net/ipv6/ip6_output.c                         |   20 +-
>  net/ipv6/raw.c                                |    6 +-
>  65 files changed, 7907 insertions(+), 92 deletions(-)

I mean ~ +8ksloc. 65 files. These should be clues for you.

>  create mode 100644 Documentation/core-api/pisa.rst
>  create mode 100644 drivers/iommu/pisa-map.c
>  create mode 100644 include/linux/pisa-iommu.h
>  create mode 100644 include/linux/pisa-mm.h
>  create mode 100644 include/linux/pisa.h
>  create mode 100644 include/net/pisa.h
>  create mode 100644 mm/pisa-alloc.c
>  create mode 100644 mm/pisa-kunit.c
>  create mode 100644 mm/pisa-meta.c
>  create mode 100644 mm/pisa-node.c
>  create mode 100644 mm/pisa-priv.h
>  create mode 100644 mm/pisa-user.c

And here is a perfect clue. 'pisa-mm.h'. 'pisa-alloc.c'. You are creating
parallel implementations of core mm mechanics _just for your feature_.

And you didn't even update MAINTAINERS :) though, moving implementation to your
own files and adding hooks elsewhere is its own kind of anti-pattern.

>  create mode 100644 net/core/pisa-net.c

What I'm trying to say, constructively, is that you have gone down the wrong
road here. And it's one we've seen many times before.

You have decided that this feature is exciting and important and really special
and unlike anything else, so it must be implemented through hooks into other
stuff, new special GFP flags, an madvise() hook which amounts to 'I can do what
I want here', your own allocator, your own node code, your own everything.

And I'm sorry but we've seen this before, and it's been a disaster. Hugetlb and
uffd are examples of code like this.

Every feature of this nature imposes a tax. It means that you 'just have to
remember' to consider and deal with feature X when you write code touching Y
because feature X sits outside of the core mm code.

So my constructive suggestion to you (and this is an RFC, so you're asking for
this kind of feedback right? :) is as follows:

Firstly - please rethink using AI to generate massive series like this. You are
not going to get anything like that merged, and it disinclines maintainers from
even looking at your code.

Secondly - think about the problem at an architectural level. Listen to people
like Jason and David who are giving you highly valuable feedback, seriously you
are lucky to get it.

Thirdly - rework it so it _fits in with the existing core mm architecture_.

Finally, please don't send it as an +8 ksloc 65 file monster. Nobody is going to
take a series like that frankly, especially not from a newcomer to the
subsystem who may or may not stick around to maintain it.

IOW - try to find ways to implement it incrementally.

Thanks!

~~

FYI: Issues my scripts picked up:

1. New revisions sent in-reply-to mail

Please always send new revisions of series individually, not in-reply-to
any other email.

The easiest way of accomplishing this is to use b4 [0] (the recommended
way of sending patches to mm), otherwise format patches like this:

For a patch series, e.g. at v2:

$ git format-patch -v2 --cover-letter --thread ...

For an individual patch, e.g. at v2:

$ git format-patch -v2 HEAD~1

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

2. Missing cc's

You are missing cc's. Fixing this is easy with b4 [0] (the recommended way
of sending patches to mm):

$ b4 prep --auto-to-cc

Alternatively, you can use scripts/get_maintainer.pl:

$ scripts/get_maintainer.pl --nogit-fallback <files-or-patches>

[0]: https://b4.docs.kernel.org/en/latest/contributor/send.html

Specifically, the following appear to be missing:

  chrisl@kernel.org
  kasong@tencent.com
  apopple@nvidia.com
  baohua@kernel.org
  baolin.wang@linux.alibaba.com
  baoquan.he@linux.dev
  byungchul@sk.com
  chengming.zhou@linux.dev
  dev.jain@arm.com
  gourry@gourry.net
  joshua.hahnjy@gmail.com
  kas@kernel.org
  lance.yang@linux.dev
  matthew.brost@intel.com
  nico.pache@linux.dev
  nphamcs@gmail.com
  peterx@redhat.com
  ryan.roberts@arm.com
  shikemeng@huaweicloud.com
  usama.arif@linux.dev
  xu.xin@linux.dev
  ying.huang@linux.alibaba.com
  youngjun.park@lge.com

--
Cheers, Lorenzo

^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2026-11-20 14:54 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <20261109182533.3594984-1-lrizzo@google.com>
2026-11-20  1:06 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 10/25] mm: Add background page scrubber for " Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
2026-11-20 14:10     ` Lorenzo Stoakes (ARM)
2026-11-20  1:06   ` [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 15/25] net: Use PISA for page_pool memory Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 23/25] net/gve: " Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation Luigi Rizzo
2026-11-20 13:50   ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
2026-11-20 14:18     ` Lorenzo Stoakes (ARM)
2026-11-20 14:19     ` Luigi Rizzo
2026-11-20 14:36       ` Lorenzo Stoakes (ARM)
2026-11-20 14:54   ` Lorenzo Stoakes (ARM)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).