From: Luigi Rizzo <lrizzo@google.com>
To: Andrew Morton <akpm@linux-foundation.org>,
David Hildenbrand <david@kernel.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Vlastimil Babka <vbabka@kernel.org>,
Joerg Roedel <joro@8bytes.org>, Will Deacon <will@kernel.org>,
Robin Murphy <robin.murphy@arm.com>,
Marek Szyprowski <m.szyprowski@samsung.com>,
Jens Axboe <axboe@kernel.dk>,
"David S . Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@kernel.org>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>,
Luigi Rizzo <rizzo.unipi@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>,
"Liam R . Howlett" <liam@infradead.org>,
Mike Rapoport <rppt@kernel.org>,
Suren Baghdasaryan <surenb@google.com>,
Michal Hocko <mhocko@suse.com>,
Brendan Jackman <brendan.jackman@linux.dev>,
Zi Yan <ziy@nvidia.com>, Harry Yoo <harry@kernel.org>,
Christoph Lameter <cl@gentwo.org>,
David Rientjes <rientjes@google.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Hao Li <hao.li@linux.dev>, Matthew Wilcox <willy@infradead.org>,
Jan Kara <jack@suse.cz>, Jann Horn <jannh@google.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
"Rafael J . Wysocki" <rafael@kernel.org>,
Danilo Krummrich <dakr@kernel.org>,
Jonathan Corbet <corbet@lwn.net>,
Simon Horman <horms@kernel.org>,
Jesper Dangaard Brouer <hawk@kernel.org>,
Ilias Apalodimas <ilias.apalodimas@linaro.org>,
Willem de Bruijn <willemb@google.com>,
Kuniyuki Iwashima <kuniyu@google.com>,
Joshua Washington <joshwash@google.com>,
Harshitha Ramamurthy <hramamurthy@google.com>,
Alexander Lobakin <aleksander.lobakin@intel.com>,
Christoph Hellwig <hch@lst.de>, Jason Gunthorpe <jgg@ziepe.ca>,
Dragos Tatulea <dtatulea@nvidia.com>,
Naman Gulati <namangulati@google.com>,
Pranjal Shrivastava <praan@google.com>,
Mostafa Saleh <smostafa@google.com>,
Randy Dunlap <rdunlap@infradead.org>,
linux-mm@kvack.org, iommu@lists.linux.dev,
netdev@vger.kernel.org, linux-block@vger.kernel.org,
linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev,
linux-arm-kernel@lists.infradead.org, linux-api@vger.kernel.org,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
Luigi Rizzo <lrizzo@google.com>
Subject: [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation
Date: Fri, 20 Nov 2026 01:06:55 +0000 [thread overview]
Message-ID: <20261120010655.467825-26-lrizzo@google.com> (raw)
In-Reply-To: <20261120010655.467825-1-lrizzo@google.com>
Document the PISA IOMMU 2MB-hugepage-backed DMA memory
subsystem in Documentation/core-api/pisa.rst.
Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
.../admin-guide/kernel-parameters.txt | 7 +
Documentation/core-api/index.rst | 1 +
Documentation/core-api/pisa.rst | 485 ++++++++++++++++++
3 files changed, 493 insertions(+)
create mode 100644 Documentation/core-api/pisa.rst
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec2..cce75a9abddad 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5345,6 +5345,13 @@ Kernel parameters
pirq= [SMP,APIC] Manual mp-table setup
See Documentation/arch/x86/i386/IO-APIC.rst.
+ pisa.*= [ARM64,X86-64] Pre-mapped IOVA Superpage Allocator (PISA)
+ tuning parameters (max_pages, max_idle, idle_ms,
+ reservoir_max, scrub_max, compact,
+ target_usage_percent, old_ms, pcp_batch, user_max_pages,
+ scrub_migrate).
+ See Documentation/core-api/pisa.rst.
+
plip= [PPT,NET] Parallel port network link
Format: { parport<nr> | timid | 0 }
See also Documentation/admin-guide/parport.rst.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..4955f477ed797 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
dma-attributes
dma-isa-lpc
swiotlb
+ pisa
mm-api
cgroup
genalloc
diff --git a/Documentation/core-api/pisa.rst b/Documentation/core-api/pisa.rst
new file mode 100644
index 0000000000000..f9690b4a9c96a
--- /dev/null
+++ b/Documentation/core-api/pisa.rst
@@ -0,0 +1,485 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=============================================
+PISA: Pre-mapped IOVA Superpage Allocator
+=============================================
+
+Overview
+========
+
+PISA is an extension to the page allocator: requests with ``__GFP_PISA``
+return memory that makes I/O-related operations cheaper when using
+``DMA-FQ``, ``DMA``, or Confidential Computing ``swiotlb``. I/O devices
+can individually opt in to use PISA for specific buffer classes (e.g. read
+or write data, headers, zerocopy, ...). Relevant kernel subsystems
+(mm, iommu, dma, disk I/O, page cache, the network stack) are modified
+to make use of PISA memory for opt-in devices.
+
+The design and implementation have the following main goals:
+
+- no changes to individual device drivers (or minimal changes for NIC drivers
+ that use custom RX page allocators)
+- minimize changes to base code: in most cases one or a few lines of code
+- safety: PISA memory is only used for devices that opt-in,
+ and it is returned to the buddy allocator only after IOMMU mappings
+ and IOTLB entries have been flushed
+- no dependency on ``struct page`` or ``struct folio``
+- compile-time option to disable or enable PISA
+- zero runtime overhead when no devices opt in (via static keys and inline hooks)
+- minimal overhead when in use (direct PFN lookup in a flat metadata table)
+- keep memory overhead as low as possible (adaptive shrinking and compaction
+ of memory)
+
+Mechanisms and performance benefits
+-----------------------------------
+PISA returns order-N blocks backed by 2MB physically contiguous pages, so it
+can use 2MB IOMMU leaves and reduce the IOTLB pressure by up to 512x.
+PISA maps memory in pre-allocated per-domain IOVA ranges, and recycles
+blocks and mappings until the 2MB page is returned to the buddy allocator.
+This makes most of dma map/unmap a no-op, also avoiding related VM exits
+in overcommitted VMs.
+In Confidential Computing VMs, PISA pages are also decrypted, removing the need
+for bounce buffers.
+The kernel identifies devices and sockets that use an
+IOMMU or Confidential Computing and have opted into PISA, and allocates (or
+migrates) I/O buffers using PISA memory so that DMA mapping and unmapping are
+mostly no-ops. All the above makes performance comparable to ``identity`` mode,
+with isolation better than ``DMA-FQ`` and comparable to ``DMA``.
+
+Devices that do not opt in continue to use their native DMA/DMA-FQ mode.
+
+Comparison with standard DMA modes
+----------------------------------
+
+The table below summarizes how PISA (layered on top of strict ``DMA``)
+compares against pass-through/identity mapping (``identity``), batched
+deferred IOTLB flush (``DMA-FQ``), and strict per-request IOMMU mapping
+(``DMA``):
+
+======================================= ==================== ================== ===============
+Feature ( > means better ) vs ``identity`` vs ``DMA-FQ`` vs ``DMA``
+======================================= ==================== ================== ===============
+``dma_map`` cost ``PISA ~= identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+``dma_unmap`` cost ``PISA == identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+IOTLB friendliness ``na`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Buffer shared across domains ``na`` ``PISA <= DMA-FQ`` ``PISA <= DMA``
+Memory remains accessible to device ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA <= DMA``
+Needs bounce buffers ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Page alloc/free cost ``PISA >= identity`` ``PISA >= DMA-FQ`` ``PISA >= DMA``
+======================================= ==================== ================== ===============
+
+Notes on the comparisons:
+
+- **dma_map / dma_unmap cost:** Once a 2MB PISA page is mapped in a domain's
+ reserved IOVA window, ``dma_map_*()`` is an offset lookup and
+ ``dma_unmap_*()`` is a no-op, avoiding IOVA tree
+ allocation, page-table walks, and synchronous IOTLB shootdowns.
+- **IOTLB friendliness:** PISA maps 2MB IOMMU leaves instead of 4KB leaves,
+ covering 512x more memory per IOTLB entry than ``DMA-FQ`` and ``DMA``.
+ A domain without a 2MB page size maps each PISA page with smaller IOMMU
+ pages: mapping is still done once per PISA page, without this benefit.
+- **Buffer shared across domains:** Even in ``DMA-FQ`` and ``DMA``, IOMMU
+ mappings operate at page granularity, so sub-page buffers from ``task_frag``
+ (shared across all sockets of a task) or ``skb`` head slabs can cause the
+ underlying page to be mapped across multiple domains. PISA exhibits the same
+ behavior, just at 2MB granularity.
+- **Memory remains accessible to device:** In ``identity`` mode, all of RAM is
+ always accessible to devices. In ``DMA-FQ``, pages returned to the buddy
+ allocator remain accessible until the asynchronous flush queue drains. Both
+ PISA and ``DMA`` unmap the page and perform a synchronous IOTLB flush before
+ returning memory to the buddy allocator. In all modes, receive buffers remain mapped
+ and accessible to the device while queued in the rx socket buffer.
+- **Needs bounce buffers:** In Confidential Computing (CoCo) VMs, PISA
+ allocators pre-decrypt 2MB pages in place, avoiding ``swiotlb`` bounce-buffer
+ copies across all DMA modes.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+ (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+ access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+ exceeds the IOTLB size, backpressuring the bus via flow control.
+ This is one of the biggest bottlenecks for high speed devices:
+ typical network devices have very little IOVA locality, and that makes
+ the IOTLB almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA
+ map/unmap costs. This is common practice for command, completion and
+ buffer queues, as well as disk I/O and network receive buffers. Network
+ TX buffers are a noticeable exception. They normally come from
+ ``alloc_pages()`` calls unaware of the leaf device, and require
+ mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+ very expensive operation (up to several microseconds to wait for
+ completion, necessary for security), but the way it is implemented
+ in DMA-FQ compromises on security: IOTLB flush requests are run
+ periodically in batches, but the underlying memory is recycled without
+ waiting. During that window, the physical memory is still accessible,
+ opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+ blocks, e.g. 2MB large pages instead of the usual 4KB (PTE). This is not
+ currently pursued by the kernel.
+
+PISA addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map,
+ unmap, and free device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB PISA pages and 2MB IOMMU
+ mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused PISA pages
+ to the system
+
+Execution is what makes PISA practical: the implementation intercepts
+a small set of kernel APIs so that devices will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since PISA intercepts allocations of device-accessible
+memory, it can transparently configure unencrypted mode, required for
+confidential computing. Under a paravirtualized IOMMU, the long-lived 2MB
+mappings also make host-side pinning of DMA memory a rare event.
+
+Limitations
+===========
+
+The implementation assumes non-preemptible spinlocks, so PISA is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+On arm64, PISA requires 4KB pages and at least 4 levels of page tables.
+This is enforced at compile time.
+
+PISA can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers follow the direction the allocator was created
+with. Descriptor queues and receive buffers are ``DMA_BIDIRECTIONAL``, so received
+buffers can be retransmitted (XDP_TX, forwarding) without a new mapping.
+Allocators that only hold transmit data, such as disk/network write buffers and socket TX fragments, are
+``DMA_TO_DEVICE``.
+
+Implementation details
+======================
+
+PISA is implemented by a few groups of commits:
+
+- mm: infrastructure to implement per-page metadata (avoid any dependency on struct page),
+ and hook into ``alloc_pages``, PISA optimizations (per-cpu caches, shrinker, background zeroing)
+- driver core: per-device sysfs entries to opt-in
+- iommu/dma: manage PISA IOVA ranges and intercept map/unmap
+- block/, mm/, net/: transparently upgrade pages to PISA for disk I/O and network tx zerocopy
+- net: identify sockets that need PISA pages, hook PISA into page frags, skb heads, pagepool
+
+Selected NIC drivers that do not use pagepool need small changes (a few lines)
+to pass ``__GFP_PISA_RW`` when allocating RX buffers.
+
+The implementation lives in ``mm/pisa-*.c``, ``drivers/iommu/pisa-map.c``,
+``net/core/pisa-net.c``, and ``include/linux/pisa*.h``.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, socket TX page frags, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+- ``__filemap_get_folio_mpol()``, ``ractl_alloc_folio()``,
+ ``writeback_get_folio()``, ``__bio_iov_iter_get_pages()``, ``madvise()``
+
+and most devices will automatically inherit the benefits. Fast-path hooks
+are guarded by static keys: ``pisa_key`` remains disabled (a single NOP)
+until the first device opts in via sysfs.
+Separate static keys gate TX socket fragments, small TX skb heads,
+user/disk page migration, and compaction checks until those features are
+enabled.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of PISA pages, based on either their Physical
+ Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to PISA page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **PISA metadata** (``pisa_pfn2slot`` and ``struct pisa_meta``):
+
+ Each 2MB page used for PISA memory is identified by a compact index (``slot``),
+ up to half of physical RAM (256 per 1GB).
+ Identification of PISA pages and per-page metadata use two flat
+ arrays in virtual address space, with holes mapped read-only
+ to a zero page:
+
+ 1. ``pisa_pfn2slot[pfn >> PISA_ORDER]`` (4 bytes per 2MB physical frame)
+ packs the page's status flags and the compact page index. It is populated for
+ system RAM at boot or via hotplug events.
+ A full 57-bit address space requires only 256GB of virtual address space
+ and 12KB for page tables, and expands by 4KB for each 2GB of physical RAM,
+ regardless of RAM layout. With ``CONFIG_KASAN_VMALLOC`` the array covers a
+ 49-bit address space (1GB of virtual address space), to bound the KASAN
+ shadow memory.
+ 2. ``pisa_meta_array[slot]`` (256B per slot/2MB PISA page)
+ holds ``struct pisa_meta``, tracking the page's base PFN, status flags, free/clean/dirty
+ subpage bitmaps, the bitmap of IOMMU domains with an active 2MB mapping
+ for the page, and list/RCU linkage to manage its lifetime.
+ The slot table is sized at boot for up to half of physical RAM (e.g. 128K
+ slots = 32MB of virtual address space for 512GB of RAM), with 4KB backing
+ pages allocated on demand up to ``pisa.max_pages`` (default 1/8th of RAM).
+
+- **Per-domain reserved PISA IOVA ranges**:
+
+ A PISA page can be mapped in multiple domains (e.g.
+ a network buffer used to route packets between different NICs). To simplify
+ address tracking,
+ PISA reserves on each domain an IOVA range covering the slot table
+ (half of physical RAM; the base address may differ per domain),
+ and the page is mapped on demand at an offset ``slot * PISA_SIZE`` in
+ each domain that requires it.
+ This makes PA-IOVA translations just an addition, uniquely identifies
+ PISA IOVAs, and does not expose physical addresses.
+
+- **alloc_pages() compatible pisa_alloc allocator for I/O buffers**:
+
+ PISA implements a pisa_alloc object for fixed-order page allocations
+ returning objects fully compatible with those returned by the page allocator,
+ and with no dependencies on the struct page.
+ ``alloc_pages()`` and related requests are routed to PISA via a
+ ``__GFP_PISA`` flag. This makes it easy to select PISA pages when needed
+ (eg. only two lines in pagepool, similar small changes in page frag refill
+ and disk I/O buffers).
+
+- **Fine grained control**
+
+ Both for experimentation and production use, it is useful to have
+ some form of control on which devices want to use PISA and for
+ what. PISA exposes the following controls. Keep in mind that full
+ performance can only be achieved if all regions are mapped using PISA.
+
+ - ``/sys/devices/*/*/pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads,tx_zerocopy}``
+ per-device entries that control whether a device uses PISA for coherent
+ allocations, RX/disk-read buffers, TX/disk-write buffers, small TX
+ skb heads, and ``MSG_ZEROCOPY`` user pages
+
+- **Safe release of memory to the system**
+
+ A key feature of PISA is that memory is not released to the system
+ while there are active IOMMU mappings. Release of buffers is managed
+ as follows
+
+ 1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to
+ PISA, IOMMU mappings are still active, and they can be recycled
+
+ 2. when all blocks in a PISA page are unused, the page is moved to the
+ allocator's "idle" list, still with active mappings and available for reuse
+
+ 3. idle pages above the per-allocator watermark (``pisa.max_idle``) are
+ released by a reclaim work item after the ``pisa.idle_ms`` grace period,
+ and under memory pressure the shrinker also releases reservoir and idle pages
+ from all allocators: the iommu mappings are removed and a synchronous IOTLB
+ flush is issued. The pages are no longer usable for allocations but not
+ returned to the buddy allocator yet
+
+ 4. once the previous step is complete, the system schedules ``call_rcu()``
+ and waits for an RCU grace period (and re-encrypts the page if it was
+ decrypted) before returning the PISA page to the buddy allocator.
+
+- **Memory compaction**
+
+ Partially allocated 2MB PISA pages are classified into dense (at most 50%
+ free blocks, placed at the head of the partial list and preferred for new
+ allocations) and sparse (at least 75% free blocks, placed at the tail to
+ drain). When ``pisa.compact`` or ``pisa.target_usage_percent`` is enabled and
+ utilization drops below the target, ``pisa_page_can_recycle()`` stops per-CPU
+ and caller-pool recycling for blocks on old sparse pages so they drain to the
+ idle list. For ``user_pa`` pages, ``pisa.scrub_migrate`` also migrates live
+ 4KB pages out of old sparse 2MB pages in the background.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+PISA integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of PISA memory is opt-in via per-device sysfs
+attributes or module parameters.
+
+- **Streaming DMA map and unmap**
+
+ - On map, any buffer belonging to a PISA page
+ (``is_pisa_phys(phys)``) is mapped via the domain's reserved
+ PISA IOVA window with a simple addition, typically reusing
+ existing mappings. Under ``dma-direct``, decrypted PISA pages
+ (``is_pisa_direct(phys, size)``) can be mapped directly without
+ ``swiotlb`` bounce.
+
+ On unmap, IOVAs can be identified as PISA buffers with a simple
+ range check and that results in a no-operation.
+
+- **Coherent DMA allocation and free**
+
+ - PISA is enabled per device via ``/sys/devices/.../pisa_coherent``
+ (``dev->pisa_coherent``). When set, sleepable ``GFP_KERNEL`` coherent
+ allocations up to ``PISA_SIZE`` are rounded up to 4KB, carved out of
+ the per-node order-0 ``pisa_alloc`` via ``pisa_alloc_map_coherent()``, and
+ mapped through the domain's PISA IOVA window (or directly at their
+ physical/decrypted DMA address for ``dma-direct``). ``dma_free_attrs()``
+ releases the order-0 pages via ``pisa_free_coherent()``, which recycles
+ them back to the allocator.
+
+- **Page allocator release**
+
+ - ``pisa_free_page()`` checks ``is_pisa_pfn()``
+ and recycles PISA subpages back to their owning ``pisa_alloc``
+ instead of releasing them to the buddy allocator while their 2MB IOMMU
+ mappings remain active.
+
+- **IOMMU domain teardown**
+
+ - A call to ``pisa_domain_release()`` before freeing a
+ domain's IOVA cookie and page tables clears cached domain state across
+ all allocators. When the last DMA API driver unbinds from a group (or
+ ownership is transferred to VFIO/iommufd), ``pisa_domain_quiesce()``
+ unmaps active PISA pages from that domain and flushes the IOTLB while
+ keeping the reserved window for future rebinds.
+
+Memory allocations are handled as follows:
+
+- **Networking receive packet buffers**
+
+ - PISA is enabled per device via ``pisa_rx_bufs``. A two-line change
+ in pagepool adds ``__GFP_PISA_RW`` to the allocation calls,
+ allocating RX pages from the per-node ``DMA_BIDIRECTIONAL`` PISA
+ allocator and falling back to the buddy allocator on exhaustion.
+
+ - small patches are needed to use PISA memory for RX buffers in devices
+ managing their own page rings e.g. see ``gve`` as an example.
+
+- **Socket TX page fragments**
+
+ - PISA is enabled per device via ``pisa_tx_bufs``, which allows
+ socket TX fragments to be allocated from PISA pages, mapped ``DMA_TO_DEVICE``.
+ - Only data routed to an opt-in device with a DMA parent, outside IPsec,
+ uses PISA memory. Loopback, virtual devices, kTLS, sockmap, KCM
+ and MPTCP use ordinary pages. ESP output uses PISA memory when its
+ egress device allows it.
+
+ The code implementing these constraints touches multiple
+ files because there isn't a single decision point.
+
+- **Small skb heads**
+
+ - PISA is enabled per device via ``pisa_tx_skb_heads``,
+ which allows small skb heads to come from an ``skbuff_small_head_pisa`` slab cache,
+ backed by PISA memory mapped ``DMA_TO_DEVICE``.
+ PISA memory is only used in the cases listed earlier.
+
+ One exception, PISA memory is never used for skb small heads in Confidential
+ Computing because it would expose kernel pointers and callback structures.
+ The fall back to swiotlb causes allocations, mappings and unmappings
+ per packet that can be avoided with ``ethtool --set-tunable <dev> tx-copybreak <bytes>``
+ which uses PISA memory if ``pisa_coherent`` is enabled.
+
+- **Zerocopy network receive**
+
+ - When ``pisa_rx_bufs`` is enabled, receive buffers allocated from
+ ``DMA_BIDIRECTIONAL`` PISA pages can be mapped directly into user space by
+ ``TCP_ZEROCOPY_RECEIVE`` (``vm_insert_pages()``). On Confidential Computing
+ guests, user PTEs mapping decrypted PISA pages are automatically marked
+ decrypted via ``pgprot_decrypted()``.
+
+- **Zerocopy network transmit**
+
+ - PISA is enabled per device via ``pisa_tx_zerocopy`` (and requires
+ ``pisa.user_max_pages``). ``MSG_ZEROCOPY`` sends best-effort migrate eligible user
+ pages into ``DMA_TO_DEVICE`` PISA blocks (skipping pages still referenced
+ by in-flight skbs). Migrated pages stay readable by the device for as
+ long as they remain in PISA, not only during the send.
+ - If ``pisa_tx_bufs`` is also enabled, TCP copies the pages that could not
+ be migrated into PISA socket fragments. Without ``pisa_tx_zerocopy``,
+ ``MSG_ZEROCOPY`` sends use ordinary per-buffer mappings.
+
+- **User and disk I/O pages**
+
+ - ``madvise(MADV_PISA)`` migrates the present private anonymous pages
+ of a range (splitting large anonymous folios to order 0) into blocks of
+ per-node ``user_pa`` allocators, so that zerocopy sends and I/O from them
+ use the PISA window instead of per-request IOMMU mappings or hypervisor
+ pinning. ``MADV_NOPISA`` migrates them back to ordinary pages. Both are
+ allowed through ``process_madvise()``.
+ Freeing the pages (``munmap()``, ``MADV_DONTNEED``, exit, reclaim)
+ returns the blocks to their allocators.
+ - ``pisa.user_max_pages``: Soft cap on user and page-cache pages held in
+ PISA blocks, in 2MB PISA pages like ``pisa.max_pages``. Setting
+ a non-zero value initializes the per-node ``user_pa`` allocators (for
+ ``DMA_BIDIRECTIONAL``, ``DMA_TO_DEVICE``, and ``DMA_FROM_DEVICE``) and
+ enables ``pisa_user_key``; 0 (default) disables it.
+ - For block devices with ``pisa_rx_bufs`` (reads) or ``pisa_tx_bufs``
+ (writes) enabled on a parent device, new order-0 page-cache folios are
+ allocated directly from ``user_pa``, and writeback or direct-I/O bio
+ preparation migrates eligible folios into ``user_pa`` for the I/O's DMA
+ direction, promoting existing read-only or write-only PISA pages to
+ ``DMA_BIDIRECTIONAL`` when accessed in the opposite direction.
+ - On CoCo guests, user PTEs mapping decrypted PISA blocks are marked
+ decrypted via ``pgprot_decrypted()`` so both user space and the device
+ access the shared page in cleartext.
+
+- **Observability**
+
+ - ``/sys/kernel/debug/pisa/allocs``: Debugfs summary of global, per-node,
+ user-page, and per-allocator allocation, mapping, and fallback counters.
+
+- **Global controls:**
+
+ Module parameters are set on the kernel command line as ``pisa.<name>`` and
+ exposed under ``/sys/module/pisa/parameters/<name>``:
+
+ - ``pisa.max_pages``: Global cap on PISA 2MB pages in use
+ (0 defaults to 1/8th of physical RAM, at least 16 pages). Can be changed
+ at runtime, max is half of physical RAM.
+ - ``pisa.reservoir_max``: Per-NUMA-node reservoir target of pre-allocated
+ PISA pages (defaults to 16 per memory node, 0 = disabled). It cannot be 0
+ in Confidential Computing guests: pages are decrypted in sleepable
+ context, so atomic allocations are only served from the reservoirs, which
+ are also filled when PISA is first enabled.
+ - ``pisa.max_idle``: Per-allocator idle PISA page watermark (default 1,
+ minimum 1) before releasing excess idle pages.
+ - ``pisa.idle_ms``: Grace period in ms (default 200, 0 = immediate) before
+ releasing idle PISA pages above ``pisa.max_idle``.
+ - ``pisa.pcp_batch``: Number of blocks moved per per-CPU cache refill or
+ flush (default 32, range 1..32).
+ - ``pisa.compact``: Usage-based compaction threshold in 2MB PISA pages
+ (0 = disabled). When free blocks across an allocator's non-idle 2MB pages
+ exceed ``compact`` 2MB pages, blocks belonging to sparse partial pages
+ bypass per-CPU caches and caller fragment pools so those sparse pages can
+ drain back to the idle list.
+ - ``pisa.target_usage_percent``: Target minimum block utilization percentage
+ across an allocator's non-idle 2MB pages (0 = disabled, 1..100). When usage
+ is below ``target_usage_percent``%, stop recycling blocks from sparse
+ partial pages older than ``old_ms`` ms so they can drain.
+ - ``pisa.old_ms``: Minimum age in ms before a sparse partial page is drained
+ by ``target_usage_percent`` or migrated by ``scrub_migrate`` (0 = disabled).
+ - ``pisa.scrub_max``: Max 4KB pages of dirty blocks scrubbed per background
+ pass (0 disables background scrubbing).
+ - ``pisa.scrub_migrate``: Background migration of old sparse ``user_pa``
+ pages when usage is below ``target_usage_percent`` (0 = disabled,
+ 1 = compact into denser PISA pages, 2 = demote back to the buddy allocator).
+ - ``pisa.user_max_pages``: Soft cap in 2MB PISA pages on user and page-cache
+ pages held in PISA blocks (0 = disabled).
--
2.56.1.809.ga0b61035b5-goog
next prev parent reply other threads:[~2026-11-20 1:07 UTC|newest]
Thread overview: 144+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table Luigi Rizzo
2026-10-03 21:44 ` Randy Dunlap
2026-10-04 9:14 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook Luigi Rizzo
2026-10-10 2:59 ` kernel test robot
2026-10-03 21:22 ` [RFC: DMA_PMD 03/22] mm: Add split_page_compound() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 04/22] iommu/dma: add DMA_PMD pool block allocation Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 05/22] iommu/dma: Global cap and shrinker for DMA_PMD pool memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 06/22] iommu/dma: reserve a per-domain IOVA window for DMA_PMD pages Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 07/22] iommu/dma: release DMA_PMD domain mappings on domain teardown Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 08/22] iommu/dma: use per-domain IOVA window to map DMA_PMD memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 09/22] iommu/dma: Add DMA_PMD arena allocator Luigi Rizzo
2026-10-08 11:25 ` kernel test robot
2026-10-03 21:22 ` [RFC: DMA_PMD 10/22] driver core: Add per-device dma_pmd_* sysfs attributes Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 11/22] dma-mapping: Use DMA_PMD arena for dma_alloc_attrs() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 13/22] net/core: Use DMA_PMD for page_pool memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 14/22] iommu/dma: Support decrypted and pinned DMA_PMD pages Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 15/22] iommu/dma: Add background page scrubber for DMA_PMD pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 16/22] iommu/dma: Add per-NUMA-node PMD page reservoir Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 17/22] net/gve: Use DMA_PMD memory for RX buffers Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 18/22] net/gve: Use DMA_PMD memory for tx header bounce buffers Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 19/22] net/mlx5e: " Luigi Rizzo
2026-10-14 17:22 ` Dragos Tatulea
2026-10-14 17:50 ` Dragos Tatulea
2026-10-14 18:28 ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 20/22] net/idpf: " Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 21/22] net/bnxt: " Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 22/22] iommu/dma: Add DMA_PMD statistics and debugfs Luigi Rizzo
2026-10-11 9:02 ` [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Pranjal Shrivastava
2026-10-12 13:51 ` David Hildenbrand (Arm)
2026-10-12 14:45 ` Luigi Rizzo
2026-10-12 19:43 ` David Hildenbrand (Arm)
2026-10-13 10:42 ` Robin Murphy
2026-10-14 10:47 ` David Hildenbrand (Arm)
2026-10-14 11:05 ` Luigi Rizzo
2026-10-17 17:11 ` Luigi Rizzo
2026-10-22 20:12 ` Jason Gunthorpe
2026-10-22 21:51 ` Luigi Rizzo
2026-10-23 14:01 ` Jason Gunthorpe
2026-10-23 14:30 ` Luigi Rizzo
2026-10-23 14:50 ` Jason Gunthorpe
2026-10-23 15:12 ` Luigi Rizzo
2026-10-23 17:27 ` Jakub Kicinski
2026-10-23 20:22 ` Luigi Rizzo
2026-11-03 8:09 ` Christoph Hellwig
2026-11-03 9:13 ` Luigi Rizzo
2026-11-03 15:10 ` Jason Gunthorpe
2026-11-06 17:21 ` Naman Gulati
2026-11-06 17:30 ` Jason Gunthorpe
2026-11-06 18:00 ` Naman Gulati
2026-11-06 18:07 ` Jason Gunthorpe
2026-11-06 18:17 ` Naman Gulati
2026-11-06 18:43 ` Jason Gunthorpe
2026-11-06 19:01 ` Luigi Rizzo
2026-11-06 19:11 ` Naman Gulati
2026-10-12 14:49 ` Mostafa Saleh
2026-10-12 15:02 ` Luigi Rizzo
2026-10-14 17:28 ` Dragos Tatulea
2026-10-14 18:14 ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 " Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 10/25] mm: Add background page scrubber for " Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
2026-11-20 14:10 ` Lorenzo Stoakes (ARM)
2026-11-20 1:06 ` [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration Luigi Rizzo
2026-11-20 16:17 ` Christoph Hellwig
2026-11-20 1:06 ` [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 15/25] net: Use PISA for page_pool memory Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 23/25] net/gve: " Luigi Rizzo
2026-11-20 1:06 ` [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
2026-11-20 1:06 ` Luigi Rizzo [this message]
2026-11-20 13:50 ` [RFC PATCH v3 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Jason Gunthorpe
2026-11-20 14:18 ` Lorenzo Stoakes (ARM)
2026-11-20 14:19 ` Luigi Rizzo
2026-11-20 14:36 ` Lorenzo Stoakes (ARM)
2026-11-20 14:54 ` Lorenzo Stoakes (ARM)
2026-11-20 16:12 ` David Hildenbrand (Arm)
2026-11-20 18:16 ` Lorenzo Stoakes (ARM)
2026-11-20 15:00 ` [RFC PATCH v2 " Matthew Wilcox
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
2026-11-10 23:12 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
2026-11-10 23:50 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
2026-11-11 0:19 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
2026-11-11 0:37 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
2026-11-11 1:03 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
2026-11-11 1:33 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
2026-11-11 2:41 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
2026-11-11 2:57 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
2026-11-11 3:13 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
2026-11-11 3:22 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
2026-11-11 3:43 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
2026-11-11 4:16 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
2026-11-11 4:24 ` sashiko-bot
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo
2026-11-11 4:37 ` sashiko-bot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120010655.467825-26-lrizzo@google.com \
--to=lrizzo@google.com \
--cc=akpm@linux-foundation.org \
--cc=aleksander.lobakin@intel.com \
--cc=axboe@kernel.dk \
--cc=brendan.jackman@linux.dev \
--cc=catalin.marinas@arm.com \
--cc=cl@gentwo.org \
--cc=corbet@lwn.net \
--cc=dakr@kernel.org \
--cc=davem@davemloft.net \
--cc=david@kernel.org \
--cc=driver-core@lists.linux.dev \
--cc=dtatulea@nvidia.com \
--cc=edumazet@kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@cmpxchg.org \
--cc=hao.li@linux.dev \
--cc=harry@kernel.org \
--cc=hawk@kernel.org \
--cc=hch@lst.de \
--cc=horms@kernel.org \
--cc=hramamurthy@google.com \
--cc=ilias.apalodimas@linaro.org \
--cc=iommu@lists.linux.dev \
--cc=jack@suse.cz \
--cc=jannh@google.com \
--cc=jgg@ziepe.ca \
--cc=joro@8bytes.org \
--cc=joshwash@google.com \
--cc=kuba@kernel.org \
--cc=kuniyu@google.com \
--cc=liam@infradead.org \
--cc=linux-api@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ljs@kernel.org \
--cc=m.szyprowski@samsung.com \
--cc=mhocko@suse.com \
--cc=namangulati@google.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=praan@google.com \
--cc=rafael@kernel.org \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=rizzo.unipi@gmail.com \
--cc=robin.murphy@arm.com \
--cc=roman.gushchin@linux.dev \
--cc=rppt@kernel.org \
--cc=smostafa@google.com \
--cc=surenb@google.com \
--cc=vbabka@kernel.org \
--cc=will@kernel.org \
--cc=willemb@google.com \
--cc=willy@infradead.org \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.