Linux driver-core infrastructure
 help / color / mirror / Atom feed
From: Luigi Rizzo <lrizzo@google.com>
To: Andrew Morton <akpm@linux-foundation.org>,
	David Hildenbrand <david@kernel.org>,
	 Johannes Weiner <hannes@cmpxchg.org>,
	Vlastimil Babka <vbabka@kernel.org>,
	Joerg Roedel <joro@8bytes.org>,  Will Deacon <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	 Marek Szyprowski <m.szyprowski@samsung.com>,
	Jens Axboe <axboe@kernel.dk>,
	 "David S . Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@kernel.org>,
	 Jakub Kicinski <kuba@kernel.org>,
	Paolo Abeni <pabeni@redhat.com>,
	Luigi Rizzo <rizzo.unipi@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>,
	"Liam R . Howlett" <liam@infradead.org>,
	Mike Rapoport <rppt@kernel.org>,
	 Suren Baghdasaryan <surenb@google.com>,
	Michal Hocko <mhocko@suse.com>,
	 Brendan Jackman <brendan.jackman@linux.dev>,
	Zi Yan <ziy@nvidia.com>, Harry Yoo <harry@kernel.org>,
	 Christoph Lameter <cl@gentwo.org>,
	David Rientjes <rientjes@google.com>,
	 Roman Gushchin <roman.gushchin@linux.dev>,
	Hao Li <hao.li@linux.dev>,  Matthew Wilcox <willy@infradead.org>,
	Jan Kara <jack@suse.cz>, Jann Horn <jannh@google.com>,
	 Catalin Marinas <catalin.marinas@arm.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	 "Rafael J . Wysocki" <rafael@kernel.org>,
	Danilo Krummrich <dakr@kernel.org>,
	Jonathan Corbet <corbet@lwn.net>,
	 Simon Horman <horms@kernel.org>,
	Jesper Dangaard Brouer <hawk@kernel.org>,
	 Ilias Apalodimas <ilias.apalodimas@linaro.org>,
	Willem de Bruijn <willemb@google.com>,
	 Kuniyuki Iwashima <kuniyu@google.com>,
	Joshua Washington <joshwash@google.com>,
	 Harshitha Ramamurthy <hramamurthy@google.com>,
	Alexander Lobakin <aleksander.lobakin@intel.com>,
	 Christoph Hellwig <hch@lst.de>, Jason Gunthorpe <jgg@ziepe.ca>,
	Dragos Tatulea <dtatulea@nvidia.com>,
	 Naman Gulati <namangulati@google.com>,
	Pranjal Shrivastava <praan@google.com>,
	 Mostafa Saleh <smostafa@google.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	linux-mm@kvack.org,  iommu@lists.linux.dev,
	netdev@vger.kernel.org, linux-block@vger.kernel.org,
	 linux-fsdevel@vger.kernel.org, driver-core@lists.linux.dev,
	 linux-arm-kernel@lists.infradead.org, linux-api@vger.kernel.org,
	 linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
	 Luigi Rizzo <lrizzo@google.com>
Subject: [RFC PATCH v3 25/25] Documentation: Add PISA architecture documentation
Date: Fri, 20 Nov 2026 01:06:55 +0000	[thread overview]
Message-ID: <20261120010655.467825-26-lrizzo@google.com> (raw)
In-Reply-To: <20261120010655.467825-1-lrizzo@google.com>

Document the PISA IOMMU 2MB-hugepage-backed DMA memory
subsystem in Documentation/core-api/pisa.rst.

Signed-off-by: Luigi Rizzo <lrizzo@google.com>
---
 .../admin-guide/kernel-parameters.txt         |   7 +
 Documentation/core-api/index.rst              |   1 +
 Documentation/core-api/pisa.rst               | 485 ++++++++++++++++++
 3 files changed, 493 insertions(+)
 create mode 100644 Documentation/core-api/pisa.rst

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 176648941aec2..cce75a9abddad 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5345,6 +5345,13 @@ Kernel parameters
 	pirq=		[SMP,APIC] Manual mp-table setup
 			See Documentation/arch/x86/i386/IO-APIC.rst.
 
+	pisa.*=		[ARM64,X86-64] Pre-mapped IOVA Superpage Allocator (PISA)
+			tuning parameters (max_pages, max_idle, idle_ms,
+			reservoir_max, scrub_max, compact,
+			target_usage_percent, old_ms, pcp_batch, user_max_pages,
+			scrub_migrate).
+			See Documentation/core-api/pisa.rst.
+
 	plip=		[PPT,NET] Parallel port network link
 			Format: { parport<nr> | timid | 0 }
 			See also Documentation/admin-guide/parport.rst.
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 92f91c6a0d79d..4955f477ed797 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -113,6 +113,7 @@ more memory-management documentation in Documentation/mm/index.rst.
    dma-attributes
    dma-isa-lpc
    swiotlb
+   pisa
    mm-api
    cgroup
    genalloc
diff --git a/Documentation/core-api/pisa.rst b/Documentation/core-api/pisa.rst
new file mode 100644
index 0000000000000..f9690b4a9c96a
--- /dev/null
+++ b/Documentation/core-api/pisa.rst
@@ -0,0 +1,485 @@
+.. SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+
+=============================================
+PISA: Pre-mapped IOVA Superpage Allocator
+=============================================
+
+Overview
+========
+
+PISA is an extension to the page allocator: requests with ``__GFP_PISA``
+return memory that makes I/O-related operations cheaper when using
+``DMA-FQ``, ``DMA``, or Confidential Computing ``swiotlb``.  I/O devices
+can individually opt in to use PISA for specific buffer classes (e.g. read
+or write data, headers, zerocopy, ...).  Relevant kernel subsystems
+(mm, iommu, dma, disk I/O, page cache, the network stack) are modified
+to make use of PISA memory for opt-in devices.
+
+The design and implementation have the following main goals:
+
+- no changes to individual device drivers (or minimal changes for NIC drivers
+  that use custom RX page allocators)
+- minimize changes to base code: in most cases one or a few lines of code
+- safety: PISA memory is only used for devices that opt-in,
+  and it is returned to the buddy allocator only after IOMMU mappings
+  and IOTLB entries have been flushed
+- no dependency on ``struct page`` or ``struct folio``
+- compile-time option to disable or enable PISA
+- zero runtime overhead when no devices opt in (via static keys and inline hooks)
+- minimal overhead when in use (direct PFN lookup in a flat metadata table)
+- keep memory overhead as low as possible (adaptive shrinking and compaction
+  of memory)
+
+Mechanisms and performance benefits
+-----------------------------------
+PISA returns order-N blocks backed by 2MB physically contiguous pages, so it
+can use 2MB IOMMU leaves and reduce the IOTLB pressure by up to 512x.
+PISA maps memory in pre-allocated per-domain IOVA ranges, and recycles
+blocks and mappings until the 2MB page is returned to the buddy allocator.
+This makes most of dma map/unmap a no-op, also avoiding related VM exits
+in overcommitted VMs.
+In Confidential Computing VMs, PISA pages are also decrypted, removing the need
+for bounce buffers.
+The kernel identifies devices and sockets that use an
+IOMMU or Confidential Computing and have opted into PISA, and allocates (or
+migrates) I/O buffers using PISA memory so that DMA mapping and unmapping are
+mostly no-ops. All the above makes performance comparable to ``identity`` mode,
+with isolation better than ``DMA-FQ`` and comparable to ``DMA``.
+
+Devices that do not opt in continue to use their native DMA/DMA-FQ mode.
+
+Comparison with standard DMA modes
+----------------------------------
+
+The table below summarizes how PISA (layered on top of strict ``DMA``)
+compares against pass-through/identity mapping (``identity``), batched
+deferred IOTLB flush (``DMA-FQ``), and strict per-request IOMMU mapping
+(``DMA``):
+
+======================================= ==================== ================== ===============
+Feature ( > means better )              vs ``identity``      vs ``DMA-FQ``      vs ``DMA``
+======================================= ==================== ================== ===============
+``dma_map`` cost                        ``PISA ~= identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+``dma_unmap`` cost                      ``PISA == identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+IOTLB friendliness                      ``na``               ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Buffer shared across domains            ``na``               ``PISA <= DMA-FQ`` ``PISA <= DMA``
+Memory remains accessible to device     ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA <= DMA``
+Needs bounce buffers                    ``PISA >> identity`` ``PISA >> DMA-FQ`` ``PISA >> DMA``
+Page alloc/free cost                    ``PISA >= identity`` ``PISA >= DMA-FQ`` ``PISA >= DMA``
+======================================= ==================== ================== ===============
+
+Notes on the comparisons:
+
+- **dma_map / dma_unmap cost:** Once a 2MB PISA page is mapped in a domain's
+  reserved IOVA window, ``dma_map_*()`` is an offset lookup and
+  ``dma_unmap_*()`` is a no-op, avoiding IOVA tree
+  allocation, page-table walks, and synchronous IOTLB shootdowns.
+- **IOTLB friendliness:** PISA maps 2MB IOMMU leaves instead of 4KB leaves,
+  covering 512x more memory per IOTLB entry than ``DMA-FQ`` and ``DMA``.
+  A domain without a 2MB page size maps each PISA page with smaller IOMMU
+  pages: mapping is still done once per PISA page, without this benefit.
+- **Buffer shared across domains:** Even in ``DMA-FQ`` and ``DMA``, IOMMU
+  mappings operate at page granularity, so sub-page buffers from ``task_frag``
+  (shared across all sockets of a task) or ``skb`` head slabs can cause the
+  underlying page to be mapped across multiple domains. PISA exhibits the same
+  behavior, just at 2MB granularity.
+- **Memory remains accessible to device:** In ``identity`` mode, all of RAM is
+  always accessible to devices. In ``DMA-FQ``, pages returned to the buddy
+  allocator remain accessible until the asynchronous flush queue drains. Both
+  PISA and ``DMA`` unmap the page and perform a synchronous IOTLB flush before
+  returning memory to the buddy allocator. In all modes, receive buffers remain mapped
+  and accessible to the device while queued in the rx socket buffer.
+- **Needs bounce buffers:** In Confidential Computing (CoCo) VMs, PISA
+  allocators pre-decrypt 2MB pages in place, avoiding ``swiotlb`` bounce-buffer
+  copies across all DMA modes.
+
+Background and motivations
+==========================
+
+I/O devices typically access at least 3-4 different memory regions on
+each transaction (network packet or disk request):
+
+- command, completion and buffer queues
+- optionally, header or metadata buffers (e.g., NIC packet headers, NVMe SGL...)
+- data buffers (TX socket buffers, NIC RX buffers, disk I/O buffers)
+
+Enabling the IOMMU impacts performance for the following reasons:
+
+- CPU cost to install/remove IOMMU Page Table Entries (PTEs)
+  (``dma_map_*()``, ``dma_unmap_*()``)
+- CPU/system overhead to flush the IOTLB, ie remove stale PTEs that could give
+  access to sensitive data or code when memory is recycled for other purposes.
+- Higher DMA read/write latency from IOMMU page walks when the working set
+  exceeds the IOTLB size, backpressuring the bus via flow control.
+  This is one of the biggest bottlenecks for high speed devices:
+  typical network devices have very little IOVA locality, and that makes
+  the IOTLB almost completely ineffective.
+
+There are known methods to mitigate these problems:
+
+1. Recycling I/O buffers and their IOMMU mappings removes most of the DMA
+   map/unmap costs. This is common practice for command, completion and
+   buffer queues, as well as disk I/O and network receive buffers.  Network
+   TX buffers are a noticeable exception. They normally come from
+   ``alloc_pages()`` calls unaware of the leaf device, and require
+   mapping/unmapping on each use.
+
+2. Lazily flushing the IOTLB (as implemented by DMA-FQ) amortizes a
+   very expensive operation (up to several microseconds to wait for
+   completion, necessary for security), but the way it is implemented
+   in DMA-FQ compromises on security: IOTLB flush requests are run
+   periodically in batches, but the underlying memory is recycled without
+   waiting. During that window, the physical memory is still accessible,
+   opening the door to exfiltration or corruption of sensitive data/code.
+
+3. The set of PTEs in active use can be significantly reduced by mapping larger
+   blocks, e.g. 2MB large pages instead of the usual 4KB (PTE). This is not
+   currently pursued by the kernel.
+
+PISA addresses all the above with a combination of simple, known concepts,
+implemented in a way that integrates smoothly with the kernel and drivers:
+
+- intercept calls to the small set of functions used to allocate, map,
+  unmap, and free device-accessible memory (rings and buffers)
+- return memory backed by physically contiguous 2MB PISA pages and 2MB IOMMU
+  mappings
+- heavily recycle memory and mappings
+- wait until the IOTLB has been flushed before returning unused PISA pages
+  to the system
+
+Execution is what makes PISA practical: the implementation intercepts
+a small set of kernel APIs so that devices will inherit the benefits with
+little if any changes, and the design is such that these intercepts have
+minimal impact on the original functions.
+
+Furthermore, since PISA intercepts allocations of device-accessible
+memory, it can transparently configure unencrypted mode, required for
+confidential computing. Under a paravirtualized IOMMU, the long-lived 2MB
+mappings also make host-side pinning of DMA memory a rare event.
+
+Limitations
+===========
+
+The implementation assumes non-preemptible spinlocks, so PISA is not
+compatible with ``PREEMPT_RT``. This is enforced at compile time.
+
+On arm64, PISA requires 4KB pages and at least 4 levels of page tables.
+This is enforced at compile time.
+
+PISA can only be used by DMA-coherent devices. This is enforced at runtime.
+
+IOMMU mappings for data buffers follow the direction the allocator was created
+with. Descriptor queues and receive buffers are ``DMA_BIDIRECTIONAL``, so received
+buffers can be retransmitted (XDP_TX, forwarding) without a new mapping.
+Allocators that only hold transmit data, such as disk/network write buffers and socket TX fragments, are
+``DMA_TO_DEVICE``.
+
+Implementation details
+======================
+
+PISA is implemented by a few groups of commits:
+
+- mm: infrastructure to implement per-page metadata (avoid any dependency on struct page),
+  and hook into ``alloc_pages``, PISA optimizations (per-cpu caches, shrinker, background zeroing)
+- driver core: per-device sysfs entries to opt-in
+- iommu/dma: manage PISA IOVA ranges and intercept map/unmap
+- block/, mm/, net/: transparently upgrade pages to PISA for disk I/O and network tx zerocopy
+- net: identify sockets that need PISA pages, hook PISA into page frags, skb heads, pagepool
+
+Selected NIC drivers that do not use pagepool need small changes (a few lines)
+to pass ``__GFP_PISA_RW`` when allocating RX buffers.
+
+The implementation lives in ``mm/pisa-*.c``, ``drivers/iommu/pisa-map.c``,
+``net/core/pisa-net.c``, and ``include/linux/pisa*.h``.
+
+The set of functions that need to be intercepted is relatively small:
+
+- ``page_pool_dev_alloc*()``, socket TX page frags, ``__free_pages()``
+- ``dma_alloc_attrs()``, ``dma_map_*()``, ``dma_unmap_*()``
+- ``__filemap_get_folio_mpol()``, ``ractl_alloc_folio()``,
+  ``writeback_get_folio()``, ``__bio_iov_iter_get_pages()``, ``madvise()``
+
+and most devices will automatically inherit the benefits. Fast-path hooks
+are guarded by static keys: ``pisa_key`` remains disabled (a single NOP)
+until the first device opts in via sysfs.
+Separate static keys gate TX socket fragments, small TX skb heads,
+user/disk page migration, and compaction checks until those features are
+enabled.
+
+Requirements (for performance and functionality)
+================================================
+
+- efficient identification of PISA pages, based on either their Physical
+  Address (PA) or their I/O Virtual Address (IOVA)
+- fast access to PISA page metadata
+- fast allocation under highly concurrent usage
+- efficient DMA map/unmap
+- comprehensive lifetime management
+- support for confidential computing (decrypted DMA buffers)
+
+Internal mechanisms
+===================
+
+We use the following components:
+
+- **PISA metadata** (``pisa_pfn2slot`` and ``struct pisa_meta``):
+
+  Each 2MB page used for PISA memory is identified by a compact index (``slot``),
+  up to half of physical RAM (256 per 1GB).
+  Identification of PISA pages and per-page metadata use two flat
+  arrays in virtual address space, with holes mapped read-only
+  to a zero page:
+
+  1. ``pisa_pfn2slot[pfn >> PISA_ORDER]`` (4 bytes per 2MB physical frame)
+     packs the page's status flags and the compact page index. It is populated for
+     system RAM at boot or via hotplug events.
+     A full 57-bit address space requires only 256GB of virtual address space
+     and 12KB for page tables, and expands by 4KB for each 2GB of physical RAM,
+     regardless of RAM layout. With ``CONFIG_KASAN_VMALLOC`` the array covers a
+     49-bit address space (1GB of virtual address space), to bound the KASAN
+     shadow memory.
+  2. ``pisa_meta_array[slot]`` (256B per slot/2MB PISA page)
+     holds ``struct pisa_meta``, tracking the page's base PFN, status flags, free/clean/dirty
+     subpage bitmaps, the bitmap of IOMMU domains with an active 2MB mapping
+     for the page, and list/RCU linkage to manage its lifetime.
+     The slot table is sized at boot for up to half of physical RAM (e.g. 128K
+     slots = 32MB of virtual address space for 512GB of RAM), with 4KB backing
+     pages allocated on demand up to ``pisa.max_pages`` (default 1/8th of RAM).
+
+- **Per-domain reserved PISA IOVA ranges**:
+
+  A PISA page can be mapped in multiple domains (e.g.
+  a network buffer used to route packets between different NICs). To simplify
+  address tracking,
+  PISA reserves on each domain an IOVA range covering the slot table
+  (half of physical RAM; the base address may differ per domain),
+  and the page is mapped on demand at an offset ``slot * PISA_SIZE`` in
+  each domain that requires it.
+  This makes PA-IOVA translations just an addition, uniquely identifies
+  PISA IOVAs, and does not expose physical addresses.
+
+- **alloc_pages() compatible pisa_alloc allocator for I/O buffers**:
+
+  PISA implements a pisa_alloc object for fixed-order page allocations
+  returning objects fully compatible with those returned by the page allocator,
+  and with no dependencies on the struct page.
+  ``alloc_pages()`` and related requests are routed to PISA via a
+  ``__GFP_PISA`` flag. This makes it easy to select PISA pages when needed
+  (eg. only two lines in pagepool, similar small changes in page frag refill
+  and disk I/O buffers).
+
+- **Fine grained control**
+
+  Both for experimentation and production use, it is useful to have
+  some form of control on which devices want to use PISA and for
+  what. PISA exposes the following controls. Keep in mind that full
+  performance can only be achieved if all regions are mapped using PISA.
+
+  - ``/sys/devices/*/*/pisa_{coherent,rx_bufs,tx_bufs,tx_skb_heads,tx_zerocopy}``
+    per-device entries that control whether a device uses PISA for coherent
+    allocations, RX/disk-read buffers, TX/disk-write buffers, small TX
+    skb heads, and ``MSG_ZEROCOPY`` user pages
+
+- **Safe release of memory to the system**
+
+  A key feature of PISA is that memory is not released to the system
+  while there are active IOMMU mappings. Release of buffers is managed
+  as follows
+
+  1. on ``__free_pages()`` or ``dma_free_attrs()``, blocks are returned to
+     PISA, IOMMU mappings are still active, and they can be recycled
+
+  2. when all blocks in a PISA page are unused, the page is moved to the
+     allocator's "idle" list, still with active mappings and available for reuse
+
+  3. idle pages above the per-allocator watermark (``pisa.max_idle``) are
+     released by a reclaim work item after the ``pisa.idle_ms`` grace period,
+     and under memory pressure the shrinker also releases reservoir and idle pages
+     from all allocators: the iommu mappings are removed and a synchronous IOTLB
+     flush is issued. The pages are no longer usable for allocations but not
+     returned to the buddy allocator yet
+
+  4. once the previous step is complete, the system schedules ``call_rcu()``
+     and waits for an RCU grace period (and re-encrypts the page if it was
+     decrypted) before returning the PISA page to the buddy allocator.
+
+- **Memory compaction**
+
+  Partially allocated 2MB PISA pages are classified into dense (at most 50%
+  free blocks, placed at the head of the partial list and preferred for new
+  allocations) and sparse (at least 75% free blocks, placed at the tail to
+  drain). When ``pisa.compact`` or ``pisa.target_usage_percent`` is enabled and
+  utilization drops below the target, ``pisa_page_can_recycle()`` stops per-CPU
+  and caller-pool recycling for blocks on old sparse pages so they drain to the
+  idle list. For ``user_pa`` pages, ``pisa.scrub_migrate`` also migrates live
+  4KB pages out of old sparse 2MB pages in the background.
+
+Modified kernel APIs and runtime controls
+=========================================
+
+PISA integrates transparently into existing kernel memory, DMA, and
+networking APIs. Allocation of PISA memory is opt-in via per-device sysfs
+attributes or module parameters.
+
+- **Streaming DMA map and unmap**
+
+  - On map, any buffer belonging to a PISA page
+    (``is_pisa_phys(phys)``) is mapped via the domain's reserved
+    PISA IOVA window with a simple addition, typically reusing
+    existing mappings. Under ``dma-direct``, decrypted PISA pages
+    (``is_pisa_direct(phys, size)``) can be mapped directly without
+    ``swiotlb`` bounce.
+
+    On unmap, IOVAs can be identified as PISA buffers with a simple
+    range check and that results in a no-operation.
+
+- **Coherent DMA allocation and free**
+
+  - PISA is enabled per device via ``/sys/devices/.../pisa_coherent``
+    (``dev->pisa_coherent``). When set, sleepable ``GFP_KERNEL`` coherent
+    allocations up to ``PISA_SIZE`` are rounded up to 4KB, carved out of
+    the per-node order-0 ``pisa_alloc`` via ``pisa_alloc_map_coherent()``, and
+    mapped through the domain's PISA IOVA window (or directly at their
+    physical/decrypted DMA address for ``dma-direct``). ``dma_free_attrs()``
+    releases the order-0 pages via ``pisa_free_coherent()``, which recycles
+    them back to the allocator.
+
+- **Page allocator release**
+
+  - ``pisa_free_page()`` checks ``is_pisa_pfn()``
+    and recycles PISA subpages back to their owning ``pisa_alloc``
+    instead of releasing them to the buddy allocator while their 2MB IOMMU
+    mappings remain active.
+
+- **IOMMU domain teardown**
+
+  - A call to ``pisa_domain_release()`` before freeing a
+    domain's IOVA cookie and page tables clears cached domain state across
+    all allocators. When the last DMA API driver unbinds from a group (or
+    ownership is transferred to VFIO/iommufd), ``pisa_domain_quiesce()``
+    unmaps active PISA pages from that domain and flushes the IOTLB while
+    keeping the reserved window for future rebinds.
+
+Memory allocations are handled as follows:
+
+- **Networking receive packet buffers**
+
+  - PISA is enabled per device via ``pisa_rx_bufs``. A two-line change
+    in pagepool adds ``__GFP_PISA_RW`` to the allocation calls,
+    allocating RX pages from the per-node ``DMA_BIDIRECTIONAL`` PISA
+    allocator and falling back to the buddy allocator on exhaustion.
+
+  - small patches are needed to use PISA memory for RX buffers in devices
+    managing their own page rings e.g. see ``gve`` as an example.
+
+- **Socket TX page fragments**
+
+  - PISA is enabled per device via ``pisa_tx_bufs``, which allows
+    socket TX fragments to be allocated from PISA pages, mapped ``DMA_TO_DEVICE``.
+  - Only data routed to an opt-in device with a DMA parent, outside IPsec,
+    uses PISA memory.  Loopback, virtual devices, kTLS, sockmap, KCM
+    and MPTCP use ordinary pages. ESP output uses PISA memory when its
+    egress device allows it.
+
+    The code implementing these constraints touches multiple
+    files because there isn't a single decision point.
+
+- **Small skb heads**
+
+  - PISA is enabled per device via ``pisa_tx_skb_heads``,
+    which allows small skb heads to come from an ``skbuff_small_head_pisa`` slab cache,
+    backed by PISA memory mapped ``DMA_TO_DEVICE``.
+    PISA memory is only used in the cases listed earlier.
+
+    One exception, PISA memory is never used for skb small heads in Confidential
+    Computing because it would expose kernel pointers and callback structures.
+    The fall back to swiotlb causes allocations, mappings and unmappings
+    per packet that can be avoided with ``ethtool --set-tunable <dev> tx-copybreak <bytes>``
+    which uses PISA memory if ``pisa_coherent`` is enabled.
+
+- **Zerocopy network receive**
+
+  - When ``pisa_rx_bufs`` is enabled, receive buffers allocated from
+    ``DMA_BIDIRECTIONAL`` PISA pages can be mapped directly into user space by
+    ``TCP_ZEROCOPY_RECEIVE`` (``vm_insert_pages()``). On Confidential Computing
+    guests, user PTEs mapping decrypted PISA pages are automatically marked
+    decrypted via ``pgprot_decrypted()``.
+
+- **Zerocopy network transmit**
+
+  - PISA is enabled per device via ``pisa_tx_zerocopy`` (and requires
+    ``pisa.user_max_pages``). ``MSG_ZEROCOPY`` sends best-effort migrate eligible user
+    pages into ``DMA_TO_DEVICE`` PISA blocks (skipping pages still referenced
+    by in-flight skbs). Migrated pages stay readable by the device for as
+    long as they remain in PISA, not only during the send.
+  - If ``pisa_tx_bufs`` is also enabled, TCP copies the pages that could not
+    be migrated into PISA socket fragments. Without ``pisa_tx_zerocopy``,
+    ``MSG_ZEROCOPY`` sends use ordinary per-buffer mappings.
+
+- **User and disk I/O pages**
+
+  - ``madvise(MADV_PISA)`` migrates the present private anonymous pages
+    of a range (splitting large anonymous folios to order 0) into blocks of
+    per-node ``user_pa`` allocators, so that zerocopy sends and I/O from them
+    use the PISA window instead of per-request IOMMU mappings or hypervisor
+    pinning. ``MADV_NOPISA`` migrates them back to ordinary pages. Both are
+    allowed through ``process_madvise()``.
+    Freeing the pages (``munmap()``, ``MADV_DONTNEED``, exit, reclaim)
+    returns the blocks to their allocators.
+  - ``pisa.user_max_pages``: Soft cap on user and page-cache pages held in
+    PISA blocks, in 2MB PISA pages like ``pisa.max_pages``. Setting
+    a non-zero value initializes the per-node ``user_pa`` allocators (for
+    ``DMA_BIDIRECTIONAL``, ``DMA_TO_DEVICE``, and ``DMA_FROM_DEVICE``) and
+    enables ``pisa_user_key``; 0 (default) disables it.
+  - For block devices with ``pisa_rx_bufs`` (reads) or ``pisa_tx_bufs``
+    (writes) enabled on a parent device, new order-0 page-cache folios are
+    allocated directly from ``user_pa``, and writeback or direct-I/O bio
+    preparation migrates eligible folios into ``user_pa`` for the I/O's DMA
+    direction, promoting existing read-only or write-only PISA pages to
+    ``DMA_BIDIRECTIONAL`` when accessed in the opposite direction.
+  - On CoCo guests, user PTEs mapping decrypted PISA blocks are marked
+    decrypted via ``pgprot_decrypted()`` so both user space and the device
+    access the shared page in cleartext.
+
+- **Observability**
+
+  - ``/sys/kernel/debug/pisa/allocs``: Debugfs summary of global, per-node,
+    user-page, and per-allocator allocation, mapping, and fallback counters.
+
+- **Global controls:**
+
+  Module parameters are set on the kernel command line as ``pisa.<name>`` and
+  exposed under ``/sys/module/pisa/parameters/<name>``:
+
+  - ``pisa.max_pages``: Global cap on PISA 2MB pages in use
+    (0 defaults to 1/8th of physical RAM, at least 16 pages). Can be changed
+    at runtime, max is half of physical RAM.
+  - ``pisa.reservoir_max``: Per-NUMA-node reservoir target of pre-allocated
+    PISA pages (defaults to 16 per memory node, 0 = disabled). It cannot be 0
+    in Confidential Computing guests: pages are decrypted in sleepable
+    context, so atomic allocations are only served from the reservoirs, which
+    are also filled when PISA is first enabled.
+  - ``pisa.max_idle``: Per-allocator idle PISA page watermark (default 1,
+    minimum 1) before releasing excess idle pages.
+  - ``pisa.idle_ms``: Grace period in ms (default 200, 0 = immediate) before
+    releasing idle PISA pages above ``pisa.max_idle``.
+  - ``pisa.pcp_batch``: Number of blocks moved per per-CPU cache refill or
+    flush (default 32, range 1..32).
+  - ``pisa.compact``: Usage-based compaction threshold in 2MB PISA pages
+    (0 = disabled). When free blocks across an allocator's non-idle 2MB pages
+    exceed ``compact`` 2MB pages, blocks belonging to sparse partial pages
+    bypass per-CPU caches and caller fragment pools so those sparse pages can
+    drain back to the idle list.
+  - ``pisa.target_usage_percent``: Target minimum block utilization percentage
+    across an allocator's non-idle 2MB pages (0 = disabled, 1..100). When usage
+    is below ``target_usage_percent``%, stop recycling blocks from sparse
+    partial pages older than ``old_ms`` ms so they can drain.
+  - ``pisa.old_ms``: Minimum age in ms before a sparse partial page is drained
+    by ``target_usage_percent`` or migrated by ``scrub_migrate`` (0 = disabled).
+  - ``pisa.scrub_max``: Max 4KB pages of dirty blocks scrubbed per background
+    pass (0 disables background scrubbing).
+  - ``pisa.scrub_migrate``: Background migration of old sparse ``user_pa``
+    pages when usage is below ``target_usage_percent`` (0 = disabled,
+    1 = compact into denser PISA pages, 2 = demote back to the buddy allocator).
+  - ``pisa.user_max_pages``: Soft cap in 2MB PISA pages on user and page-cache
+    pages held in PISA blocks (0 = disabled).
-- 
2.56.1.809.ga0b61035b5-goog


  parent reply	other threads:[~2026-11-20  1:07 UTC|newest]

Thread overview: 112+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-03 21:22 [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 01/22] iommu/dma: introduce CONFIG_DMA_PMD and metadata table Luigi Rizzo
2026-10-03 21:44   ` Randy Dunlap
2026-10-04  9:14     ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 02/22] iommu/dma: add DMA_PMD pool lifecycle and page recycle hook Luigi Rizzo
2026-10-10  2:59   ` kernel test robot
2026-10-03 21:22 ` [RFC: DMA_PMD 03/22] mm: Add split_page_compound() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 04/22] iommu/dma: add DMA_PMD pool block allocation Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 05/22] iommu/dma: Global cap and shrinker for DMA_PMD pool memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 06/22] iommu/dma: reserve a per-domain IOVA window for DMA_PMD pages Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 07/22] iommu/dma: release DMA_PMD domain mappings on domain teardown Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 08/22] iommu/dma: use per-domain IOVA window to map DMA_PMD memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 09/22] iommu/dma: Add DMA_PMD arena allocator Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 10/22] driver core: Add per-device dma_pmd_* sysfs attributes Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 11/22] dma-mapping: Use DMA_PMD arena for dma_alloc_attrs() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 12/22] net/core: Use per-CPU DMA_PMD pools for skb_page_frag_refill() Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 13/22] net/core: Use DMA_PMD for page_pool memory Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 14/22] iommu/dma: Support decrypted and pinned DMA_PMD pages Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 15/22] iommu/dma: Add background page scrubber for DMA_PMD pools Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 16/22] iommu/dma: Add per-NUMA-node PMD page reservoir Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 17/22] net/gve: Use DMA_PMD memory for RX buffers Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 18/22] net/gve: Use DMA_PMD memory for tx header bounce buffers Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 19/22] net/mlx5e: " Luigi Rizzo
2026-10-14 17:22   ` Dragos Tatulea
2026-10-14 17:50     ` Dragos Tatulea
2026-10-14 18:28       ` Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 20/22] net/idpf: " Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 21/22] net/bnxt: " Luigi Rizzo
2026-10-03 21:22 ` [RFC: DMA_PMD 22/22] iommu/dma: Add DMA_PMD statistics and debugfs Luigi Rizzo
2026-10-11  9:02 ` [RFC: DMA_PMD 00/22] DMA_PMD: PMD_SIZE-backed IO buffer pools Pranjal Shrivastava
2026-10-12 13:51 ` David Hildenbrand (Arm)
2026-10-12 14:45   ` Luigi Rizzo
2026-10-12 19:43     ` David Hildenbrand (Arm)
2026-10-13 10:42       ` Robin Murphy
2026-10-14 10:47         ` David Hildenbrand (Arm)
2026-10-14 11:05           ` Luigi Rizzo
2026-10-17 17:11       ` Luigi Rizzo
2026-10-22 20:12         ` Jason Gunthorpe
2026-10-22 21:51           ` Luigi Rizzo
2026-10-23 14:01             ` Jason Gunthorpe
2026-10-23 14:30               ` Luigi Rizzo
2026-10-23 14:50                 ` Jason Gunthorpe
2026-10-23 15:12                   ` Luigi Rizzo
2026-10-23 17:27                   ` Jakub Kicinski
2026-10-23 20:22                     ` Luigi Rizzo
2026-11-03  8:09                   ` Christoph Hellwig
2026-11-03  9:13                     ` Luigi Rizzo
2026-11-03 15:10                     ` Jason Gunthorpe
2026-11-06 17:21                       ` Naman Gulati
2026-11-06 17:30                         ` Jason Gunthorpe
2026-11-06 18:00                           ` Naman Gulati
2026-11-06 18:07                             ` Jason Gunthorpe
2026-11-06 18:17                               ` Naman Gulati
2026-11-06 18:43                                 ` Jason Gunthorpe
2026-11-06 19:01                                   ` Luigi Rizzo
2026-11-06 19:11                                   ` Naman Gulati
2026-10-12 14:49   ` Mostafa Saleh
2026-10-12 15:02     ` Luigi Rizzo
2026-10-14 17:28 ` Dragos Tatulea
2026-10-14 18:14   ` Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 00/25] PISA: Pre-mapped IOVA Superpage Allocator for DMA and CoCo Luigi Rizzo
2026-11-20  1:06   ` [RFC PATCH v3 " Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 01/25] mm: Introduce CONFIG_PISA and metadata table Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 02/25] mm: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 03/25] mm: Add per-NUMA-node PISA page reservoir Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 04/25] mm/page_alloc: Extend alloc_pages() to __GFP_PISA and PISA block allocation Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 05/25] mm: Add global cap and shrinker for PISA pages Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 06/25] iommu/dma: Map PISA memory in a per-domain IOVA window Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 07/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 08/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 09/25] mm: Add per-CPU block caches to PISA allocators Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 10/25] mm: Add background page scrubber for " Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 11/25] mm: Migrate user pages into PISA Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 12/25] block, mm: Support PISA for disk I/O and scrub migration Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 13/25] net: Cache leaf PISA TX flags on struct net_device Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 14/25] net: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 15/25] net: Use PISA for page_pool memory Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 18/25] net: Migrate user pages into PISA for MSG_ZEROCOPY Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 19/25] mm, iommu/dma: Support decrypted PISA pages Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 20/25] arm64/mm, mm: Support decrypted PISA pages in Realms Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 21/25] mm: Add PISA statistics and debugfs Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 22/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 23/25] net/gve: " Luigi Rizzo
2026-11-20  1:06     ` [RFC PATCH v3 24/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
2026-11-20  1:06     ` Luigi Rizzo [this message]
2026-11-09 18:25 ` [RFC PATCH v2 01/25] iommu/dma: Introduce CONFIG_PISA and metadata table Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 02/25] iommu/dma: Add PISA allocator lifecycle and page recycle hook Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 03/25] iommu/dma: Add per-NUMA-node PISA page reservoir Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 04/25] iommu/dma: Add PISA block allocation Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 05/25] iommu/dma: Add global cap and shrinker for PISA pages Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 06/25] iommu/dma: Reserve a per-domain IOVA window " Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 07/25] iommu/dma: Release PISA domain mappings on domain teardown Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 08/25] iommu/dma: Use per-domain IOVA window to map PISA memory Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 09/25] driver core: Add per-device pisa_* sysfs attributes for PISA optin Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 10/25] iommu/dma: Use PISA for coherent DMA allocations Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 11/25] iommu/dma: Add per-CPU block caches to PISA allocators Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 12/25] net/core: Use per-node PISA allocators for skb_page_frag_refill() Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 13/25] net/core: Use PISA for page_pool memory Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 14/25] iommu/dma: Support decrypted PISA pages Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 15/25] iommu/dma: Add background page scrubber for PISA allocators Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 16/25] mm/slub: Add SLAB_CACHE_PISA_RO for PISA-backed slab caches Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 17/25] net: Allocate small skb heads from PISA Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 18/25] iommu/dma: Migrate user pages into PISA for TX zerocopy Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 19/25] iommu/dma: Stop recycling old PISA pages when allocator waste is high Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 20/25] net/libeth: Use GFP_KERNEL when pre-filling RX rings Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 21/25] net/gve: " Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 22/25] net/gve: Use PISA memory for RX buffers Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 23/25] block, mm, iommu/dma: Support PISA for disk I/O and scrub migration Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 24/25] Documentation: Add PISA architecture documentation Luigi Rizzo
2026-11-09 18:25 ` [RFC PATCH v2 25/25] iommu/dma: Add PISA statistics and debugfs Luigi Rizzo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120010655.467825-26-lrizzo@google.com \
    --to=lrizzo@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=aleksander.lobakin@intel.com \
    --cc=axboe@kernel.dk \
    --cc=brendan.jackman@linux.dev \
    --cc=catalin.marinas@arm.com \
    --cc=cl@gentwo.org \
    --cc=corbet@lwn.net \
    --cc=dakr@kernel.org \
    --cc=davem@davemloft.net \
    --cc=david@kernel.org \
    --cc=driver-core@lists.linux.dev \
    --cc=dtatulea@nvidia.com \
    --cc=edumazet@kernel.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=hannes@cmpxchg.org \
    --cc=hao.li@linux.dev \
    --cc=harry@kernel.org \
    --cc=hawk@kernel.org \
    --cc=hch@lst.de \
    --cc=horms@kernel.org \
    --cc=hramamurthy@google.com \
    --cc=ilias.apalodimas@linaro.org \
    --cc=iommu@lists.linux.dev \
    --cc=jack@suse.cz \
    --cc=jannh@google.com \
    --cc=jgg@ziepe.ca \
    --cc=joro@8bytes.org \
    --cc=joshwash@google.com \
    --cc=kuba@kernel.org \
    --cc=kuniyu@google.com \
    --cc=liam@infradead.org \
    --cc=linux-api@vger.kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=ljs@kernel.org \
    --cc=m.szyprowski@samsung.com \
    --cc=mhocko@suse.com \
    --cc=namangulati@google.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=praan@google.com \
    --cc=rafael@kernel.org \
    --cc=rdunlap@infradead.org \
    --cc=rientjes@google.com \
    --cc=rizzo.unipi@gmail.com \
    --cc=robin.murphy@arm.com \
    --cc=roman.gushchin@linux.dev \
    --cc=rppt@kernel.org \
    --cc=smostafa@google.com \
    --cc=surenb@google.com \
    --cc=vbabka@kernel.org \
    --cc=will@kernel.org \
    --cc=willemb@google.com \
    --cc=willy@infradead.org \
    --cc=ziy@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox