[PATCH 0/9] lib/folio_pool: Direct-Map Large Folio Pool & Scratchpad bump allocators

[Jim Cromie via B4 Relay <devnull+jim.cromie.gmail.com@kernel.org>]

Introduce two light-weight bump allocators backed directly by compound
folio pages from the buddy allocator (skipping SLUB metadata overhead):

1. struct folio_scratchpad - Variable-sized, alignment-aware bump allocator
   for bursty, append-mostly lifecycles with bulk teardown.
2. struct folio_pool - Fixed uniform-slot bump allocator for homogeneous
   descriptors.

These serve 2 kinds of use-cases:

1: Ephemeral Batch/transaction Lifecycles (Netfilter, BPF, DRM GPUVM)

- Transactional subsystems allocate bursts of hundreds or thousands of
  homogeneous or heterogeneous objects, only to tear them all down
  simultaneously at batch completion or error abort.

- Under SLUB: Each descriptor incurs freelist traversal, lock
  contention, and O(N) kfree() loops on teardown.

- Under Folio-Scratchpad: Allocations are straight-line pointer bumps,
  bulk teardown is O(1) folio_put(), and consecutive Netlink
  transactions reuse warm L1/L2 cachelines without buddy lock churn
  via folio_scratchpad_reset().

2: Dynamic Long-Lived Graph Scaling (Lockdep)

- Core subsystems historically avoided SLUB by statically
  preallocating massive compile-time arrays in .bss
  (e.g. list_entries[32768], 1.31 MB) to prevent SLUB allocator
  recursion deadlocks.

- Fixed arrays exhaust their limit and permanently disable validation
  (BUG: MAX_LOCKDEP_ENTRIES too low!).
  
- Under Folio-Pool: Lockdep uses a 4,096-entry __initdata bootstrap
  buffer for early boot prior to buddy initialization, then seamlessly
  migrates and compacts all dependency edges into 64 KB direct-map
  folios at late_initcall.
  
- Permanent static .bss allocation for dependency edges is reduced
  from 1.31 MB to 0 KB (100% reclamation at free_initmem()).

Core Primitives in include/linux/folio_pool.h & lib/folio_pool.c:
-----------------------------------------------------------------
0. struct folio_scratchpad (Variable-Sized Elements):
   - Dynamic alignment-aware bump pointer with zero padding waste.
   - API: folio_scratchpad_init(), folio_scratchpad_alloc(),
     folio_scratchpad_reset(), folio_scratchpad_free(),
     folio_scratchpad_stats().
   - Typed macros: folio_scratchpad_alloc_obj(),
     folio_scratchpad_alloc_bytes(), DEFINE_FREE(folio_scratchpad, ...).

1. struct folio_pool (Fixed Uniform Elements):
   - Thin wrapper embedding struct folio_scratchpad for homogeneous objects.
   - API: folio_pool_init(), folio_pool_alloc(), folio_pool_free(),
     folio_pool_stats().
   - Typed macros: folio_pool_alloc_obj(), folio_pool_alloc_type().

2. Encapsulated Slab Discrimination & Reallocation:
   - is_folio_pool_ptr(ptr): Direct-map folios are identified via
     virt_to_folio(ptr) && !folio_test_slab(f), avoiding custom bitflags.
   - folio_scratchpad_free_elem(ptr): Safe no-op for folio-backed objects;
     delegates to kvfree() for SLUB/vmalloc fallback objects.
   - folio_scratchpad_realloc(ptr, old_size, new_size, gfp): Delegates to
     krealloc() for SLUB objects; allocates a fresh SLUB buffer and copies
     payload for folio-backed descriptors.

3. Subsystem Autonomy & Runtime Static-Key Control:
   - DEFINE_FOLIO_POOL_STATIC_KEY_PARAM() allows maintainers of each
     subsystem (Netfilter, Lockdep, DRM, BPF) to choose their own default
     enablement policy (DECLARE_STATIC_KEY_TRUE vs FALSE) and independent
     module/boot parameters (e.g. lockdep.folio_pool, nf_tables.trans_scratchpad).
   - Enables isolated, zero-overhead A/B benchmarking and production triage
     without cross-subsystem coupling or kernel recompilation.

Active Proofs & Performance Profiling:
--------------------------------------
0. Virtualized Guest Netfilter Benchmark (KVM / virtme-ng ftrace):
   Controlled A/B benchmark across 10,025 Netlink transaction descriptors
   and 5 asynchronous commit/abort teardown passes:

   Function: nft_trans_alloc (10,025 descriptor allocations)
   -----------------------------------------------------------------------------
   Mode                                 Total Time       Avg / Alloc      Delta
   -----------------------------------------------------------------------------
   folio_scratchpad Enabled (Bump)      9,158.02 us          0.914 us    -19.4%
   SLUB Fallback (kzalloc)             11,366.53 us          1.134 us  Baseline
   -----------------------------------------------------------------------------
   Net allocation latency reduction:   -2,208.52 us (-2.21 ms)

   Function: nf_tables_trans_destroy_work (5 asynchronous teardown passes)
   -----------------------------------------------------------------------------
   Mode                                 Total Time       Avg / Pass       Delta
   -----------------------------------------------------------------------------
   folio_scratchpad Enabled (Bulk)     36,065.65 us          7.21 ms      -8.6%
   SLUB Fallback (O(N) kfree)          39,449.65 us          7.89 ms   Baseline
   -----------------------------------------------------------------------------
   Net teardown latency reduction:     -3,384.00 us (-3.38 ms)

   Combined In-Kernel Netlink Transaction Lifecycle Savings: -5.59 ms (-13.3%)

1. Bare-Metal Host Netfilter Profile (AMD Zen 3 Hardware PMU & ftrace):
   - Asynchronous Teardown: 53.77 ms vs 56.03 ms (-2.26 ms / -4.0%).
   - Instruction Retirement: 281.29M -> 282.46M (-1,174,062 instructions).
   - Branch Instructions:     62.94M ->  63.20M (-265,961 branches).

2. Bare-Metal Host Lockdep Telemetry (Physical Host frodo, AMD Zen SMP):
   - Pre-Buddy Watermark: 928/4096 bootstrap entries consumed during boot.
   - Graph Compaction: Migrated and compacted bootstrap dependency edges
     into folio_pool at late_initcall; bootstrap array reclaimed by
     free_initmem() (0 KB permanent .bss cost).
   - Live procfs Telemetry (/proc/lockdep_stats):
     lock-classes:                          996 [max: 8192]
     direct dependencies:                  2324 [dynamic: 7 x 64 kB, tail: 17 kB/64 kB]
     indirect dependencies:               24612
     all direct dependencies:             81884
     zapped classes:                          2
     zapped lock chains:                    173
   - Active Invariant Validation: Live module unloads exercised modernized
     zap_class() direct list unlinking across dynamic folios with zero
     global bitmap scans and zero assertions.

What's Unchanged:
-----------------
- All consumer object structures, alignment rules, and lifecycle boundaries
  remain strictly identical.
- Memory safety invariants are preserved: individual deallocation is a safe
  no-op for arena-backed objects while SLUB objects continue to use kfree().

Subsystem Adoptions in this Series:
-----------------------------------
0. lib/folio_pool: Core direct-map large-folio pool and scratchpad allocators.
1. netfilter/nf_tables: Pack netlink transaction descriptors into per-net
   trans_scratchpad with bulk reclaim & warm chunk reset.
2. bpf/verifier: Route verifier stack state node allocations to folio_pool.
3. drm/gpuvm: Route gpuva_op allocations to folio_scratchpad.
4. bpf/syscall: Route generic_map_update_batch key/value allocations to
   folio_scratchpad.
5. locking/lockdep: Fallback to folio_pool in alloc_list_entry when static
   pool is full and expose dynamic folio telemetry in /proc/lockdep_stats.
6. locking/lockdep: Traverse adjacency lists directly in zap_class().
7. locking/lockdep: Shrink static list_entries array to early bootstrap buffer.
8. locking/lockdep: Migrate and compact boot-time dependency graph from __initdata.

Patches in this series:
-----------------------
[PATCH 1/9] lib/folio_pool: Introduce Direct-Map Large Folio Pool & Scratchpad bump allocators
[PATCH 2/9] netfilter/nf_tables: Add folio_scratchpad collector to struct nftables_pernet
[PATCH 3/9] bpf/verifier: Route verifier stack state node allocations to folio_pool
[PATCH 4/9] drm/gpuvm: Route gpuva_op allocations to folio_scratchpad
[PATCH 5/9] bpf/syscall: Route generic_map_update_batch key/value allocations to folio_scratchpad
[PATCH 6/9] locking/lockdep: Fallback to folio_pool in alloc_list_entry when static pool is full
[PATCH 7/9] locking/lockdep: Traverse adjacency lists directly in zap_class()
[PATCH 8/9] locking/lockdep: Shrink static list_entries array to early bootstrap buffer
[PATCH 9/9] locking/lockdep: Migrate and compact boot-time dependency graph from __initdata

Signed-off-by: Jim Cromie <jim.cromie@gmail.com>
---
Jim Cromie (9):
      lib/folio_pool: Introduce Direct-Map Large Folio Pool & Scratchpad bump allocators
      netfilter/nf_tables: Add folio_scratchpad collector to struct nftables_pernet
      bpf/verifier: Route verifier stack state node allocations to folio_pool
      drm/gpuvm: Route gpuva_op allocations to folio_scratchpad
      bpf/syscall: Route generic_map_update_batch key/value allocations to folio_scratchpad
      locking/lockdep: Fallback to folio_pool in alloc_list_entry when static pool is full
      locking/lockdep: Traverse adjacency lists directly in zap_class()
      locking/lockdep: Shrink static list_entries array to early bootstrap buffer
      locking/lockdep: Migrate and compact boot-time dependency graph from __initdata

 drivers/gpu/drm/drm_gpuvm.c        |  11 +-
 include/drm/drm_gpuvm.h            |   6 +
 include/linux/bpf_verifier.h       |   3 +
 include/linux/folio_pool.h         | 279 +++++++++++++++++++++++++++++++++++++
 include/net/netfilter/nf_tables.h  |   3 +
 kernel/bpf/syscall.c               |  17 ++-
 kernel/bpf/verifier.c              |  16 ++-
 kernel/locking/lockdep.c           | 278 +++++++++++++++++++++++++++++-------
 kernel/locking/lockdep_internals.h |   2 +
 kernel/locking/lockdep_proc.c      |  16 ++-
 lib/Makefile                       |   2 +-
 lib/folio_pool.c                   | 230 ++++++++++++++++++++++++++++++
 net/netfilter/nf_tables_api.c      |  51 +++++--
 13 files changed, 837 insertions(+), 77 deletions(-)
---
base-commit: 1a7ac48cd3e62f281ce62f8ed89dc2cc8866eaa4
change-id: 20260815-folio-pool-v1-cb75c247692c

Best regards,
-- 
Jim Cromie <jim.cromie@gmail.com>