[RFC PATCH v2 0/3] mm: avoid large folio splits when swap is unavailable

[Xueyuan Chen <xueyuan.chen21@gmail.com>]

This is an RFC v2 of Barry's original RFC patch, "mm: Avoiding split
large folios if swap has no space":

https://lore.kernel.org/r/20260618221720.71768-1-baohua@kernel.org

Barry's RFC showed the no-swap case with MADV_PAGEOUT on 16KB mTHP: the
large-folio split counter increased by 1024 even though no swapout
progress was possible. Skipping the split in that case kept the counter
at 0.

This version keeps that behavior, but makes folio_alloc_swap() classify
the failure. The helper has both the swap allocation result and the memcg
swap charge result, so vmscan only needs to split when folio_alloc_swap()
reports that a smaller folio might still be swapped out.

Patch #1 adds page_counter_margin(), a small helper that computes the
minimum remaining chargeable space across a page_counter hierarchy.

Patch #2 uses that helper in the memcg swap path and lets
folio_alloc_swap() distinguish large-folio swap allocation failures:

  - -E2BIG: splitting may let smaller folios make progress
  - -ENOSPC: no global swap space is available
  - -ENOMEM: splitting is not expected to help, including memcg swap
    charge failures with no remaining swap capacity

Patch #3 makes vmscan split a large folio only when folio_alloc_swap()
returns -E2BIG. Other failures keep the existing activation path and avoid
destroying the large folio when no smaller part can be backed by swap
either.

RFC v1 -> RFC v2:
- Split the RFC into helper, swap allocation, and vmscan patches.
- Add page_counter_margin() and use it for hierarchical memcg swap
  capacity checks.
- Make folio_alloc_swap() return -E2BIG only when a smaller folio may
  still be swapped out.
- Return -ENOSPC for no global swap space and -ENOMEM when splitting is
  not expected to help, including memcg swap exhaustion.
- Make vmscan split large folios only on -E2BIG from folio_alloc_swap().

Barry Song (Xiaomi) (1):
  mm/vmscan: avoid pointless large folio splits without swap

Xueyuan Chen (2):
  mm: add page_counter_margin()
  mm: distinguish large folio swap allocation failures

 include/linux/page_counter.h |  1 +
 include/linux/swap.h         | 10 ++++++----
 mm/memcontrol.c              | 15 ++++++++++-----
 mm/page_counter.c            | 20 ++++++++++++++++++++
 mm/swapfile.c                | 21 +++++++++++++++------
 mm/vmscan.c                  |  7 +++++--
 6 files changed, 57 insertions(+), 17 deletions(-)

-- 
2.47.3