[PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro <dipiets@amazon.it>]

Since iomap gained per-block dirty tracking, a large folio can be
partially dirty: struct iomap_folio_state records exactly which
blocks are dirty, and writeback only writes those ranges.  The mm
dirty accounting, however, still charges the whole folio even if a
single block is dirty (up to 512x over-accounting for a 2 MB folio
over 4 KB blocks).

balance_dirty_pages() then throttles writers against a number that
is unrelated to how much data actually needs writing back.  On
workloads that repeatedly dirty small ranges of large folios --
e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
is crossed and writers are throttled long before the real dirty data
justifies it.  On the benchmark below, run on v7.3-rc7, the
unpatched kernel charges ~159 TB of dirtied pages while writing
~6 TB to the array, a factor of 27; with this series the dirtied
pages charged match the data actually written.

This series makes iomap charge and discharge the mm dirty counters
per dirty page for folios that carry per-block dirty state, so a
partially dirty large folio is accounted for the dirty part only:

  - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
    mapping flag, an @accounted argument to __folio_mark_dirty()
    and __filemap_dirty_folio() to skip the generic per-folio
    charge, and an address_space_operations::accounted_dirty_pages()
    callback used by the discharge and migration paths instead of
    folio_nr_pages().  No filesystem opts in, so it is a no-op on
    its own.

  - Patch 2 makes iomap charge per dirty page from the per-block
    dirty bitmap and wires iomap_accounted_dirty_pages() into every
    iomap-based filesystem.

This was benchmarked with PostgreSQL using the pgbench simple-update
workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
205.2k tps) and run-to-run variability drops from a 23.60% to a
3.98% coefficient of variation: the worst iteration improves from
97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.

Ten iterations per kernel, same database and workload (tps):

  Kernel                     Avg     Min     Max      CV   iters   iters
                                                             1-5    6-10
  ______________________  ______  ______  ______  ______  ______  ______
  v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
  v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k

On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
and throughput decays as the run progresses: iterations 6-10 average
30% below iterations 1-5.  With this series the decay is gone -- the
second half is similar to the first -- and the coefficient of
variation stays at 3.98%.

The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
per-block dirty state tracking to improve performance"), merged in
v6.6, where the ifs bitmap started recording partial dirty state while
the mm counters kept charging whole folios.

The throughput collapse bisects to the v6.17 readahead series, first
bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
file_ra_state"), which lets readahead stay at the largest folio order
once it has ramped up, so a single dirty block charges far more than
is written.  Patch 2 therefore carries:

  Fixes: 4ce02c679722 ("iomap: Add per-block dirty state tracking to improve performance")
  Cc: stable@vger.kernel.org # v6.17+

Test environment:

  Hardware:  2x AWS EC2 m8g.24xlarge
             (SUT with 12x 1TB io2, 32000 IOPS each, RAID0, XFS)
  OS:        AL2023
  Kernels:   v7.3-rc7, v7.3-rc7 + this series
  Database:  PostgreSQL 17 (with huge_pages=on)
  Workload:  pgbench simple-update
             1024 clients, 96 threads, 600s duration
             scale factor 8470, fillfactor=90, prepared protocol
  Runs:      10 iterations per configuration

How to replicate results:

This script starts two AWS EC2 m8g.24xlarge instances (SUT + driver),
then installs, loads and benchmarks PostgreSQL using the
repro-collection repository.  To run it:

  # Reproducer code (dirty-folio-regression branch):
  git clone -b dirty-folio-regression \
    https://github.com/aws/repro-collection.git ~/repro-collection

  # Concatenate patches 1 and 2 of this series, in order, into one file:
  cat 0001-*.patch 0002-*.patch > dirty-per-block.patch

  # Setup and start PostgreSQL benchmark:
  KERNEL_REPO="https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git" \
  KERNEL_BRANCH="v7.3-rc7" \
  ITERS=10 \
  PLACEMENT_GROUP="pg-test" \
  KEY="$HOME/.ssh/id_rsa" \
  KEYPAIR="<aws keypair>" \
  ./repros/repro-postgresql-dirty-folios/folio-reproduce.sh <benchmark name> \
  --patch="<path to dirty-per-block.patch>"

Salvatore Dipietro (2):
  mm: add infrastructure for per-dirty-page accounting
  mm, iomap: account dirty at page granularity for per-block dirty
    folios

 fs/buffer.c             |   4 +-
 fs/exfat/inode.c        |   1 +
 fs/fuse/file.c          |   1 +
 fs/gfs2/aops.c          |   1 +
 fs/iomap/buffered-io.c  | 167 ++++++++++++++++++++++++++++++++++++++--
 fs/ntfs/aops.c          |   2 +
 fs/ntfs3/inode.c        |   2 +
 fs/xfs/xfs_aops.c       |   1 +
 fs/zonefs/file.c        |   1 +
 include/linux/fs.h      |   7 ++
 include/linux/iomap.h   |   1 +
 include/linux/pagemap.h |  20 ++++-
 mm/internal.h           |   3 +
 mm/migrate.c            |  17 +++-
 mm/page-writeback.c     |  91 +++++++++++++++++-----
 15 files changed, 286 insertions(+), 33 deletions(-)


base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico