Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro <dipiets@amazon.com>]

Hi all,

Thanks Jan, Pedro, Dave, David and Kiryl for the feedback.

Following Dave's question about how PostgreSQL gets large folios on
files it writes 8 KiB at a time, we looked deeper and found a
reproducer that needs only fio and a single disk.

What happens:

  1. A buffered sequential read (VACUUM, a backup, a scan) builds large
     folios through readahead.  They stay large in the page cache until
     they are evicted.
  2. The first small write into one of these clean folios charges the
     whole folio as dirty: a 4 KiB write into a 256 KiB folio charges
     256 KiB, while writeback only writes the dirty 4 KiB.
  3. With random writes, most writes land in a different clean folio,
     so the dirty count reaches the limits with little real data.
     balance_dirty_pages() then throttles the writers, and they end up
     paced by how many folios writeback can clean per second rather
     than by the bytes to write.

Reproducer: XFS on one EBS gp3 volume (3000 IOPS), m8g.8xlarge
(32 CPUs, 128 GiB), default read_ahead_kb=128. fio's 1 MiB reads grow
the readahead window up to the device's max_sectors_kb (256 KiB), so
the read builds 256 KiB folios.

  ; create.fio: 16 x 3 GiB files, direct I/O so nothing is cached
  [global]
  directory=/mnt/xfs/fio
  filename_format=f.$jobnum
  size=3072m
  numjobs=16
  group_reporting=1
  [create]
  rw=write
  bs=1m
  direct=1
  ioengine=libaio
  iodepth=16

  ; load.fio: buffered sequential read; builds the large folios
  [global]
  directory=/mnt/xfs/fio
  filename_format=f.$jobnum
  size=3072m
  numjobs=16
  group_reporting=1
  [load]
  rw=read
  bs=1m
  direct=0
  ioengine=psync

  ; randwrite.fio: random 4 KiB buffered writes, 32 x 312 = ~10k IOPS
  ; invalidate=0 keeps the cached folios when fio opens the files
  [randwrite]
  directory=/mnt/xfs/fio
  filename=f.0:f.1:f.2:f.3:f.4:f.5:f.6:f.7:f.8:f.9:f.10:f.11:f.12:f.13:f.14:f.15
  rw=randwrite
  bs=4k
  direct=0
  ioengine=psync
  invalidate=0
  numjobs=32
  rate_iops=312
  time_based=1
  runtime=120
  group_reporting=1

To reproduce it, run (as root):

  fio create.fio
  sync; echo 3 > /proc/sys/vm/drop_caches
  fio load.fio          # skip this step for the control run
  fio randwrite.fio

fio asks for ~10k write IOPS, more than the volume's 3000, so dirty
data builds up in both runs.  In the control run (no load step) each
4 KiB write creates a 4 KiB folio: after 120 s only 3.3G is dirty,
below the 12.2G background threshold, and nothing is throttled.  After
the load step the same writes land in 256 KiB folios, and each write
into a clean folio is charged 256 KiB: the dirty count passes the
background threshold after 6 s and writers are throttled from 13 s.

Results on v7.4-rc2, one run per kernel, each on a fresh instance.
Writer sleep is the total balance_dirty_pages() pause time over the
32 writers during the 120 s write phase:

                         write   p99 / max   writer   max    folio
                          IOPS     lat ms    sleep s  dirty  size
  baseline, control      9,984  0.01 / 0.30      0    3.3G    -
  baseline               4,061    60 / 160     179   23.5G   256K
  v1 (this series)       9,984  0.01 / 0.67      0    3.6G   256K
  max order cap 64K      7,071    50 / 280      73   22.3G    64K
  max order cap 16K      9,984  0.02 / 0.05      0   11.9G    16K
  split on write         9,984  0.02 / 0.17      0    3.6G   256K
  evict on write         9,984  0.03 / 0.19      0    3.3G   256K
  POSIX_FADV_RANDOM      9,984  0.02 / 0.05      0    3.3G     4K

The control run gave 9,984 IOPS and max latency up to 0.30 ms on every
kernel.

  - max order cap: the global cap in mapping_max_folio_order() that Jan
    suggested trying.  At 64 KiB a 4 KiB write still charges 16x and is
    throttled.  At 16 KiB this run is not throttled, but dirty reached
    11.9G, just below the 12.2G background threshold.  The cap that
    avoids throttling depends on the write size, and it removes large
    folios for every file, including ones that never see small writes.
  - split on write: when a buffered write (iomap opts in, not for
    zeroing or unshare) finds a clean, uptodate, unmapped folio at least
    4x the write size, folio_split() it down to the write's order before
    dirtying it.  The rest of the folio stays cached in large folios.
    On the concerns Jan raised about splitting [1]: split on write only
    splits clean, uptodate, unmapped folios, so it needs no new
    filesystem state beyond an opt-in flag in iomap, and if
    folio_split() fails the write simply proceeds as today; evict on
    write avoids splitting altogether.
  - evict on write: same trigger, but mapping_evict_folio() the folio
    instead of splitting it (~40 lines).  This discards the folio's
    other cached data; nothing reads it back in this test, so the read
    cost does not show here.
  - POSIX_FADV_RANDOM (Dave's question): fio's fadvise_hint=random on
    the load job.  It avoids the problem without a kernel change, but
    only where the application opts in: the hint is per open file, so
    every process that reads a file must issue it, and a reader that
    doesn't (a backup or copy tool, for example) still builds large
    folios.  Every application that reads files sequentially and then
    rewrites them in place would have to adopt it, and keeps the
    regression until it does.  It also turns readahead off for those
    files, so their sequential reads lose readahead.

Split and evict on write keep whole-folio accounting and act at write
time, so they also cover large folios already in the page cache,
whoever built them, without each application having to opt in.  Both
only touch clean, uptodate, unmapped folios at least 4x the write
size, and only on iomap buffered writes.  Split keeps the rest of the
folio cached at the cost of the split; evict is simpler but drops it.
I'd appreciate your opinion on these options, or any other direction
you think is more appropriate.

Thanks,
Salvatore

[1] https://lore.kernel.org/all/oocwnxqgd4klrem6hkya3sddqoxkosml4ysz23if4j7wj2qdwi@5skgc4ftu7io/



AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico