[PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro]

Since iomap gained per-block dirty tracking, a large folio can be
partially dirty: struct iomap_folio_state records exactly which
blocks are dirty, and writeback only writes those ranges.  The mm
dirty accounting, however, still charges the whole folio even if a
single block is dirty (up to 512x over-accounting for a 2 MB folio
over 4 KB blocks).

balance_dirty_pages() then throttles writers against a number that
is unrelated to how much data actually needs writing back.  On
workloads that repeatedly dirty small ranges of large folios --
e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
is crossed and writers are throttled long before the real dirty data
justifies it.  On the benchmark below, run on v7.3-rc7, the
unpatched kernel charges ~159 TB of dirtied pages while writing
~6 TB to the array, a factor of 27; with this series the dirtied
pages charged match the data actually written.

This series makes iomap charge and discharge the mm dirty counters
per dirty page for folios that carry per-block dirty state, so a
partially dirty large folio is accounted for the dirty part only:

  - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
    mapping flag, an @accounted argument to __folio_mark_dirty()
    and __filemap_dirty_folio() to skip the generic per-folio
    charge, and an address_space_operations::accounted_dirty_pages()
    callback used by the discharge and migration paths instead of
    folio_nr_pages().  No filesystem opts in, so it is a no-op on
    its own.

  - Patch 2 makes iomap charge per dirty page from the per-block
    dirty bitmap and wires iomap_accounted_dirty_pages() into every
    iomap-based filesystem.

This was benchmarked with PostgreSQL using the pgbench simple-update
workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
205.2k tps) and run-to-run variability drops from a 23.60% to a
3.98% coefficient of variation: the worst iteration improves from
97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.

Ten iterations per kernel, same database and workload (tps):

  Kernel                     Avg     Min     Max      CV   iters   iters
                                                             1-5    6-10
  ______________________  ______  ______  ______  ______  ______  ______
  v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
  v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k

On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
and throughput decays as the run progresses: iterations 6-10 average
30% below iterations 1-5.  With this series the decay is gone -- the
second half is similar to the first -- and the coefficient of
variation stays at 3.98%.

The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
per-block dirty state tracking to improve performance"), merged in
v6.6, where the ifs bitmap started recording partial dirty state while
the mm counters kept charging whole folios.

The throughput collapse bisects to the v6.17 readahead series, first
bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
file_ra_state"), which lets readahead stay at the largest folio order
once it has ramped up, so a single dirty block charges far more than
is written.  Patch 2 therefore carries:

  Fixes: 4ce02c679722 ("iomap: Add per-block dirty state tracking to improve performance")
  Cc: stable@vger.kernel.org # v6.17+

Test environment:

  Hardware:  2x AWS EC2 m8g.24xlarge
             (SUT with 12x 1TB io2, 32000 IOPS each, RAID0, XFS)
  OS:        AL2023
  Kernels:   v7.3-rc7, v7.3-rc7 + this series
  Database:  PostgreSQL 17 (with huge_pages=on)
  Workload:  pgbench simple-update
             1024 clients, 96 threads, 600s duration
             scale factor 8470, fillfactor=90, prepared protocol
  Runs:      10 iterations per configuration

How to replicate results:

This script starts two AWS EC2 m8g.24xlarge instances (SUT + driver),
then installs, loads and benchmarks PostgreSQL using the
repro-collection repository.  To run it:

  # Reproducer code (dirty-folio-regression branch):
  git clone -b dirty-folio-regression \
    https://github.com/aws/repro-collection.git ~/repro-collection

  # Concatenate patches 1 and 2 of this series, in order, into one file:
  cat 0001-*.patch 0002-*.patch > dirty-per-block.patch

  # Setup and start PostgreSQL benchmark:
  KERNEL_REPO="https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git" \
  KERNEL_BRANCH="v7.3-rc7" \
  ITERS=10 \
  PLACEMENT_GROUP="pg-test" \
  KEY="$HOME/.ssh/id_rsa" \
  KEYPAIR="<aws keypair>" \
  ./repros/repro-postgresql-dirty-folios/folio-reproduce.sh <benchmark name> \
  --patch="<path to dirty-per-block.patch>"

Salvatore Dipietro (2):
  mm: add infrastructure for per-dirty-page accounting
  mm, iomap: account dirty at page granularity for per-block dirty
    folios

 fs/buffer.c             |   4 +-
 fs/exfat/inode.c        |   1 +
 fs/fuse/file.c          |   1 +
 fs/gfs2/aops.c          |   1 +
 fs/iomap/buffered-io.c  | 167 ++++++++++++++++++++++++++++++++++++++--
 fs/ntfs/aops.c          |   2 +
 fs/ntfs3/inode.c        |   2 +
 fs/xfs/xfs_aops.c       |   1 +
 fs/zonefs/file.c        |   1 +
 include/linux/fs.h      |   7 ++
 include/linux/iomap.h   |   1 +
 include/linux/pagemap.h |  20 ++++-
 mm/internal.h           |   3 +
 mm/migrate.c            |  17 +++-
 mm/page-writeback.c     |  91 +++++++++++++++++-----
 15 files changed, 286 insertions(+), 33 deletions(-)


base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

[PATCH 1/2] mm: add infrastructure for per-dirty-page accounting

[Salvatore Dipietro]

Add the mm-side machinery that lets a filesystem charge the dirty
counters (NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING, WB_RECLAIMABLE) per
dirty page instead of per whole folio, without yet enabling it for
any filesystem.

Add an AS_DIRTY_PER_BLOCK mapping flag with
mapping_set_dirty_per_block()/mapping_dirty_per_block() helpers, an
accounted_dirty_pages() address_space_operations callback, and an
@accounted argument to __folio_mark_dirty() (with filemap_dirty_folio()
split into __filemap_dirty_folio()) so the generic per-folio charge can
be suppressed for folios a filesystem has already accounted.

folio_accounted_dirty_pages() dispatches through the new a_ops callback,
falling back to folio_nr_pages() when no filesystem opts in, and
folio_account_cleaned() and folio migration use it in place of
folio_nr_pages().  folio_clear_dirty_for_io() leaves the charge of
per-block folios to the filesystem.  No filesystem sets the flag or
registers the callback yet, so this commit is a no-op on its own.

This is a prerequisite for the following iomap patch, which fixes the
over-accounting; both are needed for stable.

Cc: stable@vger.kernel.org # v6.17+
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
Reviewed-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 fs/buffer.c             |  4 +-
 include/linux/fs.h      |  7 ++++
 include/linux/pagemap.h | 20 ++++++++-
 mm/internal.h           |  3 ++
 mm/migrate.c            | 17 ++++++--
 mm/page-writeback.c     | 91 ++++++++++++++++++++++++++++++++---------
 6 files changed, 115 insertions(+), 27 deletions(-)

diff --git a/fs/buffer.c b/fs/buffer.c
index ed966fa73b1b..7ed5250dda9d 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -693,7 +693,7 @@ bool block_dirty_folio(struct address_space *mapping, struct folio *folio)
 	spin_unlock(&mapping->i_private_lock);
 
 	if (newly_dirty)
-		__folio_mark_dirty(folio, mapping, 1);
+		__folio_mark_dirty(folio, mapping, 1, false);
 
 	if (newly_dirty)
 		__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
@@ -1011,7 +1011,7 @@ void mark_buffer_dirty(struct buffer_head *bh)
 		if (!folio_test_set_dirty(folio)) {
 			mapping = folio->mapping;
 			if (mapping)
-				__folio_mark_dirty(folio, mapping, 0);
+				__folio_mark_dirty(folio, mapping, 0, false);
 		}
 		if (mapping)
 			__mark_inode_dirty(mapping->host, I_DIRTY_PAGES);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index deb411e86661..5abd2af50b59 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -407,6 +407,13 @@ struct address_space_operations {
 	/* Mark a folio dirty.  Return true if this dirtied it */
 	bool (*dirty_folio)(struct address_space *, struct folio *);
 
+	/*
+	 * Pages of the folio currently charged to the dirty counters, for
+	 * filesystems that account dirty state at sub-folio (per-block)
+	 * granularity.
+	 */
+	long (*accounted_dirty_pages)(struct folio *folio);
+
 	void (*readahead)(struct readahead_control *);
 
 	int (*write_begin)(const struct kiocb *, struct address_space *mapping,
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 0adfa6605653..5bbcd9da1a6b 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -210,6 +210,7 @@ enum mapping_flags {
 	AS_WRITEBACK_MAY_DEADLOCK_ON_RECLAIM = 9,
 	AS_KERNEL_FILE = 10,	/* mapping for a fake kernel file that shouldn't
 				   account usage to user cgroups */
+	AS_DIRTY_PER_BLOCK = 11, /* fs accounts dirty pages, not whole folios */
 	/* Bits 16-25 are used for FOLIO_ORDER */
 	AS_FOLIO_ORDER_BITS = 5,
 	AS_FOLIO_ORDER_MIN = 16,
@@ -304,6 +305,20 @@ static inline void mapping_clear_release_always(struct address_space *mapping)
 	clear_bit(AS_RELEASE_ALWAYS, &mapping->flags);
 }
 
+/*
+ * Set by filesystems that charge the mm dirty counters per dirty page rather
+ * than per folio, so the generic paths must not also account whole folios.
+ */
+static inline void mapping_set_dirty_per_block(struct address_space *mapping)
+{
+	set_bit(AS_DIRTY_PER_BLOCK, &mapping->flags);
+}
+
+static inline bool mapping_dirty_per_block(const struct address_space *mapping)
+{
+	return test_bit(AS_DIRTY_PER_BLOCK, &mapping->flags);
+}
+
 static inline bool mapping_stable_writes(const struct address_space *mapping)
 {
 	return test_bit(AS_STABLE_WRITES, &mapping->flags);
@@ -1311,7 +1326,10 @@ void folio_end_writeback(struct folio *folio);
 void folio_end_writeback_no_dropbehind(struct folio *folio);
 void folio_end_dropbehind(struct folio *folio);
 void folio_wait_stable(struct folio *folio);
-void __folio_mark_dirty(struct folio *folio, struct address_space *, int warn);
+void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
+		int warn, bool accounted);
+bool __filemap_dirty_folio(struct address_space *mapping, struct folio *folio,
+		bool accounted);
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb);
 void __folio_cancel_dirty(struct folio *folio);
 static inline void folio_cancel_dirty(struct folio *folio)
diff --git a/mm/internal.h b/mm/internal.h
index 38b1165212c9..4647aca5fe8d 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -40,6 +40,9 @@ void *workingset_eviction(struct folio *folio,
 void workingset_refault(struct folio *folio, void *shadow);
 void workingset_activation(struct folio *folio);
 
+/* mm/page-writeback.c */
+long folio_accounted_dirty_pages(struct folio *folio);
+
 /* mm/folio.c */
 void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma);
 
diff --git a/mm/migrate.c b/mm/migrate.c
index 15b45832bcfa..0e7b7834a9fd 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -588,6 +588,7 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 	struct zone *oldzone, *newzone;
 	int dirty;
 	long nr = folio_nr_pages(folio);
+	long nr_dirty = nr;
 
 	if (!mapping) {
 		/* Take off deferred split queue while frozen and memcg set */
@@ -614,6 +615,14 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 	oldzone = folio_zone(folio);
 	newzone = folio_zone(newfolio);
 
+	/*
+	 * Only what was charged moves between the zones below, which may be less
+	 * than the whole folio.  Sample it outside the i_pages lock, as it walks
+	 * the per-block state, and while the folio is still dirty.
+	 */
+	if (folio_test_dirty(folio))
+		nr_dirty = folio_accounted_dirty_pages(folio);
+
 	if (folio_test_swapcache(folio))
 		ci = swap_cluster_get_and_lock_irq(folio);
 	else
@@ -708,10 +717,10 @@ static int __folio_migrate_mapping(struct address_space *mapping,
 		}
 #endif
 		if (dirty && mapping_can_writeback(mapping)) {
-			mod_lruvec_state(old_lruvec, NR_FILE_DIRTY, -nr);
-			__mod_zone_page_state(oldzone, NR_ZONE_WRITE_PENDING, -nr);
-			mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr);
-			__mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr);
+			mod_lruvec_state(old_lruvec, NR_FILE_DIRTY, -nr_dirty);
+			__mod_zone_page_state(oldzone, NR_ZONE_WRITE_PENDING, -nr_dirty);
+			mod_lruvec_state(new_lruvec, NR_FILE_DIRTY, nr_dirty);
+			__mod_zone_page_state(newzone, NR_ZONE_WRITE_PENDING, nr_dirty);
 		}
 		rcu_read_unlock();
 	}
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index eeab25d6ce36..6b4a8b6ad34c 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2601,7 +2601,7 @@ EXPORT_SYMBOL(noop_dirty_folio);
  * NOTE: This relies on being atomic wrt interrupts.
  */
 static void folio_account_dirtied(struct folio *folio,
-		struct address_space *mapping)
+		struct address_space *mapping, bool accounted)
 {
 	struct inode *inode = mapping->host;
 
@@ -2614,14 +2614,21 @@ static void folio_account_dirtied(struct folio *folio,
 		inode_attach_wb(inode, folio);
 		wb = inode_to_wb(inode);
 
-		lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
-		if (folio_test_dropbehind(folio))
-			wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr);
-		__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
-		__node_stat_mod_folio(folio, NR_DIRTIED, nr);
-		wb_stat_mod(wb, WB_RECLAIMABLE, nr);
-		wb_stat_mod(wb, WB_DIRTIED, nr);
-		task_io_account_write(nr * PAGE_SIZE);
+		/*
+		 * @accounted means the filesystem charged these per dirty page
+		 * already; the rate limiters below stay per-folio.  Keep in sync
+		 * with folio_account_cleaned() and folio_clear_dirty_for_io().
+		 */
+		if (!accounted) {
+			lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
+			if (folio_test_dropbehind(folio))
+				wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr);
+			__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
+			wb_stat_mod(wb, WB_RECLAIMABLE, nr);
+			__node_stat_mod_folio(folio, NR_DIRTIED, nr);
+			wb_stat_mod(wb, WB_DIRTIED, nr);
+			task_io_account_write(nr * PAGE_SIZE);
+		}
 		current->nr_dirtied += nr;
 		__this_cpu_add(bdp_ratelimits, nr);
 
@@ -2629,13 +2636,41 @@ static void folio_account_dirtied(struct folio *folio,
 	}
 }
 
+/**
+ * folio_accounted_dirty_pages - pages of @folio charged to the dirty counters
+ * @folio: folio to query
+ *
+ * Use this rather than folio_nr_pages() when moving or discharging a folio's dirty
+ * accounting, as a folio is not necessarily charged for all of its pages.  Walks
+ * per-block state that truncation frees, so the caller must hold the folio lock.
+ *
+ * Return: the whole folio for the usual per-folio accounting, otherwise only the
+ * pages holding a dirty block, and zero once those have been discharged.
+ */
+long folio_accounted_dirty_pages(struct folio *folio)
+{
+	struct address_space *mapping = folio->mapping;
+
+	if (mapping && mapping_dirty_per_block(mapping) && folio->private &&
+	    mapping->a_ops->accounted_dirty_pages)
+		return mapping->a_ops->accounted_dirty_pages(folio);
+	return folio_nr_pages(folio);
+}
+
 /*
  * Helper function for deaccounting dirty page without writeback.
  *
  */
 void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)
 {
-	long nr = folio_nr_pages(folio);
+	/*
+	 * Discharge exactly what was charged; zero if the fs already did it.
+	 * Counters below mirror folio_account_dirtied().
+	 */
+	long nr = folio_accounted_dirty_pages(folio);
+
+	if (!nr)
+		return;
 
 	lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
 	if (folio_test_dropbehind(folio))
@@ -2661,7 +2696,7 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)
  * try_to_free_buffers() to fail.
  */
 void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
-			     int warn)
+			     int warn, bool accounted)
 {
 	unsigned long flags;
 
@@ -2674,7 +2709,7 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
 	xa_lock_irqsave(&mapping->i_pages, flags);
 	if (folio->mapping) {	/* Race with truncate? */
 		WARN_ON_ONCE(warn && !folio_test_uptodate(folio));
-		folio_account_dirtied(folio, mapping);
+		folio_account_dirtied(folio, mapping, accounted);
 		__xa_set_mark(&mapping->i_pages, folio->index,
 			      PAGECACHE_TAG_DIRTY);
 	}
@@ -2700,12 +2735,13 @@ void __folio_mark_dirty(struct folio *folio, struct address_space *mapping,
  * simply hold the folio lock, but e.g. zap_pte_range() calls with the
  * folio mapped and the pte lock held, which also locks out truncation.
  */
-bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
+bool __filemap_dirty_folio(struct address_space *mapping, struct folio *folio,
+		bool accounted)
 {
 	if (folio_test_set_dirty(folio))
 		return false;
 
-	__folio_mark_dirty(folio, mapping, !folio_test_private(folio));
+	__folio_mark_dirty(folio, mapping, !folio_test_private(folio), accounted);
 
 	if (mapping->host) {
 		/* !PageAnon && !swapper_space */
@@ -2713,6 +2749,12 @@ bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
 	}
 	return true;
 }
+EXPORT_SYMBOL_GPL(__filemap_dirty_folio);
+
+bool filemap_dirty_folio(struct address_space *mapping, struct folio *folio)
+{
+	return __filemap_dirty_folio(mapping, folio, false);
+}
 EXPORT_SYMBOL(filemap_dirty_folio);
 
 /**
@@ -2912,12 +2954,21 @@ bool folio_clear_dirty_for_io(struct folio *folio)
 		 */
 		wb = unlocked_inode_to_wb_begin(inode, &cookie);
 		if (folio_test_clear_dirty(folio)) {
-			long nr = folio_nr_pages(folio);
-			lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
-			if (folio_test_dropbehind(folio))
-				wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
-			zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
-			wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+			/*
+			 * Per-block folios keep their dirty bits, and the charge,
+			 * until the fs writes them back.  Folios of such a mapping
+			 * that never got per-block state were charged whole, so
+			 * they discharge here as usual.
+			 */
+			if (!mapping_dirty_per_block(mapping) || !folio->private) {
+				long nr = folio_nr_pages(folio);
+
+				lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+				if (folio_test_dropbehind(folio))
+					wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
+				zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
+				wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
+			}
 			ret = true;
 		}
 		unlocked_inode_to_wb_end(inode, &cookie);
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

[PATCH 2/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Salvatore Dipietro]

The mm dirty counters (NR_FILE_DIRTY, NR_ZONE_WRITE_PENDING,
WB_RECLAIMABLE) are charged per folio, which charges up to 512
pages for a single dirty 4 kB block of a 2 MB folio.  iomap knows
exactly which blocks are dirty and writeback only writes those
ranges, so balance_dirty_pages() ends up throttling writers
against a dirty count far larger than the data that actually
needs writeback.

Charge the counters per dirty page for folios carrying per-block
dirty state, using the infrastructure from the previous patch:
struct iomap_folio_state counts the pages holding a dirty block, and
each dirty bitmap change charges or discharges the difference.  The
charge survives folio_clear_dirty_for_io() and is dropped when
writeback or invalidation clears the blocks.  The monotonic
NR_DIRTIED / WB_DIRTIED / task_io totals are charged only for newly
dirtied data, and the dirty rate limiters stay per-folio.

Every iomap-based filesystem sets .accounted_dirty_pages alongside
.dirty_folio = iomap_dirty_folio.

On pgbench simple-update (1024 clients, XFS, 96-vCPU arm64,
10 iterations) on v7.3-rc7, the unpatched kernel charges 159 TB
of dirtied pages while writing 6 TB to the array, a factor of 27;
with this patch the dirtied pages charged match the data written.
Unpatched, nr_dirty is above dirty_threshold in ~17% of 5s samples
and its peak exceeds it in 9 of 10 iterations; patched, it never
reaches the threshold.

On v7.3-rc7, average throughput rises 46.8% (139.8k -> 205.2k tps)
and run-to-run variability drops from 23.60% to 3.98% coefficient
of variation.  The best iteration improves by 8% (199.4k -> 215.9k
tps); most of the gain comes from the worst iteration, up from 97.9k
to 191.5k tps, i.e. from throttling episodes that no longer happen.

Fixes: 4ce02c679722 ("iomap: Add per-block dirty state tracking to improve performance")
Cc: stable@vger.kernel.org # v6.17+
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
Reviewed-by: Hazem Mohamed Abuelfotoh <abuehaze@amazon.com>
---
 fs/exfat/inode.c       |   1 +
 fs/fuse/file.c         |   1 +
 fs/gfs2/aops.c         |   1 +
 fs/iomap/buffered-io.c | 167 +++++++++++++++++++++++++++++++++++++++--
 fs/ntfs/aops.c         |   2 +
 fs/ntfs3/inode.c       |   2 +
 fs/xfs/xfs_aops.c      |   1 +
 fs/zonefs/file.c       |   1 +
 include/linux/iomap.h  |   1 +
 9 files changed, 171 insertions(+), 6 deletions(-)

diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index ccd13630187e..48704fe5d59b 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -302,6 +302,7 @@ static const struct address_space_operations exfat_aops = {
 	.readahead		= exfat_readahead,
 	.writepages		= exfat_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= exfat_aop_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 9a36d0329e22..eb7179f854ee 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -3101,6 +3101,7 @@ static const struct address_space_operations fuse_file_aops  = {
 	.writepages	= fuse_writepages,
 	.launder_folio	= fuse_launder_folio,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 	.is_partially_uptodate = iomap_is_partially_uptodate,
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 0a7b8076af3a..ea77af00b111 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -713,6 +713,7 @@ static const struct address_space_operations gfs2_aops = {
 	.read_folio = gfs2_read_folio,
 	.readahead = gfs2_readahead,
 	.dirty_folio = iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio = iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 	.bmap = gfs2_bmap,
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 0a5ebfda90f1..75671b29f7cc 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -5,6 +5,8 @@
  */
 #include <linux/iomap.h>
 #include <linux/buffer_head.h>
+#include <linux/backing-dev.h>
+#include <linux/task_io_accounting_ops.h>
 #include <linux/writeback.h>
 #include <linux/swap.h>
 #include <linux/migrate.h>
@@ -24,6 +26,13 @@ struct iomap_folio_state {
 	unsigned int		read_bytes_pending;
 	atomic_t		write_bytes_pending;
 
+	/*
+	 * Pages holding at least one dirty block, the unit the mm dirty counters
+	 * work in.  Maintained under @state_lock alongside the dirty bits below,
+	 * so it is what those counters were charged for.
+	 */
+	unsigned int		dirty_pages;
+
 	/*
 	 * Each block has two bits in this bitmap:
 	 * Bits [0..blocks_per_folio) has the uptodate status.
@@ -180,6 +189,89 @@ static unsigned iomap_find_dirty_range(struct folio *folio, u64 *range_start,
 	return range_end - *range_start;
 }
 
+/*
+ * Count pages in [first_blk, last_blk] holding at least one dirty block.  Callers
+ * take this before and after flipping the bitmap for an exact delta, so each page
+ * is scanned over its full block span to land in the same page in both counts.
+ * state_lock must be held.
+ */
+static unsigned int ifs_count_dirty_pages(struct folio *folio,
+		struct iomap_folio_state *ifs, unsigned int first_blk,
+		unsigned int last_blk)
+{
+	struct inode *inode = folio->mapping->host;
+	unsigned int blks_per_folio = i_blocks_per_folio(inode, folio);
+	unsigned int blkbits = inode->i_blkbits;
+	size_t fsize = folio_size(folio);
+	size_t start = (size_t)first_blk << blkbits;
+	size_t end = min(((size_t)last_blk << blkbits) + (1UL << blkbits) - 1,
+			 fsize - 1);
+	unsigned long first_pg = start >> PAGE_SHIFT;
+	unsigned long last_pg = end >> PAGE_SHIFT;
+	unsigned int dirty_pages = 0;
+	unsigned long pg;
+
+	for (pg = first_pg; pg <= last_pg; pg++) {
+		size_t pg_start = pg << PAGE_SHIFT;
+		size_t pg_end = min(pg_start + PAGE_SIZE - 1, fsize - 1);
+		unsigned int b_lo = pg_start >> blkbits;
+		unsigned int b_hi = pg_end >> blkbits;
+		unsigned int b;
+
+		for (b = b_lo; b <= b_hi; b++) {
+			if (test_bit(b + blks_per_folio, ifs->state)) {
+				dirty_pages++;
+				break;
+			}
+		}
+	}
+	return dirty_pages;
+}
+
+/*
+ * Mirror folio_account_dirtied() in page-sized increments; @nr_pages is positive
+ * to charge and negative to discharge.
+ *
+ * The NR_DIRTIED/WB_DIRTIED/task_io totals are monotonic, so only @new_dirty data
+ * may charge them; writeback re-marks an already charged folio.
+ *
+ * Only the folio lock is held, not i_pages, hence the IRQ-safe stat helpers.
+ */
+static void iomap_account_dirty_pages(struct folio *folio, long nr_pages,
+		bool new_dirty)
+{
+	struct address_space *mapping = folio->mapping;
+	struct inode *inode = mapping->host;
+	struct wb_lock_cookie cookie = {};
+	struct bdi_writeback *wb;
+	bool dirtied = nr_pages > 0 && new_dirty;
+
+	if (!nr_pages || !mapping_can_writeback(mapping))
+		return;
+
+	lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr_pages);
+	zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr_pages);
+	if (dirtied)
+		node_stat_mod_folio(folio, NR_DIRTIED, nr_pages);
+
+	/* Pin the wb against a concurrent inode_do_switch_wbs(). */
+	inode_attach_wb(inode, folio);
+	wb = unlocked_inode_to_wb_begin(inode, &cookie);
+	wb_stat_mod(wb, WB_RECLAIMABLE, nr_pages);
+	if (folio_test_dropbehind(folio))
+		wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr_pages);
+	if (dirtied)
+		wb_stat_mod(wb, WB_DIRTIED, nr_pages);
+	unlocked_inode_to_wb_end(inode, &cookie);
+
+	if (dirtied) {
+		task_io_account_write(nr_pages * PAGE_SIZE);
+	} else if (nr_pages < 0 && !folio_test_writeback(folio)) {
+		/* truncated or invalidated before reaching disk, so refund */
+		task_io_account_cancelled_write((-nr_pages) * PAGE_SIZE);
+	}
+}
+
 /*
  * Clear the per-block dirty bits for the range [@off, @off + @len) within a
  * folio.  The range is rounded inwards so that only blocks fully covered by
@@ -195,15 +287,22 @@ static void ifs_clear_range_dirty(struct folio *folio,
 	unsigned int first_blk = round_up(off, i_blocksize(inode)) >>
 				 inode->i_blkbits;
 	unsigned int last_blk = (off + len) >> inode->i_blkbits;
+	unsigned int before, after;
 	unsigned long flags;
 
 	if (first_blk >= last_blk)
 		return;
 
 	spin_lock_irqsave(&ifs->state_lock, flags);
+	before = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk - 1);
 	bitmap_clear(ifs->state, first_blk + blks_per_folio,
 		     last_blk - first_blk);
+	after = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk - 1);
+	ifs->dirty_pages -= before - after;
 	spin_unlock_irqrestore(&ifs->state_lock, flags);
+
+	/* Pages that lost their last dirty block, so before >= after. */
+	iomap_account_dirty_pages(folio, -(long)(before - after), false);
 }
 
 static void iomap_clear_range_dirty(struct folio *folio, size_t off, size_t len)
@@ -222,11 +321,13 @@ static void iomap_clear_range_dirty(struct folio *folio, size_t off, size_t len)
  * data loss when partial blocks are written.
  */
 static void ifs_set_range_dirty(struct folio *folio,
-		struct iomap_folio_state *ifs, size_t off, size_t len)
+		struct iomap_folio_state *ifs, size_t off, size_t len,
+		bool new_dirty)
 {
 	struct inode *inode = folio->mapping->host;
 	unsigned int blks_per_folio = i_blocks_per_folio(inode, folio);
 	unsigned int first_blk, last_blk;
+	unsigned int before, after;
 	unsigned long flags;
 
 	if (!len)
@@ -235,9 +336,15 @@ static void ifs_set_range_dirty(struct folio *folio,
 	first_blk = off >> inode->i_blkbits;
 	last_blk = (off + len - 1) >> inode->i_blkbits;
 	spin_lock_irqsave(&ifs->state_lock, flags);
+	before = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk);
 	bitmap_set(ifs->state, first_blk + blks_per_folio,
 		   last_blk - first_blk + 1);
+	after = ifs_count_dirty_pages(folio, ifs, first_blk, last_blk);
+	ifs->dirty_pages += after - before;
 	spin_unlock_irqrestore(&ifs->state_lock, flags);
+
+	/* Pages that gained their first dirty block, so after >= before. */
+	iomap_account_dirty_pages(folio, (long)(after - before), new_dirty);
 }
 
 static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)
@@ -245,7 +352,20 @@ static void iomap_set_range_dirty(struct folio *folio, size_t off, size_t len)
 	struct iomap_folio_state *ifs = folio->private;
 
 	if (ifs)
-		ifs_set_range_dirty(folio, ifs, off, len);
+		ifs_set_range_dirty(folio, ifs, off, len, true);
+}
+
+/*
+ * Set the dirty bits for blocks that were already dirty, and already counted,
+ * before this folio had per-block state: the reclaimable counters still move,
+ * but the dirtied totals must not be charged twice.
+ */
+static void iomap_redirty_range(struct folio *folio, size_t off, size_t len)
+{
+	struct iomap_folio_state *ifs = folio->private;
+
+	if (ifs)
+		ifs_set_range_dirty(folio, ifs, off, len, false);
 }
 
 static struct iomap_folio_state *ifs_alloc(struct inode *inode,
@@ -276,9 +396,13 @@ static struct iomap_folio_state *ifs_alloc(struct inode *inode,
 	spin_lock_init(&ifs->state_lock);
 	if (folio_test_uptodate(folio))
 		bitmap_set(ifs->state, 0, nr_blocks);
-	if (folio_test_dirty(folio))
+	if (folio_test_dirty(folio)) {
 		bitmap_set(ifs->state, nr_blocks, nr_blocks);
+		ifs->dirty_pages = folio_nr_pages(folio);
+	}
 	folio_attach_private(folio, ifs);
+	/* The mm dirty counters are now charged per dirty page for this file. */
+	mapping_set_dirty_per_block(inode->i_mapping);
 
 	return ifs;
 }
@@ -839,6 +963,11 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len)
 	 */
 	if (offset == 0 && len == folio_size(folio)) {
 		WARN_ON_ONCE(folio_test_writeback(folio));
+		/*
+		 * Discharge while the bitmap is still attached; ifs_free() below
+		 * drops the state that records how many pages were charged.
+		 */
+		iomap_clear_range_dirty(folio, 0, folio_size(folio));
 		folio_cancel_dirty(folio);
 		ifs_free(folio);
 	} else {
@@ -854,10 +983,31 @@ bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio)
 
 	ifs_alloc(inode, folio, 0);
 	iomap_set_range_dirty(folio, 0, len);
-	return filemap_dirty_folio(mapping, folio);
+	return __filemap_dirty_folio(mapping, folio, folio->private);
 }
 EXPORT_SYMBOL_GPL(iomap_dirty_folio);
 
+/**
+ * iomap_accounted_dirty_pages - pages of @folio charged to the dirty counters
+ * @folio: folio to query
+ *
+ * For callers that discharge what iomap charged.  The caller must hold the folio
+ * lock, and @folio must have per-block state attached.
+ *
+ * Return: pages of @folio holding at least one dirty block, zero once they have
+ * been cleared, so a later discharge does not double count.
+ */
+long iomap_accounted_dirty_pages(struct folio *folio)
+{
+	struct iomap_folio_state *ifs = folio->private;
+
+	if (WARN_ON_ONCE(!ifs))
+		return 0;
+
+	return ifs->dirty_pages;
+}
+EXPORT_SYMBOL_GPL(iomap_accounted_dirty_pages);
+
 static void
 iomap_write_failed(struct inode *inode, loff_t pos, unsigned len)
 {
@@ -1137,7 +1287,7 @@ static bool __iomap_write_end(struct inode *inode, loff_t pos, size_t len,
 		return false;
 	iomap_set_range_uptodate(folio, offset_in_folio(folio, pos), len);
 	iomap_set_range_dirty(folio, offset_in_folio(folio, pos), copied);
-	filemap_dirty_folio(inode->i_mapping, folio);
+	__filemap_dirty_folio(inode->i_mapping, folio, folio->private);
 	return true;
 }
 
@@ -2001,8 +2151,13 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)
 
 	if (i_blocks_per_folio(inode, folio) > 1) {
 		if (!ifs) {
+			/*
+			 * Dirtied before it had per-block state, so charged
+			 * per-folio.  Re-mark the blocks for the walk below,
+			 * paired with the clear at the end of this function.
+			 */
 			ifs = ifs_alloc(inode, folio, 0);
-			iomap_set_range_dirty(folio, 0, end_pos - pos);
+			iomap_redirty_range(folio, 0, end_pos - pos);
 		}
 
 		iomap_writeback_init(inode, folio);
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
index 517d9a1563e7..d26ddd5ae118 100644
--- a/fs/ntfs/aops.c
+++ b/fs/ntfs/aops.c
@@ -308,6 +308,7 @@ const struct address_space_operations ntfs_aops = {
 	.readahead		= ntfs_readahead,
 	.writepages		= ntfs_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= ntfs_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
@@ -322,6 +323,7 @@ const struct address_space_operations ntfs_mft_aops = {
 	.readahead		= ntfs_readahead,
 	.writepages		= ntfs_mft_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.bmap			= ntfs_bmap,
 	.migrate_folio		= filemap_migrate_folio,
 	.is_partially_uptodate	= iomap_is_partially_uptodate,
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index 4ac26c80bd34..509570091e48 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -2301,6 +2301,7 @@ const struct address_space_operations ntfs_aops = {
 	.writepages	= ntfs_writepages,
 	.bmap		= ntfs_bmap,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.migrate_folio	= filemap_migrate_folio,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
@@ -2309,6 +2310,7 @@ const struct address_space_operations ntfs_aops = {
 const struct address_space_operations ntfs_aops_cmpr = {
 	.read_folio	= ntfs_read_folio,
 	.dirty_folio	= iomap_dirty_folio,
+	.accounted_dirty_pages = iomap_accounted_dirty_pages,
 	.release_folio	= iomap_release_folio,
 	.invalidate_folio = iomap_invalidate_folio,
 };
diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c
index 8b6119776fb3..869248241f3b 100644
--- a/fs/xfs/xfs_aops.c
+++ b/fs/xfs/xfs_aops.c
@@ -682,6 +682,7 @@ const struct address_space_operations xfs_address_space_operations = {
 	.readahead		= xfs_vm_readahead,
 	.writepages		= xfs_vm_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.release_folio		= iomap_release_folio,
 	.invalidate_folio	= iomap_invalidate_folio,
 	.bmap			= xfs_vm_bmap,
diff --git a/fs/zonefs/file.c b/fs/zonefs/file.c
index 5b34849be7a2..9847ce9cd68b 100644
--- a/fs/zonefs/file.c
+++ b/fs/zonefs/file.c
@@ -192,6 +192,7 @@ const struct address_space_operations zonefs_file_aops = {
 	.readahead		= zonefs_readahead,
 	.writepages		= zonefs_writepages,
 	.dirty_folio		= iomap_dirty_folio,
+	.accounted_dirty_pages	= iomap_accounted_dirty_pages,
 	.release_folio		= iomap_release_folio,
 	.invalidate_folio	= iomap_invalidate_folio,
 	.migrate_folio		= filemap_migrate_folio,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index bc7ae6327dbf..fbf6480ac928 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -444,6 +444,7 @@ bool iomap_release_folio(struct folio *folio, gfp_t gfp_flags);
 void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len);
 bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio);
 void iomap_folio_mark_uptodate(struct folio *folio);
+long iomap_accounted_dirty_pages(struct folio *folio);
 int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len,
 		const struct iomap_ops *ops,
 		const struct iomap_write_ops *write_ops);
-- 
2.50.1




AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[David Hildenbrand (Arm)]

On 10/27/26 12:27, Salvatore Dipietro wrote:
> Since iomap gained per-block dirty tracking, a large folio can be
> partially dirty: struct iomap_folio_state records exactly which
> blocks are dirty, and writeback only writes those ranges.  The mm
> dirty accounting, however, still charges the whole folio even if a
> single block is dirty (up to 512x over-accounting for a 2 MB folio
> over 4 KB blocks).
> 
> balance_dirty_pages() then throttles writers against a number that
> is unrelated to how much data actually needs writing back.  On
> workloads that repeatedly dirty small ranges of large folios --
> e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> is crossed and writers are throttled long before the real dirty data
> justifies it.  On the benchmark below, run on v7.3-rc7, the
> unpatched kernel charges ~159 TB of dirtied pages while writing
> ~6 TB to the array, a factor of 27; with this series the dirtied
> pages charged match the data actually written.
> 
> This series makes iomap charge and discharge the mm dirty counters
> per dirty page for folios that carry per-block dirty state, so a
> partially dirty large folio is accounted for the dirty part only:
> 
>   - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
>     mapping flag, an @accounted argument to __folio_mark_dirty()
>     and __filemap_dirty_folio() to skip the generic per-folio
>     charge, and an address_space_operations::accounted_dirty_pages()
>     callback used by the discharge and migration paths instead of
>     folio_nr_pages().  No filesystem opts in, so it is a no-op on
>     its own.
> 
>   - Patch 2 makes iomap charge per dirty page from the per-block
>     dirty bitmap and wires iomap_accounted_dirty_pages() into every
>     iomap-based filesystem.
> 
> This was benchmarked with PostgreSQL using the pgbench simple-update
> workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
> 205.2k tps) and run-to-run variability drops from a 23.60% to a
> 3.98% coefficient of variation: the worst iteration improves from
> 97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.
> 
> Ten iterations per kernel, same database and workload (tps):
> 
>   Kernel                     Avg     Min     Max      CV   iters   iters
>                                                              1-5    6-10
>   ______________________  ______  ______  ______  ______  ______  ______
>   v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
>   v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k
> 
> On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
> and throughput decays as the run progresses: iterations 6-10 average
> 30% below iterations 1-5.  With this series the decay is gone -- the
> second half is similar to the first -- and the coefficient of
> variation stays at 3.98%.
> 
> The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
> per-block dirty state tracking to improve performance"), merged in
> v6.6, where the ifs bitmap started recording partial dirty state while
> the mm counters kept charging whole folios.
> 
> The throughput collapse bisects to the v6.17 readahead series, first
> bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
> file_ra_state"), which lets readahead stay at the largest folio order
> once it has ramped up, so a single dirty block charges far more than
> is written.  Patch 2 therefore carries:


How does this relate to

[RFC PATCH 0/5] mm: sub-folio dirty tracking for PTE-mapped mmap writes

https://lore.kernel.org/all/20260903182943.662461-1-kirill@shutemov.name/

?

I assume they try to solve different problems, but would more fine-grained folio
dirtying also make your solution benefit?

-- 
Cheers,

David

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Kiryl Shutsemau]

On Tue, Oct 27, 2026 at 01:22:22PM +0100, David Hildenbrand (Arm) wrote:
> On 10/27/26 12:27, Salvatore Dipietro wrote:
> > Since iomap gained per-block dirty tracking, a large folio can be
> > partially dirty: struct iomap_folio_state records exactly which
> > blocks are dirty, and writeback only writes those ranges.  The mm
> > dirty accounting, however, still charges the whole folio even if a
> > single block is dirty (up to 512x over-accounting for a 2 MB folio
> > over 4 KB blocks).
> > 
> > balance_dirty_pages() then throttles writers against a number that
> > is unrelated to how much data actually needs writing back.  On
> > workloads that repeatedly dirty small ranges of large folios --
> > e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> > is crossed and writers are throttled long before the real dirty data
> > justifies it.  On the benchmark below, run on v7.3-rc7, the
> > unpatched kernel charges ~159 TB of dirtied pages while writing
> > ~6 TB to the array, a factor of 27; with this series the dirtied
> > pages charged match the data actually written.
> > 
> > This series makes iomap charge and discharge the mm dirty counters
> > per dirty page for folios that carry per-block dirty state, so a
> > partially dirty large folio is accounted for the dirty part only:
> > 
> >   - Patch 1 adds the mm infrastructure: an AS_DIRTY_PER_BLOCK
> >     mapping flag, an @accounted argument to __folio_mark_dirty()
> >     and __filemap_dirty_folio() to skip the generic per-folio
> >     charge, and an address_space_operations::accounted_dirty_pages()
> >     callback used by the discharge and migration paths instead of
> >     folio_nr_pages().  No filesystem opts in, so it is a no-op on
> >     its own.
> > 
> >   - Patch 2 makes iomap charge per dirty page from the per-block
> >     dirty bitmap and wires iomap_accounted_dirty_pages() into every
> >     iomap-based filesystem.
> > 
> > This was benchmarked with PostgreSQL using the pgbench simple-update
> > workload on v7.3-rc7.  Average throughput rises 46.8% (139.8k ->
> > 205.2k tps) and run-to-run variability drops from a 23.60% to a
> > 3.98% coefficient of variation: the worst iteration improves from
> > 97.9k to 191.5k tps, while the best gains 8%, 199.4k -> 215.9k.
> > 
> > Ten iterations per kernel, same database and workload (tps):
> > 
> >   Kernel                     Avg     Min     Max      CV   iters   iters
> >                                                              1-5    6-10
> >   ______________________  ______  ______  ______  ______  ______  ______
> >   v7.3-rc7                139.8k   97.9k  199.4k  23.60%  164.4k  115.2k
> >   v7.3-rc7 + this series  205.2k  191.5k  215.9k   3.98%  200.1k  210.2k
> > 
> > On unpatched v7.3-rc7 the worst iteration lands ~51% below the best,
> > and throughput decays as the run progresses: iterations 6-10 average
> > 30% below iterations 1-5.  With this series the decay is gone -- the
> > second half is similar to the first -- and the coefficient of
> > variation stays at 3.98%.
> > 
> > The over-accounting has existed since commit 4ce02c679722 ("iomap: Add
> > per-block dirty state tracking to improve performance"), merged in
> > v6.6, where the ifs bitmap started recording partial dirty state while
> > the mm counters kept charging whole folios.
> > 
> > The throughput collapse bisects to the v6.17 readahead series, first
> > bad commit c4602f9fa77f ("mm/readahead: store folio order in struct
> > file_ra_state"), which lets readahead stay at the largest folio order
> > once it has ramped up, so a single dirty block charges far more than
> > is written.  Patch 2 therefore carries:
> 
> 
> How does this relate to
> 
> [RFC PATCH 0/5] mm: sub-folio dirty tracking for PTE-mapped mmap writes
> 
> https://lore.kernel.org/all/20260903182943.662461-1-kirill@shutemov.name/
> 
> ?
> 
> I assume they try to solve different problems, but would more fine-grained folio
> dirtying also make your solution benefit?

Yeah, it solves different problem.

And I abandoned my patchset. The right way to do this is to make
page_mkwrite() per-page, not folio.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH 0/2] mm, iomap: account dirty at page granularity for per-block dirty folios

[Pedro Falcato]

On Tue, Oct 27, 2026 at 11:27:00AM +0000, Salvatore Dipietro wrote:
> Since iomap gained per-block dirty tracking, a large folio can be
> partially dirty: struct iomap_folio_state records exactly which
> blocks are dirty, and writeback only writes those ranges.  The mm
> dirty accounting, however, still charges the whole folio even if a
> single block is dirty (up to 512x over-accounting for a 2 MB folio
> over 4 KB blocks).
> 
> balance_dirty_pages() then throttles writers against a number that
> is unrelated to how much data actually needs writing back.  On
> workloads that repeatedly dirty small ranges of large folios --
> e.g. PostgreSQL, which writes mostly 8/16 KB pages -- dirty_thresh
> is crossed and writers are throttled long before the real dirty data
> justifies it.  On the benchmark below, run on v7.3-rc7, the
> unpatched kernel charges ~159 TB of dirtied pages while writing
> ~6 TB to the array, a factor of 27; with this series the dirtied
> pages charged match the data actually written.
> 
> This series makes iomap charge and discharge the mm dirty counters
> per dirty page for folios that carry per-block dirty state, so a
> partially dirty large folio is accounted for the dirty part only:

Why? This looks wrong. Yes, you may only have dirtied e.g 4K out of a 2MB
folio, but the whole 2MB folio is still effectively pinned by the VM, awaiting
cleaning + writeback. And that's precisely what we want to measure and
throttle in certain cases, so that e.g the VM system never finds itself
short of clean pages it can reclaim, and no easy way to start writeback
on them.

-- 
Pedro

[syzbot ci] Re: mm, iomap: account dirty at page granularity for per-block dirty folios

[syzbot ci]

syzbot ci has tested the following series

[v1] mm, iomap: account dirty at page granularity for per-block dirty folios
https://lore.kernel.org/all/20261027112702.3747009-1-dipiets@amazon.it
* [PATCH 1/2] mm: add infrastructure for per-dirty-page accounting
* [PATCH 2/2] mm, iomap: account dirty at page granularity for per-block dirty folios

and found the following issues:
* KASAN: slab-out-of-bounds Read in folio_accounted_dirty_pages
* KASAN: slab-use-after-free Read in folio_accounted_dirty_pages

Full report is available here:
https://ci.syzbot.org/series/05041e6c-d357-4fc5-9bbe-6aae1760d639

***

KASAN: slab-out-of-bounds Read in folio_accounted_dirty_pages

tree:      axboe
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/axboe/linux.git
base:      24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
arch:      amd64
compiler:  Debian clang version 22.1.8 (1~deb13u4), Debian LLD 22.1.8
config:    https://ci.syzbot.org/builds/9865cf75-69fb-4fbd-b5fe-d0be9cd03802/config
syz repro: https://ci.syzbot.org/findings/98cdf846-0b4c-4f94-8430-3b7db9efe057/syz_repro

==================================================================
BUG: KASAN: slab-out-of-bounds in instrument_atomic_read include/linux/instrumented.h:82 [inline]
BUG: KASAN: slab-out-of-bounds in _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
BUG: KASAN: slab-out-of-bounds in mapping_dirty_per_block include/linux/pagemap.h:319 [inline]
BUG: KASAN: slab-out-of-bounds in folio_accounted_dirty_pages+0x53/0x1f0 mm/page-writeback.c:2654
Read of size 8 at addr ffff888168669f89 by task syz.1.47/5860

CPU: 0 UID: 0 PID: 5860 Comm: syz.1.47 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
 <TASK>
 dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
 print_address_description+0x55/0x1e0 mm/kasan/report.c:378
 print_report+0x58/0x70 mm/kasan/report.c:482
 kasan_report+0x117/0x150 mm/kasan/report.c:595
 check_region_inline mm/kasan/generic.c:-1 [inline]
 kasan_check_range+0x264/0x2c0 mm/kasan/generic.c:200
 instrument_atomic_read include/linux/instrumented.h:82 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 mapping_dirty_per_block include/linux/pagemap.h:319 [inline]
 folio_accounted_dirty_pages+0x53/0x1f0 mm/page-writeback.c:2654
 __folio_migrate_mapping+0x3d0/0x2830 mm/migrate.c:624
 __migrate_folio mm/migrate.c:893 [inline]
 migrate_folio+0x10a/0x1a0 mm/migrate.c:920
 move_to_new_folio+0x1ed/0x460 mm/migrate.c:1135
 migrate_folio_move mm/migrate.c:1401 [inline]
 migrate_folios_move mm/migrate.c:1758 [inline]
 migrate_pages_batch+0x30b3/0x4f50 mm/migrate.c:2014
 migrate_pages_sync mm/migrate.c:2044 [inline]
 migrate_pages+0x1b9f/0x2920 mm/migrate.c:2153
 migrate_to_node mm/mempolicy.c:1322 [inline]
 do_migrate_pages+0x656/0x8b0 mm/mempolicy.c:1421
 kernel_migrate_pages mm/mempolicy.c:1935 [inline]
 __do_sys_migrate_pages mm/mempolicy.c:1953 [inline]
 __se_sys_migrate_pages+0xaa2/0xc90 mm/mempolicy.c:1949
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f04fcf9f559
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f04fc5fe028 EFLAGS: 00000246 ORIG_RAX: 0000000000000100
RAX: ffffffffffffffda RBX: 00007f04fd225fa0 RCX: 00007f04fcf9f559
RDX: 0000200000000040 RSI: 0000000000000009 RDI: 0000000000000000
RBP: 00007f04fd036b20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000200000000380 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f04fd226038 R14: 00007f04fd225fa0 R15: 00007ffc667db8b8
 </TASK>

Allocated by task 5729:
 kasan_save_stack mm/kasan/common.c:57 [inline]
 kasan_save_track+0x3e/0x80 mm/kasan/common.c:78
 unpoison_slab_object mm/kasan/common.c:351 [inline]
 __kasan_slab_alloc+0x6c/0x80 mm/kasan/common.c:377
 kasan_slab_alloc include/linux/kasan.h:253 [inline]
 slab_post_alloc_hook mm/slub.c:4683 [inline]
 slab_alloc_node mm/slub.c:4996 [inline]
 kmem_cache_alloc_noprof+0x2b9/0x600 mm/slub.c:5010
 anon_vma_alloc mm/rmap.c:94 [inline]
 anon_vma_fork+0x88/0x540 mm/rmap.c:395
 dup_mmap+0x954/0x1dc0 mm/mmap.c:1781
 dup_mm kernel/fork.c:1543 [inline]
 copy_mm+0x115/0x480 kernel/fork.c:1595
 copy_process+0x1e79/0x4400 kernel/fork.c:2311
 kernel_clone+0x2d7/0x940 kernel/fork.c:2771
 __do_sys_clone kernel/fork.c:2913 [inline]
 __se_sys_clone kernel/fork.c:2897 [inline]
 __x64_sys_clone+0x1b6/0x230 kernel/fork.c:2897
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

The buggy address belongs to the object at ffff888168669e70
 which belongs to the cache anon_vma of size 200
The buggy address is located 81 bytes to the right of
 allocated 200-byte region [ffff888168669e70, ffff888168669f38)

The buggy address belongs to the physical page:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff888168669108 pfn:0x168669
memcg:ffff88816db0e201
flags: 0x57ff00000000200(workingset|node=1|zone=2|lastcpupid=0x7ff)
page_type: f5(slab)
raw: 057ff00000000200 ffff888101687140 ffff88816040c7c8 ffffea0005b62950
raw: ffff888168669108 00000002000f000b 00000000f5000000 ffff88816db0e201
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0xd2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 5535, tgid 5535 (rm), ts 43255388535
 set_page_owner include/linux/page_owner.h:33 [inline]
 post_alloc_hook+0x1f9/0x250 mm/page_alloc.c:1871
 prep_new_page mm/page_alloc.c:1879 [inline]
 get_page_from_freelist+0x2209/0x2280 mm/page_alloc.c:3943
 __alloc_frozen_pages_noprof+0x217/0x5a0 mm/page_alloc.c:5436
 alloc_slab_page mm/slub.c:3347 [inline]
 allocate_slab+0x7d/0x620 mm/slub.c:3462
 new_slab mm/slub.c:3513 [inline]
 refill_objects+0x2d5/0x350 mm/slub.c:7418
 refill_sheaf mm/slub.c:2885 [inline]
 __pcs_replace_empty_main+0x2c8/0x6c0 mm/slub.c:4774
 alloc_from_pcs mm/slub.c:4850 [inline]
 slab_alloc_node mm/slub.c:4984 [inline]
 kmem_cache_alloc_noprof+0x399/0x600 mm/slub.c:5010
 anon_vma_alloc mm/rmap.c:94 [inline]
 __anon_vma_prepare+0x10e/0x4d0 mm/rmap.c:201
 __vmf_anon_prepare mm/memory.c:3961 [inline]
 vmf_anon_prepare mm/internal.h:572 [inline]
 do_anonymous_page mm/memory.c:5478 [inline]
 do_pte_missing+0x2f18/0x34b0 mm/memory.c:4698
 handle_pte_fault mm/memory.c:6558 [inline]
 __handle_mm_fault mm/memory.c:6707 [inline]
 handle_mm_fault+0x1a53/0x3750 mm/memory.c:6876
 do_user_addr_fault+0xa4f/0x1340 arch/x86/mm/fault.c:1343
 handle_page_fault arch/x86/mm/fault.c:1483 [inline]
 exc_page_fault+0x6a/0xc0 arch/x86/mm/fault.c:1536
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:595
page last free pid 5049 tgid 5049 ts 18476194117 stack trace:
 reset_page_owner include/linux/page_owner.h:26 [inline]
 __free_pages_prepare mm/page_alloc.c:1418 [inline]
 __free_frozen_pages+0xc93/0xd90 mm/page_alloc.c:2962
 __slab_free+0x274/0x2c0 mm/slub.c:5823
 qlink_free mm/kasan/quarantine.c:163 [inline]
 qlist_free_all+0x99/0x100 mm/kasan/quarantine.c:179
 kasan_quarantine_reduce+0x148/0x160 mm/kasan/quarantine.c:286
 __kasan_slab_alloc+0x22/0x80 mm/kasan/common.c:361
 kasan_slab_alloc include/linux/kasan.h:253 [inline]
 slab_post_alloc_hook mm/slub.c:4683 [inline]
 slab_alloc_node mm/slub.c:4996 [inline]
 kmem_cache_alloc_node_noprof+0x354/0x610 mm/slub.c:5068
 __alloc_skb+0x1db/0x7a0 net/core/skbuff.c:704
 alloc_skb include/linux/skbuff.h:1384 [inline]
 alloc_skb_with_frags+0xc6/0x760 net/core/skbuff.c:6803
 sock_alloc_send_pskb+0x878/0x990 net/core/sock.c:3022
 unix_dgram_sendmsg+0x4da/0x17b0 net/unix/af_unix.c:2144
 sock_sendmsg_nosec+0x13a/0x180 net/socket.c:800
 __sock_sendmsg net/socket.c:815 [inline]
 sock_write_iter+0x2de/0x3e0 net/socket.c:1266
 new_sync_write fs/read_write.c:595 [inline]
 vfs_write+0x612/0xba0 fs/read_write.c:687
 ksys_write+0x150/0x270 fs/read_write.c:739
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

Memory state around the buggy address:
 ffff888168669e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 ffff888168669f00: 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc fc
>ffff888168669f80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
                      ^
 ffff88816866a000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88816866a080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fc fc
==================================================================


***

KASAN: slab-use-after-free Read in folio_accounted_dirty_pages

tree:      axboe
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/axboe/linux.git
base:      24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
arch:      amd64
compiler:  Debian clang version 22.1.8 (1~deb13u4), Debian LLD 22.1.8
config:    https://ci.syzbot.org/builds/9865cf75-69fb-4fbd-b5fe-d0be9cd03802/config
syz repro: https://ci.syzbot.org/findings/f9e47a91-9c37-465b-b43c-7404e404cbfd/syz_repro

==================================================================
BUG: KASAN: slab-use-after-free in instrument_atomic_read include/linux/instrumented.h:82 [inline]
BUG: KASAN: slab-use-after-free in _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
BUG: KASAN: slab-use-after-free in mapping_dirty_per_block include/linux/pagemap.h:319 [inline]
BUG: KASAN: slab-use-after-free in folio_accounted_dirty_pages+0x53/0x1f0 mm/page-writeback.c:2654
Read of size 8 at addr ffff888111a3d539 by task syz.0.17/5870

CPU: 0 UID: 0 PID: 5870 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
Call Trace:
 <TAS
 <TASK>
 dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120
 print_address_description+0x55/0x1e0 mm/kasan/report.c:378
 print_report+0x58/0x70 mm/kasan/report.c:482
 kasan_report+0x117/0x150 mm/kasan/report.c:595
 check_region_inline mm/kasan/generic.c:-1 [inline]
 kasan_check_range+0x264/0x2c0 mm/kasan/generic.c:200
 instrument_atomic_read include/linux/instrumented.h:82 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 mapping_dirty_per_block include/linux/pagemap.h:319 [inline]
 folio_accounted_dirty_pages+0x53/0x1f0 mm/page-writeback.c:2654
 __folio_migrate_mapping+0x3d0/0x2830 mm/migrate.c:624
 __migrate_folio mm/migrate.c:893 [inline]
 migrate_folio+0x10a/0x1a0 mm/migrate.c:920
 move_to_new_folio+0x1ed/0x460 mm/migrate.c:1135
 migrate_folio_move mm/migrate.c:1401 [inline]
 migrate_folios_move mm/migrate.c:1758 [inline]
 migrate_pages_batch+0x30b3/0x4f50 mm/migrate.c:2014
 migrate_pages_sync mm/migrate.c:2044 [inline]
 migrate_pages+0x1b9f/0x2920 mm/migrate.c:2153
 migrate_to_node mm/mempolicy.c:1322 [inline]
 do_migrate_pages+0x656/0x8b0 mm/mempolicy.c:1421
 kernel_migrate_pages mm/mempolicy.c:1935 [inline]
 __do_sys_migrate_pages mm/mempolicy.c:1953 [inline]
 __se_sys_migrate_pages+0xaa2/0xc90 mm/mempolicy.c:1949
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f600f39f559
Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f6010302028 EFLAGS: 00000246 ORIG_RAX: 0000000000000100
RAX: ffffffffffffffda RBX: 00007f600f625fa0 RCX: 00007f600f39f559
RDX: 0000200000000040 RSI: 0000000000000009 RDI: 0000000000000000
RBP: 00007f600f436b20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000200000000380 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f600f626038 R14: 00007f600f625fa0 R15: 00007ffcbfa14028
 </TASK>

Allocated by task 5645:
 kasan_save_stack mm/kasan/common.c:57 [inline]
 kasan_save_track+0x3e/0x80 mm/kasan/common.c:78
 unpoison_slab_object mm/kasan/common.c:351 [inline]
 __kasan_slab_alloc+0x6c/0x80 mm/kasan/common.c:377
 kasan_slab_alloc include/linux/kasan.h:253 [inline]
 slab_post_alloc_hook mm/slub.c:4683 [inline]
 slab_alloc_node mm/slub.c:4996 [inline]
 kmem_cache_alloc_noprof+0x2b9/0x600 mm/slub.c:5010
 anon_vma_alloc mm/rmap.c:94 [inline]
 __anon_vma_prepare+0x10e/0x4d0 mm/rmap.c:201
 __vmf_anon_prepare mm/memory.c:3961 [inline]
 vmf_anon_prepare mm/internal.h:572 [inline]
 do_anonymous_page mm/memory.c:5478 [inline]
 do_pte_missing+0x2f18/0x34b0 mm/memory.c:4698
 handle_pte_fault mm/memory.c:6558 [inline]
 __handle_mm_fault mm/memory.c:6707 [inline]
 handle_mm_fault+0x1a53/0x3750 mm/memory.c:6876
 do_user_addr_fault+0xa4f/0x1340 arch/x86/mm/fault.c:1343
 handle_page_fault arch/x86/mm/fault.c:1483 [inline]
 exc_page_fault+0x6a/0xc0 arch/x86/mm/fault.c:1536
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:595

Freed by task 5646:
 kasan_save_stack mm/kasan/common.c:57 [inline]
 kasan_save_track+0x3e/0x80 mm/kasan/common.c:78
 kasan_save_free_info+0x40/0x50 mm/kasan/generic.c:584
 poison_slab_object mm/kasan/common.c:264 [inline]
 __kasan_slab_free+0x5c/0x80 mm/kasan/common.c:296
 kasan_slab_free include/linux/kasan.h:235 [inline]
 slab_free_hook mm/slub.c:2748 [inline]
 slab_free_after_rcu_debug+0x12a/0x220 mm/slub.c:6571
 rcu_do_batch kernel/rcu/tree.c:2650 [inline]
 rcu_core+0x926/0x1260 kernel/rcu/tree.c:2919
 handle_softirqs+0x226/0x860 kernel/softirq.c:645
 __do_softirq kernel/softirq.c:679 [inline]
 invoke_softirq kernel/softirq.c:519 [inline]
 __irq_exit_rcu+0xcb/0x220 kernel/softirq.c:758
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:775
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1062 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1062
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:674

Last potentially related work creation:
 kasan_save_stack+0x3e/0x60 mm/kasan/common.c:57
 kasan_record_aux_stack+0xbd/0xd0 mm/kasan/generic.c:556
 slab_free_hook mm/slub.c:2709 [inline]
 slab_free mm/slub.c:6509 [inline]
 kmem_cache_free+0x45e/0x650 mm/slub.c:6636
 anon_vma_free mm/rmap.c:137 [inline]
 __put_anon_vma+0x12b/0x2d0 mm/rmap.c:2974
 put_anon_vma mm/internal.h:293 [inline]
 unlink_anon_vmas+0x544/0x720 mm/rmap.c:540
 free_pgtables+0x663/0xb70 mm/memory.c:401
 unmap_region+0x29d/0x330 mm/vma.c:534
 vms_clear_ptes mm/vma.c:1383 [inline]
 vms_complete_munmap_vmas+0x427/0xbe0 mm/vma.c:1425
 do_vmi_align_munmap+0x3b7/0x4b0 mm/vma.c:1684
 do_vmi_munmap+0x252/0x2d0 mm/vma.c:1732
 __vm_munmap+0x241/0x3e0 mm/vma.c:3415
 __do_sys_munmap mm/mmap.c:1094 [inline]
 __se_sys_munmap mm/mmap.c:1091 [inline]
 __x64_sys_munmap+0x60/0x70 mm/mmap.c:1091
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

The buggy address belongs to the object at ffff888111a3d528
 which belongs to the cache anon_vma of size 200
The buggy address is located 17 bytes inside of
 freed 200-byte region [ffff888111a3d528, ffff888111a3d5f0)

The buggy address belongs to the physical page:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0xffff888111a3d528 pfn:0x111a3d
memcg:ffff888111a3df79
flags: 0x17ff00000000200(workingset|node=0|zone=2|lastcpupid=0x7ff)
page_type: f5(slab)
raw: 017ff00000000200 ffff888101688140 ffff8881006957c8 ffffea00047d6bd0
raw: ffff888111a3d528 00000002000f0009 00000000f5000000 ffff888111a3df79
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0xd2cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP|__GFP_NOMEMALLOC), pid 5634, tgid 5634 (syz-executor), ts 59634551582
 set_page_owner include/linux/page_owner.h:33 [inline]
 post_alloc_hook+0x1f9/0x250 mm/page_alloc.c:1871
 prep_new_page mm/page_alloc.c:1879 [inline]
 get_page_from_freelist+0x2209/0x2280 mm/page_alloc.c:3943
 __alloc_frozen_pages_noprof+0x217/0x5a0 mm/page_alloc.c:5436
 alloc_slab_page mm/slub.c:3347 [inline]
 allocate_slab+0x7d/0x620 mm/slub.c:3462
 new_slab mm/slub.c:3513 [inline]
 refill_objects+0x2d5/0x350 mm/slub.c:7418
 refill_sheaf mm/slub.c:2885 [inline]
 __pcs_replace_empty_main+0x2c8/0x6c0 mm/slub.c:4774
 alloc_from_pcs mm/slub.c:4850 [inline]
 slab_alloc_node mm/slub.c:4984 [inline]
 kmem_cache_alloc_noprof+0x399/0x600 mm/slub.c:5010
 anon_vma_alloc mm/rmap.c:94 [inline]
 anon_vma_fork+0x88/0x540 mm/rmap.c:395
 dup_mmap+0x954/0x1dc0 mm/mmap.c:1781
 dup_mm kernel/fork.c:1543 [inline]
 copy_mm+0x115/0x480 kernel/fork.c:1595
 copy_process+0x1e79/0x4400 kernel/fork.c:2311
 kernel_clone+0x2d7/0x940 kernel/fork.c:2771
 __do_sys_clone kernel/fork.c:2913 [inline]
 __se_sys_clone kernel/fork.c:2897 [inline]
 __x64_sys_clone+0x1b6/0x230 kernel/fork.c:2897
 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline]
 do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
page last free pid 805 tgid 805 ts 42732995302 stack trace:
 reset_page_owner include/linux/page_owner.h:26 [inline]
 __free_pages_prepare mm/page_alloc.c:1418 [inline]
 __free_frozen_pages+0xc93/0xd90 mm/page_alloc.c:2962
 __slab_free+0x274/0x2c0 mm/slub.c:5823
 qlink_free mm/kasan/quarantine.c:163 [inline]
 qlist_free_all+0x99/0x100 mm/kasan/quarantine.c:179
 kasan_quarantine_reduce+0x148/0x160 mm/kasan/quarantine.c:286
 __kasan_slab_alloc+0x22/0x80 mm/kasan/common.c:361
 kasan_slab_alloc include/linux/kasan.h:253 [inline]
 slab_post_alloc_hook mm/slub.c:4683 [inline]
 slab_alloc_node mm/slub.c:4996 [inline]
 __kmalloc_cache_noprof+0x29f/0x600 mm/slub.c:5559
 _kmalloc_noprof include/linux/slab.h:991 [inline]
 _kzalloc_noprof include/linux/slab.h:1312 [inline]
 drm_atomic_commit_alloc+0xa9/0x100 drivers/gpu/drm/drm_atomic.c:258
 drm_atomic_helper_dirtyfb+0x129/0xfd0 drivers/gpu/drm/drm_damage_helper.c:128
 drm_fbdev_shmem_helper_fb_dirty+0x160/0x310 drivers/gpu/drm/drm_fbdev_shmem.c:117
 drm_fb_helper_fb_dirty drivers/gpu/drm/drm_fb_helper.c:339 [inline]
 drm_fb_helper_damage_work+0x84c/0xf20 drivers/gpu/drm/drm_fb_helper.c:365
 process_one_work kernel/workqueue.c:3396 [inline]
 process_scheduled_works+0xc3d/0x1630 kernel/workqueue.c:3479
 worker_thread+0xa47/0xfb0 kernel/workqueue.c:3560
 kthread+0x38b/0x480 kernel/kthread.c:436
 ret_from_fork+0x514/0xb70 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245

Memory state around the buggy address:
 ffff888111a3d400: fc fc fc fc 00 00 00 00 00 00 00 00 00 00 00 00
 ffff888111a3d480: 00 00 00 00 00 00 00 00 00 00 00 00 00 fc fc fc
>ffff888111a3d500: fc fc fc fc fc fa fb fb fb fb fb fb fb fb fb fb
                                        ^
 ffff888111a3d580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fc fc
 ffff888111a3d600: fc fc fc fc fc fc fa fb fb fb fb fb fb fb fb fb
==================================================================


***

If these findings have caused you to resend the series or submit a
separate fix, please add the following tag to your commit message:
  Tested-by: syzbot@syzkaller.appspotmail.com

---
This report is generated by a bot. It may contain errors.
syzbot ci engineers can be reached at syzkaller@googlegroups.com.

To test a fix for this bug, please reply with `#syz test`
(on a separate line) and attach the patch to the email.

Notes:
- The patch will be applied on top of the tested series (as an
  incremental fix).
- To test a new version of the whole series, please send it directly
  to syzbot@lists.linux.dev.
- Arguments like custom git repos and branches are not supported.