[PATCH 1/1] iomap: avoid compaction for costly folio order allocation

Salvatore Dipietro
2026-04-03T19:36:21Z

Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
introduced high-order folio allocations in the buffered write
path. When memory is fragmented, each failed allocation triggers
compaction and drain_all_pages() via __alloc_pages_slowpath(),
causing a 0.75x throughput drop on pgbench (simple-update) with
1024 clients on a 96-vCPU arm64 system.

Strip __GFP_DIRECT_RECLAIM from folio allocations in
iomap_get_folio() when the order exceeds PAGE_ALLOC_COSTLY_ORDER,
making them purely opportunistic.

Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
Cc: stable@vger.kernel.org
Signed-off-by: Salvatore Dipietro <dipiets@amazon.it>
---
fs/iomap/buffered-io.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index 92a831cf4bf1..cb843d54b4d9 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -715,6 +715,7 @@ EXPORT_SYMBOL_GPL(iomap_is_partially_uptodate);
struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
{
fgf_t fgp = FGP_WRITEBEGIN | FGP_NOFS;
+ gfp_t gfp;

if (iter->flags & IOMAP_NOWAIT)
fgp |= FGP_NOWAIT;
@@ -722,8 +723,20 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len)
fgp |= FGP_DONTCACHE;
fgp |= fgf_set_order(len);

+ gfp = mapping_gfp_mask(iter->inode->i_mapping);
+
+ /*
+ * If the folio order hint exceeds PAGE_ALLOC_COSTLY_ORDER,
+ * strip __GFP_DIRECT_RECLAIM to make the allocation purely
+ * opportunistic. This avoids compaction + drain_all_pages()
+ * in __alloc_pages_slowpath() that devastate throughput
+ * on large systems during buffered writes.
+ */
+ if (FGF_GET_ORDER(fgp) > PAGE_ALLOC_COSTLY_ORDER)
+ gfp &= ~__GFP_DIRECT_RECLAIM;
+
return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT,
- fgp, mapping_gfp_mask(iter->inode->i_mapping));
+ fgp, gfp);
}
EXPORT_SYMBOL_GPL(iomap_get_folio);

--
2.50.1 (Apple Git-155)

AMAZON DEVELOPMENT CENTER ITALY SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle Imprese di Milano Monza Brianza Lodi REA n. 2504859, Capitale Sociale: 10.000 EUR i.v., Cod. Fisc. e P.IVA 10100050961, Societa con Socio Unico