From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk2-f9.google.com (mail-qk2-f9.google.com [74.125.230.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CFB155C315 for ; Fri, 11 Sep 2026 20:44:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789159459; cv=none; b=RGv/rKJuAhhvxc8V+wzBeQzTwwXtGnPDU5oVSC1fDyX9jSl+qPpCEiunTC9VlGO+lB5Q2lS0ppWKNOo3J3wK+e6Z+/m9eFqP1MxdxvuwgnD6outEOpSpwGrI5fPWTRHzRaui/FNGesRpNhPzE1Us8o1S3hkM5iMKVL9KJ4mUKyY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789159459; c=relaxed/simple; bh=9KRmBLpDfExqlzXDVHUVK59qu4C/6ghjdOd9zFQjQ0Y=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=O09tSmoguEfUL9NBBjfiexEpHc9UK9sr3jGPjnlGQZ4vd1d9ZwhLr/+bovgYBBa5tar/0z4NVyQL4RjNclnSMH21KYid9KxBBnAZWgoXHZJyHx1lbJgaF+VeT0Qv+dRXvLAnOEBA79CqNYZBk2Jds/MeczHAgkhfWynHKPkQJ4U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=qyoduPel; arc=none smtp.client-ip=74.125.230.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="qyoduPel" Received: by mail-qk2-f9.google.com with SMTP id d75a77b69052e-5309ccafbc8so1915401cf.0 for ; Fri, 11 Sep 2026 13:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1789159456; x=1789764256; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=FDi78tn+rNyw2f5XP4H8xqn6L7v0YajTpN+MIEXmNi8=; b=qyoduPel33nPtdq/ko3a8YtkDb3zIeUPwdUpVbazYDFTpundfnYW9Oxh4tzKauU9Ei BGisEXp6AQZvGgAYzf7RAEaofve4Af7emK0sCBGXYPsVCBVBzhe46sUKd7ufjzEUWOXJ U4ViDbp1OjP8TqGvyPewYd6vHWnI+IBEutNiacFeVtNmTq+KLDrabYp8n4p6pYJ7g8kt QVTf8aZNDRfCFaU0dYrLN36lPNdDgoEc+JH3O060cahbwXtByrZMeAF8raN/NZxLDS18 B9+N0GjWoqs/ZafGY7BYh/KPEbh7+ihPQTNKYfNO+sC9DsqLjJ9jvdq+Ex1ciaH5c64D JjjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789159456; x=1789764256; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=FDi78tn+rNyw2f5XP4H8xqn6L7v0YajTpN+MIEXmNi8=; b=ZBgHO0nEXCNZR/p8h7wrFgZRk3Mc2cmD5LUiiYyWRQAN87qESSN/DpPJId3rcatX8E lnECFa8W2bXjj47uGMwk+HhyqrN0vbj6TyKfdLAqMmYKqpcqngows3927AQ+s5bpg2Xc OBhmIExkznQcxfeSy3ZbINcJH4ZGy0CXBPaScSe7L6VZi4iu5AT7t30w1skg2V8YdEU5 JvZZa4yPNWmyGWcsaYKurBo4rNj5PgRE5XBDpQ2ngId56St106mXmZ+tqw8bUE/AL5vN OymgpcmCA0IEYQl3iaz0V5aWjl92wq5eeUjS7Z4ynwZXuDPaDTRm2HYq2qF+TkSsDPq4 RJQA== X-Forwarded-Encrypted: i=1; AKwUvBwoUxO5cy4Ed06GS+hAHqXMc4rjoAdAPNv9D0JDkruNboIzxLkmXzBIk2qY0FG3XyhcGjkZgJrKqGIrheOb@vger.kernel.org X-Gm-Message-State: AFuF++kqRTvtNEN3f3L6Ito5abFzPpR98VweQ9LU17984bUn8TiRrNnZ X9LaZXNk3Mx8x0k+LmDACAks2YOLEIlYE+pqXzpYJC6+cypgyjOfnIsR/XH1DZWWc48= X-Gm-Gg: AYBFou382DDTpuDSp3C6bbZrOLSdSWc1ZCHg1wK9TpFbN4K1cWCJTDaDbCKiZD57N4R qS7RiieyuvM5hkW4dsc99IXPM1YsJa9x/Geso2Qfb9LqlVVPd6n46VhZawuI39qwhGVr0qC+oaJ 4FbGqPrwaBLNqV52QAvvB6chd9qeE1ymzCkSvTkJIU4aB26IeqUbj+0Ew/xCCz9Bw0IPRbDOtR5 l1rvTZXbK3H0djXrj199lml/6XozfK0bXvC2zgEfyxxGJNihmSMu+cpF0cnQQ+sDwysKkPvCFyg gxP0Y1zNiCA9bckxWoEm6PHK61j9O4Ewj202n2+Uo9LeMhGCVBzOiFQOr2Z8pmLPFLscTY79biy zjoOkJzWHRwieIyXZf/RNJl7vxlzwUPc8+BDAkb1m6CigdyRwk+17vmGZI3MiSMlXShhPwdU/cF QpKtT7aFcMF1Ubkjj1H9GhzmKgXeUw2qrbIR085LStpBohr9lNSJGuS6rz/KttjCmpkg7k4w== X-Received: by 2002:ac8:5e52:0:b0:530:dc5e:27b8 with SMTP id d75a77b69052e-530dc5e2fb1mr37003331cf.27.1789159456019; Fri, 11 Sep 2026 13:44:16 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca511ff0sm28939081cf.31.2026.09.11.13.44.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 13:44:15 -0700 (PDT) Date: Fri, 11 Sep 2026 16:44:14 -0400 From: Johannes Weiner To: Salvatore Dipietro Cc: linux-kernel@vger.kernel.org, hch@infradead.org, abuehaze@amazon.com, akpm@linux-foundation.org, alisaidi@amazon.com, blakgeof@amazon.com, brauner@kernel.org, dgc@kernel.org, dipietro.salvatore@gmail.com, djwong@kernel.org, jackmanb@google.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-xfs@vger.kernel.org, mhocko@suse.com, ritesh.list@gmail.com, rvvandan@amazon.com, stable@vger.kernel.org, surenb@google.com, vbabka@kernel.org, willy@infradead.org, ziy@nvidia.com, Vlastimil Babka , David Hildenbrand , Christoph Hellwig , Brendan Jackman Subject: Re: [PATCH v5] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations Message-ID: References: <20260911142102.2294202-1-dipiets@amazon.it> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260911142102.2294202-1-dipiets@amazon.it> On Fri, Sep 11, 2026 at 02:21:02PM +0000, Salvatore Dipietro wrote: > Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") > introduced high-order folio allocations in the iomap buffered write > path. When memory is fragmented, each failed costly-order allocation > enters __alloc_pages_slowpath() which runs direct compaction and > drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL > pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system. > > The root issue is that direct compaction is too expensive for hot > allocation paths that have fallbacks to smaller allocations. > __filemap_get_folio_mpol() already marks higher-order allocations with > __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle > failure. However, the page allocator still attempts full direct > compaction for costly orders with __GFP_NORETRY, which is unnecessarily > aggressive when the caller will simply retry at a lower order. > > For costly-order allocations with __GFP_NORETRY, suppress direct reclaim > for the whole slowpath by computing can_direct_reclaim (and hence > can_compact) as false. The !can_direct_reclaim check near the top of > the slowpath then short-circuits to nopage: no direct reclaim, no direct > compaction and no drain_all_pages() IPI across every CPU. kswapd (and > in turn kcompactd) is still woken further down for background > defragmentation, so compaction keeps working for long-term system health > while being removed from the latency-critical direct allocation path. > > Allocations that also request __GFP_THISNODE are exempted. That flag > pairing identifies the local-node-first THP attempt issued by > alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to > form transparent huge pages. __GFP_NOFAIL is exempted as well, so a > must-not-fail allocation is never made to fail. > > Test environment: > Hardware: AWS EC2 m8g.24xlarge (96 vCPU, arm64) > 12x 1TB IO2 32000 IOPS RAID0 XFS > OS: AL2023 > Kernel: v7.3-rc1 > Database: PostgreSQL 18.4 > Workload: pgbench simple-update, 1024 clients, 96 threads, 1200s > > Results (average of 3 runs, TPS): > Config Avg TPS % vs baseline > AL2023 stock 6.1 kernel (pre-5d8edfb900d5) 136,942 n/a > v7.3-rc1 baseline (no patch) 59,408 - > v7.3-rc1 + this patch 161,994 +172.7% > > The patch fully recovers the pre-5d8edfb900d5 performance, bringing > throughput back above the pre-regression level and well clear of the > ~59k baseline. The AL2023 6.1 row runs a kernel that predates commit > 5d8edfb900d5 ("iomap: Copy larger chunks from userspace"), so it is not > directly comparable, but it shows the pre-regression level and confirms > that the ~59k baseline is the anomaly and not the norm. > > > Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@amazon.it/T/#t [v1] > Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@amazon.it/T/#u [v2] > Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@amazon.it/T/#u [v3] > Link: https://lore.kernel.org/all/20260904115629.3993331-1-dipiets@amazon.it/T/#u [v4] > Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace") > Cc: stable@vger.kernel.org > Cc: Andrew Morton > Cc: Vlastimil Babka > Cc: David Hildenbrand > Cc: Michal Hocko > Cc: Johannes Weiner > Cc: Matthew Wilcox > Cc: Christoph Hellwig > Cc: Dave Chinner > Cc: Ritesh Harjani > Cc: Zi Yan > Cc: linux-mm@kvack.org > Cc: linux-fsdevel@vger.kernel.org > Cc: linux-xfs@vger.kernel.org > Signed-off-by: Salvatore Dipietro > --- > v5: Derive the decision into a local flag instead of mutating gfp_mask I commented on the v5 variant here: https://lore.kernel.org/all/aqQlUr-1h07G8JWM@cmpxchg.org/ I think v4 was better.