* FAILED: patch "[PATCH] ceph: do not repeat ceph_trim_dentries() if no progress" failed to apply to 5.15-stable tree
@ 2026-09-03 13:23 gregkh
2026-09-06 16:41 ` [PATCH 5.15.y] ceph: do not repeat ceph_trim_dentries() if no progress possible Sasha Levin
0 siblings, 1 reply; 2+ messages in thread
From: gregkh @ 2026-09-03 13:23 UTC (permalink / raw)
To: max.kellermann, amarkuze, idryomov; +Cc: stable
The patch below does not apply to the 5.15-stable tree.
If someone wants it applied there, or to any other stable or longterm
tree, then please email the backport, including the original git commit
id to <stable@vger.kernel.org>.
To reproduce the conflict and resubmit, you may use the following commands:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-5.15.y
git checkout FETCH_HEAD
git cherry-pick -x e7d7aa7b730178278109c41fa1b17b06873065d5
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<stable@vger.kernel.org>' --in-reply-to '2026090311-shiny-gills-15eb@gregkh' --subject-prefix 'PATCH 5.15.y' 'HEAD^..'
Possible dependencies:
thanks,
greg k-h
------------------ original commit in Linus's tree ------------------
From e7d7aa7b730178278109c41fa1b17b06873065d5 Mon Sep 17 00:00:00 2001
From: Max Kellermann <max.kellermann@ionos.com>
Date: Tue, 7 Jul 2026 23:42:28 +0200
Subject: [PATCH] ceph: do not repeat ceph_trim_dentries() if no progress
possible
ceph_cap_reclaim_work() re-queues itself for as long as
ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease
walk exhausts its `nr_to_scan` budget. This creates a busy loop that
consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (`count==0`) and every scanned lease
still valid, each pass runs the full scan budget down to zero and
returns `-EAGAIN`, only to be queued again immediately.
The dir-lease walk made this worse. When `expire_dir_lease` is
`false` (i.e. we have no intention of reclaiming dir leases),
__dir_lease_check() returned `TOUCH` for every valid lease. `TOUCH`
moves the dentry to the tail of the list and resets `di->time` via
__dentry_dir_lease_touch(), so a walk over N valid leases pointlessly
rewrote the list, refreshed the timestamps (preventing them from ever
aging out) and always drained `nr_to_scan`, guaranteeing the `-EAGAIN`
requeue.
Fix this in three steps:
- Return `KEEP` instead of `TOUCH` when `expire_dir_lease` is
`false`. If we are not going to reclaim the lease, leave it in
place instead of churning the list and resetting its timestamp; the
walk then terminates naturally (or via `STOP` at the first fresh
lease).
- Only return `-EAGAIN` from the first (dentry-lease) walk when something
was actually freed. A full batch that frees nothing means retrying
the same list immediately is futile; fall through to the dir-lease
walk instead.
- After both walks, bail out with success (0) when nothing was freed
and there is no cap pressure (`count==0`). There is no reason to
keep retrying when we are not over the cap limit and made no
progress.
Under real cap pressure (`count>0`) the reclaim path is unchanged and
still retries via `-EAGAIN`.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on
our web servers. This is very visible in `/proc/lock_stat` (5 minute
capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------
&mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
-----------------------
&mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------
&mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
-----------------------
&mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
__dentry_leases_walk() is almost gone. The total wait time is reduced
by a factor of 483. That will give some latency gains to
ceph_readdir().
Cc: stable@vger.kernel.org
Fixes: 37c4efc1ddf9 ("ceph: periodically trim stale dentries")
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Reviewed-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c
index d5d8f935fb62..f4e0bf244fd2 100644
--- a/fs/ceph/dir.c
+++ b/fs/ceph/dir.c
@@ -1770,11 +1770,11 @@ static int __dir_lease_check(const struct dentry *dentry,
if (ret > 0) {
if (time_before(jiffies, di->time + lwc->dir_lease_ttl))
return STOP;
+ if (!lwc->expire_dir_lease)
+ return KEEP;
/* Move dentry to tail of dir lease list if we don't want
* to delete it. So dentries in the list are checked in a
* round robin manner */
- if (!lwc->expire_dir_lease)
- return TOUCH;
if (dentry->d_lockref.count > 0 ||
(di->flags & CEPH_DENTRY_REFERENCED))
return TOUCH;
@@ -1801,7 +1801,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.dir_lease = false;
lwc.nr_to_scan = CEPH_CAPS_PER_RELEASE * 2;
freed = __dentry_leases_walk(mdsc, &lwc);
- if (!lwc.nr_to_scan) /* more invalid leases */
+ if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */
return -EAGAIN;
if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE)
@@ -1811,6 +1811,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.expire_dir_lease = freed < count;
lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ;
freed +=__dentry_leases_walk(mdsc, &lwc);
+ if (freed == 0 && count == 0)
+ /* no progress possible currently, retry futile */
+ return 0;
+
if (!lwc.nr_to_scan) /* more to check */
return -EAGAIN;
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [PATCH 5.15.y] ceph: do not repeat ceph_trim_dentries() if no progress possible
2026-09-03 13:23 FAILED: patch "[PATCH] ceph: do not repeat ceph_trim_dentries() if no progress" failed to apply to 5.15-stable tree gregkh
@ 2026-09-06 16:41 ` Sasha Levin
0 siblings, 0 replies; 2+ messages in thread
From: Sasha Levin @ 2026-09-06 16:41 UTC (permalink / raw)
To: stable; +Cc: Max Kellermann, Alex Markuze, Ilya Dryomov, Sasha Levin
From: Max Kellermann <max.kellermann@ionos.com>
[ Upstream commit e7d7aa7b730178278109c41fa1b17b06873065d5 ]
ceph_cap_reclaim_work() re-queues itself for as long as
ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease
walk exhausts its `nr_to_scan` budget. This creates a busy loop that
consumes CPU without making any progress when there is nothing to
reclaim: with no cap pressure (`count==0`) and every scanned lease
still valid, each pass runs the full scan budget down to zero and
returns `-EAGAIN`, only to be queued again immediately.
The dir-lease walk made this worse. When `expire_dir_lease` is
`false` (i.e. we have no intention of reclaiming dir leases),
__dir_lease_check() returned `TOUCH` for every valid lease. `TOUCH`
moves the dentry to the tail of the list and resets `di->time` via
__dentry_dir_lease_touch(), so a walk over N valid leases pointlessly
rewrote the list, refreshed the timestamps (preventing them from ever
aging out) and always drained `nr_to_scan`, guaranteeing the `-EAGAIN`
requeue.
Fix this in three steps:
- Return `KEEP` instead of `TOUCH` when `expire_dir_lease` is
`false`. If we are not going to reclaim the lease, leave it in
place instead of churning the list and resetting its timestamp; the
walk then terminates naturally (or via `STOP` at the first fresh
lease).
- Only return `-EAGAIN` from the first (dentry-lease) walk when something
was actually freed. A full batch that frees nothing means retrying
the same list immediately is futile; fall through to the dir-lease
walk instead.
- After both walks, bail out with success (0) when nothing was freed
and there is no cap pressure (`count==0`). There is no reason to
keep retrying when we are not over the cap limit and made no
progress.
Under real cap pressure (`count>0`) the reclaim path is unchanged and
still retries via `-EAGAIN`.
Without this patch, I saw 500 ceph_trim_dentries() calls per second on
our web servers. This is very visible in `/proc/lock_stat` (5 minute
capture):
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00
-----------------------
&mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
-----------------------
&mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8
With this patch:
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
&mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27
-----------------------
&mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
&mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
-----------------------
&mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0
&mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8
&mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8
&mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8
__dentry_leases_walk() is almost gone. The total wait time is reduced
by a factor of 483. That will give some latency gains to
ceph_readdir().
Cc: stable@vger.kernel.org
Fixes: 37c4efc1ddf9 ("ceph: periodically trim stale dentries")
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Reviewed-by: Alex Markuze <amarkuze@redhat.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
[ retained callback arguments for __dentry_leases_walk() required by Linux 6.6 ]
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
fs/ceph/dir.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c
index 74845a793aa36..ea3b2abb38152 100644
--- a/fs/ceph/dir.c
+++ b/fs/ceph/dir.c
@@ -1503,11 +1503,11 @@ static int __dir_lease_check(struct dentry *dentry, void *arg)
if (ret > 0) {
if (time_before(jiffies, di->time + lwc->dir_lease_ttl))
return STOP;
+ if (!lwc->expire_dir_lease)
+ return KEEP;
/* Move dentry to tail of dir lease list if we don't want
* to delete it. So dentries in the list are checked in a
* round robin manner */
- if (!lwc->expire_dir_lease)
- return TOUCH;
if (dentry->d_lockref.count > 0 ||
(di->flags & CEPH_DENTRY_REFERENCED))
return TOUCH;
@@ -1534,7 +1534,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.dir_lease = false;
lwc.nr_to_scan = CEPH_CAPS_PER_RELEASE * 2;
freed = __dentry_leases_walk(mdsc, &lwc, __dentry_lease_check);
- if (!lwc.nr_to_scan) /* more invalid leases */
+ if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */
return -EAGAIN;
if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE)
@@ -1544,6 +1544,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.expire_dir_lease = freed < count;
lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ;
freed +=__dentry_leases_walk(mdsc, &lwc, __dir_lease_check);
+ if (freed == 0 && count == 0)
+ /* no progress possible currently, retry futile */
+ return 0;
+
if (!lwc.nr_to_scan) /* more to check */
return -EAGAIN;
--
2.53.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-09-06 16:41 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-03 13:23 FAILED: patch "[PATCH] ceph: do not repeat ceph_trim_dentries() if no progress" failed to apply to 5.15-stable tree gregkh
2026-09-06 16:41 ` [PATCH 5.15.y] ceph: do not repeat ceph_trim_dentries() if no progress possible Sasha Levin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.