From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CFA24B0CB9 for ; Thu, 3 Sep 2026 13:46:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788443232; cv=none; b=X+i+4dKBKgLOfvsvGGdkOYpE1IpqbNdogoSmH+KkSr0qB5vGlc0nnCr07VCjCLs3GninzyWr5kHxPzwC7eEr22kzcw6LWoJqkvUgwb6pGxnbRRhrWVbjR9tROgbVlHbyVQsmRa02W9n0+dbGVhKDpQbW3yfihF/svojkm8NALlI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788443232; c=relaxed/simple; bh=CidmxBxdg63rrWehrM2qC4+Wy9qNuc6w+lnGi+tcVnE=; h=Subject:To:Cc:From:Date:Message-ID:MIME-Version:Content-Type; b=kUxRAEPpGViGQ9hN540P/VkoAo0sNhniXSf+t3E07KY5kd2fhDUHioq+DL8YN7zJAEwcgJ+QEJ9pKPdxJwJQBCmOq0Nh4PSmudo7myFA2mJsDpAGvjz18TkR3shqwi8LrJSsuWzeFBsCZrXTclT8IJVljXB9RbHV8LDP590Axxw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b=WpbuVAR5; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b="WpbuVAR5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E92121F00A3D; Thu, 3 Sep 2026 13:46:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1788443219; bh=X52jVvM02fpb8Y05peD5j2pjR2wdDCDLmDbBecJr4wA=; h=Subject:To:Cc:From:Date; b=WpbuVAR5way+syGfaoinpwvKWNBc1R4YiQTM7E8orZIT0AjNKm6fEA99YKfPHBl0A 6o4dJhAnxW4tDPfCiaamumLgUDV7aIhBDOsAQzjRy/fCMjvSjCxN+gaTQTVHj4OIEp e61n8JhEtYMgeLYv6MYMGe5ZIUIAZzv/XAvYNKuI= Subject: FAILED: patch "[PATCH] ceph: do not repeat ceph_trim_dentries() if no progress" failed to apply to 5.15-stable tree To: max.kellermann@ionos.com,amarkuze@redhat.com,idryomov@gmail.com Cc: From: Date: Thu, 03 Sep 2026 15:23:11 +0200 Message-ID: <2026090311-shiny-gills-15eb@gregkh> Precedence: bulk X-Mailing-List: stable@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=ANSI_X3.4-1968 Content-Transfer-Encoding: 8bit The patch below does not apply to the 5.15-stable tree. If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to . To reproduce the conflict and resubmit, you may use the following commands: git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-5.15.y git checkout FETCH_HEAD git cherry-pick -x e7d7aa7b730178278109c41fa1b17b06873065d5 # git commit -s git send-email --to '' --in-reply-to '2026090311-shiny-gills-15eb@gregkh' --subject-prefix 'PATCH 5.15.y' 'HEAD^..' Possible dependencies: thanks, greg k-h ------------------ original commit in Linus's tree ------------------ >From e7d7aa7b730178278109c41fa1b17b06873065d5 Mon Sep 17 00:00:00 2001 From: Max Kellermann Date: Tue, 7 Jul 2026 23:42:28 +0200 Subject: [PATCH] ceph: do not repeat ceph_trim_dentries() if no progress possible ceph_cap_reclaim_work() re-queues itself for as long as ceph_trim_dentries() returns -EAGAIN, which happens whenever a lease walk exhausts its `nr_to_scan` budget. This creates a busy loop that consumes CPU without making any progress when there is nothing to reclaim: with no cap pressure (`count==0`) and every scanned lease still valid, each pass runs the full scan budget down to zero and returns `-EAGAIN`, only to be queued again immediately. The dir-lease walk made this worse. When `expire_dir_lease` is `false` (i.e. we have no intention of reclaiming dir leases), __dir_lease_check() returned `TOUCH` for every valid lease. `TOUCH` moves the dentry to the tail of the list and resets `di->time` via __dentry_dir_lease_touch(), so a walk over N valid leases pointlessly rewrote the list, refreshed the timestamps (preventing them from ever aging out) and always drained `nr_to_scan`, guaranteeing the `-EAGAIN` requeue. Fix this in three steps: - Return `KEEP` instead of `TOUCH` when `expire_dir_lease` is `false`. If we are not going to reclaim the lease, leave it in place instead of churning the list and resetting its timestamp; the walk then terminates naturally (or via `STOP` at the first fresh lease). - Only return `-EAGAIN` from the first (dentry-lease) walk when something was actually freed. A full batch that frees nothing means retrying the same list immediately is futile; fall through to the dir-lease walk instead. - After both walks, bail out with success (0) when nothing was freed and there is no cap pressure (`count==0`). There is no reason to keep retrying when we are not over the cap limit and made no progress. Under real cap pressure (`count>0`) the reclaim path is unchanged and still retries via `-EAGAIN`. Without this patch, I saw 500 ceph_trim_dentries() calls per second on our web servers. This is very visible in `/proc/lock_stat` (5 minute capture): class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg &mdsc->dentry_list_lock: 126180 128218 0.04 8063.44 15986965.20 124.69 1573354 5296812 0.04 8291.28 74164526.48 14.00 ----------------------- &mdsc->dentry_list_lock 111736 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2631 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 3878 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 9973 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 ----------------------- &mdsc->dentry_list_lock 123621 [<0000000050597999>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 1822 [<000000007b11e319>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 2720 [<000000002f27cb6f>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 55 [<00000000c0022f62>] __ceph_dentry_lease_touch+0x5c/0xa8 With this patch: class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg &mdsc->dentry_list_lock: 1203 1215 0.16 408.88 33082.88 27.23 4320501 7357389 0.04 500.64 1961578.00 0.27 ----------------------- &mdsc->dentry_list_lock 1029 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 169 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 16 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8 &mdsc->dentry_list_lock 1 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 ----------------------- &mdsc->dentry_list_lock 158 [<000000002038c577>] __dentry_lease_unlist+0x50/0xa0 &mdsc->dentry_list_lock 858 [<000000003c9aea8a>] __ceph_dentry_dir_lease_touch+0x7c/0xa8 &mdsc->dentry_list_lock 182 [<00000000612fe15f>] __dentry_leases_walk+0x64/0x2c8 &mdsc->dentry_list_lock 17 [<00000000c991106d>] __ceph_dentry_lease_touch+0x5c/0xa8 __dentry_leases_walk() is almost gone. The total wait time is reduced by a factor of 483. That will give some latency gains to ceph_readdir(). Cc: stable@vger.kernel.org Fixes: 37c4efc1ddf9 ("ceph: periodically trim stale dentries") Signed-off-by: Max Kellermann Reviewed-by: Alex Markuze Signed-off-by: Ilya Dryomov diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index d5d8f935fb62..f4e0bf244fd2 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -1770,11 +1770,11 @@ static int __dir_lease_check(const struct dentry *dentry, if (ret > 0) { if (time_before(jiffies, di->time + lwc->dir_lease_ttl)) return STOP; + if (!lwc->expire_dir_lease) + return KEEP; /* Move dentry to tail of dir lease list if we don't want * to delete it. So dentries in the list are checked in a * round robin manner */ - if (!lwc->expire_dir_lease) - return TOUCH; if (dentry->d_lockref.count > 0 || (di->flags & CEPH_DENTRY_REFERENCED)) return TOUCH; @@ -1801,7 +1801,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.dir_lease = false; lwc.nr_to_scan = CEPH_CAPS_PER_RELEASE * 2; freed = __dentry_leases_walk(mdsc, &lwc); - if (!lwc.nr_to_scan) /* more invalid leases */ + if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */ return -EAGAIN; if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE) @@ -1811,6 +1811,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.expire_dir_lease = freed < count; lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ; freed +=__dentry_leases_walk(mdsc, &lwc); + if (freed == 0 && count == 0) + /* no progress possible currently, retry futile */ + return 0; + if (!lwc.nr_to_scan) /* more to check */ return -EAGAIN;