[PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[Max Kellermann]

We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
that serves as the CacheFiles backing store for a Ceph (fscache)
client.  The stall is always the journal commit thread spinning on
journal->j_list_lock:

  rcu: INFO: rcu_sched self-detected stall on CPU
  rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
  CPU: 106 Comm: jbd2/nvme1n1p1-
  RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
  Call Trace:
   jbd2_journal_write_metadata_buffer+0x1c0/0x310
   jbd2_journal_commit_transaction+0x5e2/0x16e0
   kjournald2+0xa1/0x220
   kthread+0xe4/0x1d0

kjournald2 has already passed the "wait for outstanding handles"
barrier and is in the metadata write-out loop; it is simply unable to
acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
shrinker.

Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
walks a transaction's checkpoint list under j_list_lock.  On this
workload, the lists are large and dominated by busy buffers (dirty /
under writeback / attached to the running transaction), and
journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
proportional to the whole list length, for two reasons:

1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
   the need_resched() check, so the scan does not yield even when a
   reschedule is pending.

2. The scan budget (nr_to_scan) is decremented only for buffers that
   are actually freed, so busy buffers do not consume it and the loop
   is not bounded by the shrinker's batch at all.

Both were introduced by commit b98dba273a0e ("jbd2: remove
journal_clean_one_cp_list()"), which folded
journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
dropped the per-examined-buffer budget that the shrinker previously
had.

The result is a self-reinforcing collapse: shrinker instances across
many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
journal fills, CacheFiles lookups block in ext4, fscache cookies get
stuck in LOOKING_UP, and netfs I/O times out.

Max Kellermann (2):
  jbd2: check need_resched() when skipping busy checkpoint buffers
  jbd2: bound shrinker scans by examined checkpoint buffers

 fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

-- 
2.47.3

[PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers

[Max Kellermann]

journal_shrink_one_cp_list() skips busy checkpoint buffers when called
with JBD2_SHRINK_BUSY_SKIP.  The continue statement on this path also
skips the need_resched() check at the end of the loop body.

Consequently, when a checkpoint list contains mostly busy buffers, the
shrinker can walk the entire list while holding journal->j_list_lock,
even when a reschedule has been requested.  Large checkpoint lists under
memory pressure can therefore cause long lock hold times and leave other
CPUs spinning on j_list_lock, resulting in soft lockups or RCU stalls.

Route the busy-buffer path through the need_resched() check so that the
shrinker can release j_list_lock and reschedule promptly, restoring
parity with the clean-buffer path, which already checks need_resched().
This does not change which checkpoint buffers are eligible for removal.

Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()")
Cc: stable@vger.kernel.org
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 fs/jbd2/checkpoint.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 1508e2f54462..5266017565ac 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -389,7 +389,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
 			ret = jbd2_journal_try_remove_checkpoint(jh);
 			if (ret < 0) {
 				if (type == JBD2_SHRINK_BUSY_SKIP)
-					continue;
+					goto next;
 				break;
 			}
 		}
@@ -400,6 +400,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
 			break;
 		}
 
+next:
 		if (need_resched())
 			break;
 	} while (jh != last_jh);
-- 
2.47.3

[PATCH 2/2] jbd2: bound shrinker scans by examined checkpoint buffers

[Max Kellermann]

The jbd2 shrinker currently accounts only checkpoint buffers that it
successfully releases against nr_to_scan.  Busy buffers therefore do not
consume the scan budget.

If a checkpoint transaction contains mostly busy buffers, the shrinker
can scan its entire checkpoint list while holding journal->j_list_lock.
Large checkpoint lists can result in excessive lock hold times and leave
other CPUs spinning on j_list_lock, causing soft lockups or RCU stalls.

Pass nr_to_scan into journal_shrink_one_cp_list() and decrement it for
every buffer examined, including busy buffers.  Pass NULL from checkpoint
cleanup paths so their existing full-list behavior is preserved.

This restores the scan-budget semantics that existed before
journal_shrink_one_cp_list() was changed to always scan a complete
checkpoint list.

Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()")
Cc: stable@vger.kernel.org
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
 fs/jbd2/checkpoint.c | 25 +++++++++++++------------
 1 file changed, 13 insertions(+), 12 deletions(-)

diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 5266017565ac..513273712010 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -358,15 +358,16 @@ int jbd2_cleanup_journal_tail(journal_t *journal)
 /*
  * journal_shrink_one_cp_list
  *
- * Find all the written-back checkpoint buffers in the given list
- * and try to release them. If the whole transaction is released, set
- * the 'released' parameter. Return the number of released checkpointed
- * buffers.
+ * Find written-back checkpoint buffers in the given list and try to release
+ * them. If 'nr_to_scan' is set, scan at most that many buffers. If the whole
+ * transaction is released, set the 'released' parameter. Return the number of
+ * released checkpointed buffers.
  *
  * Called with j_list_lock held.
  */
 static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
 						enum jbd2_shrink_type type,
+						unsigned long *nr_to_scan,
 						bool *released)
 {
 	struct journal_head *last_jh;
@@ -375,13 +376,15 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
 	int ret;
 
 	*released = false;
-	if (!jh)
+	if (!jh || (nr_to_scan && !*nr_to_scan))
 		return 0;
 
 	last_jh = jh->b_cpprev;
 	do {
 		jh = next_jh;
 		next_jh = jh->b_cpnext;
+		if (nr_to_scan)
+			(*nr_to_scan)--;
 
 		if (type == JBD2_SHRINK_DESTROY) {
 			ret = __jbd2_journal_remove_checkpoint(jh);
@@ -403,7 +406,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
 next:
 		if (need_resched())
 			break;
-	} while (jh != last_jh);
+	} while (jh != last_jh && (!nr_to_scan || *nr_to_scan));
 
 	return nr_freed;
 }
@@ -425,7 +428,6 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal,
 	tid_t first_tid = 0, last_tid = 0, next_tid = 0;
 	tid_t tid = 0;
 	unsigned long nr_freed = 0;
-	unsigned long freed;
 	bool first_set = false;
 
 again:
@@ -458,10 +460,9 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal,
 		next_transaction = transaction->t_cpnext;
 		tid = transaction->t_tid;
 
-		freed = journal_shrink_one_cp_list(transaction->t_checkpoint_list,
-						   JBD2_SHRINK_BUSY_SKIP, &released);
-		nr_freed += freed;
-		(*nr_to_scan) -= min(*nr_to_scan, freed);
+		nr_freed += journal_shrink_one_cp_list(transaction->t_checkpoint_list,
+						       JBD2_SHRINK_BUSY_SKIP,
+						       nr_to_scan, &released);
 		if (*nr_to_scan == 0)
 			break;
 		if (need_resched() || spin_needbreak(&journal->j_list_lock))
@@ -517,7 +518,7 @@ void __jbd2_journal_clean_checkpoint_list(journal_t *journal,
 		transaction = next_transaction;
 		next_transaction = transaction->t_cpnext;
 		journal_shrink_one_cp_list(transaction->t_checkpoint_list,
-					   type, &released);
+					   type, NULL, &released);
 		/*
 		 * This function only frees up some memory if possible so we
 		 * dont have an obligation to finish processing. Bail out if
-- 
2.47.3

Re: [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers

[Jan Kara]

On Mon 13-07-26 12:22:28, Max Kellermann wrote:
> journal_shrink_one_cp_list() skips busy checkpoint buffers when called
> with JBD2_SHRINK_BUSY_SKIP.  The continue statement on this path also
> skips the need_resched() check at the end of the loop body.
> 
> Consequently, when a checkpoint list contains mostly busy buffers, the
> shrinker can walk the entire list while holding journal->j_list_lock,
> even when a reschedule has been requested.  Large checkpoint lists under
> memory pressure can therefore cause long lock hold times and leave other
> CPUs spinning on j_list_lock, resulting in soft lockups or RCU stalls.
> 
> Route the busy-buffer path through the need_resched() check so that the
> shrinker can release j_list_lock and reschedule promptly, restoring
> parity with the clean-buffer path, which already checks need_resched().
> This does not change which checkpoint buffers are eligible for removal.
> 
> Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

Good catch! Thanks for fixing this. Feel free to add:

Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  fs/jbd2/checkpoint.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
> index 1508e2f54462..5266017565ac 100644
> --- a/fs/jbd2/checkpoint.c
> +++ b/fs/jbd2/checkpoint.c
> @@ -389,7 +389,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
>  			ret = jbd2_journal_try_remove_checkpoint(jh);
>  			if (ret < 0) {
>  				if (type == JBD2_SHRINK_BUSY_SKIP)
> -					continue;
> +					goto next;
>  				break;
>  			}
>  		}
> @@ -400,6 +400,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
>  			break;
>  		}
>  
> +next:
>  		if (need_resched())
>  			break;
>  	} while (jh != last_jh);
> -- 
> 2.47.3
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH 2/2] jbd2: bound shrinker scans by examined checkpoint buffers

[Jan Kara]

On Mon 13-07-26 12:22:29, Max Kellermann wrote:
> The jbd2 shrinker currently accounts only checkpoint buffers that it
> successfully releases against nr_to_scan.  Busy buffers therefore do not
> consume the scan budget.
> 
> If a checkpoint transaction contains mostly busy buffers, the shrinker
> can scan its entire checkpoint list while holding journal->j_list_lock.
> Large checkpoint lists can result in excessive lock hold times and leave
> other CPUs spinning on j_list_lock, causing soft lockups or RCU stalls.
> 
> Pass nr_to_scan into journal_shrink_one_cp_list() and decrement it for
> every buffer examined, including busy buffers.  Pass NULL from checkpoint
> cleanup paths so their existing full-list behavior is preserved.
> 
> This restores the scan-budget semantics that existed before
> journal_shrink_one_cp_list() was changed to always scan a complete
> checkpoint list.
> 
> Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()")
> Cc: stable@vger.kernel.org
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

Looks good. Feel free to add:

Reviewed-by: Jan Kara <jack@suse.cz>

								Honza

> ---
>  fs/jbd2/checkpoint.c | 25 +++++++++++++------------
>  1 file changed, 13 insertions(+), 12 deletions(-)
> 
> diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
> index 5266017565ac..513273712010 100644
> --- a/fs/jbd2/checkpoint.c
> +++ b/fs/jbd2/checkpoint.c
> @@ -358,15 +358,16 @@ int jbd2_cleanup_journal_tail(journal_t *journal)
>  /*
>   * journal_shrink_one_cp_list
>   *
> - * Find all the written-back checkpoint buffers in the given list
> - * and try to release them. If the whole transaction is released, set
> - * the 'released' parameter. Return the number of released checkpointed
> - * buffers.
> + * Find written-back checkpoint buffers in the given list and try to release
> + * them. If 'nr_to_scan' is set, scan at most that many buffers. If the whole
> + * transaction is released, set the 'released' parameter. Return the number of
> + * released checkpointed buffers.
>   *
>   * Called with j_list_lock held.
>   */
>  static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
>  						enum jbd2_shrink_type type,
> +						unsigned long *nr_to_scan,
>  						bool *released)
>  {
>  	struct journal_head *last_jh;
> @@ -375,13 +376,15 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
>  	int ret;
>  
>  	*released = false;
> -	if (!jh)
> +	if (!jh || (nr_to_scan && !*nr_to_scan))
>  		return 0;
>  
>  	last_jh = jh->b_cpprev;
>  	do {
>  		jh = next_jh;
>  		next_jh = jh->b_cpnext;
> +		if (nr_to_scan)
> +			(*nr_to_scan)--;
>  
>  		if (type == JBD2_SHRINK_DESTROY) {
>  			ret = __jbd2_journal_remove_checkpoint(jh);
> @@ -403,7 +406,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh,
>  next:
>  		if (need_resched())
>  			break;
> -	} while (jh != last_jh);
> +	} while (jh != last_jh && (!nr_to_scan || *nr_to_scan));
>  
>  	return nr_freed;
>  }
> @@ -425,7 +428,6 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal,
>  	tid_t first_tid = 0, last_tid = 0, next_tid = 0;
>  	tid_t tid = 0;
>  	unsigned long nr_freed = 0;
> -	unsigned long freed;
>  	bool first_set = false;
>  
>  again:
> @@ -458,10 +460,9 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal,
>  		next_transaction = transaction->t_cpnext;
>  		tid = transaction->t_tid;
>  
> -		freed = journal_shrink_one_cp_list(transaction->t_checkpoint_list,
> -						   JBD2_SHRINK_BUSY_SKIP, &released);
> -		nr_freed += freed;
> -		(*nr_to_scan) -= min(*nr_to_scan, freed);
> +		nr_freed += journal_shrink_one_cp_list(transaction->t_checkpoint_list,
> +						       JBD2_SHRINK_BUSY_SKIP,
> +						       nr_to_scan, &released);
>  		if (*nr_to_scan == 0)
>  			break;
>  		if (need_resched() || spin_needbreak(&journal->j_list_lock))
> @@ -517,7 +518,7 @@ void __jbd2_journal_clean_checkpoint_list(journal_t *journal,
>  		transaction = next_transaction;
>  		next_transaction = transaction->t_cpnext;
>  		journal_shrink_one_cp_list(transaction->t_checkpoint_list,
> -					   type, &released);
> +					   type, NULL, &released);
>  		/*
>  		 * This function only frees up some memory if possible so we
>  		 * dont have an obligation to finish processing. Bail out if
> -- 
> 2.47.3
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[Zhang Yi]

On 7/13/2026 6:22 PM, Max Kellermann wrote:
> We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
> that serves as the CacheFiles backing store for a Ceph (fscache)
> client.  The stall is always the journal commit thread spinning on
> journal->j_list_lock:
> 
>   rcu: INFO: rcu_sched self-detected stall on CPU
>   rcu:   106-....: (2099 ticks this GP) ... (t=2100 jiffies)
>   CPU: 106 Comm: jbd2/nvme1n1p1-
>   RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
>   Call Trace:
>    jbd2_journal_write_metadata_buffer+0x1c0/0x310
>    jbd2_journal_commit_transaction+0x5e2/0x16e0
>    kjournald2+0xa1/0x220
>    kthread+0xe4/0x1d0

Hi, Max!

Thanks for the report!

I'm a bit curious about which kernel version and CONFIG you
encountered this issue on. IIRC, after commits [1] and [2], the
spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
always be effective (unless the hardware does not support it, i.e.,
ARCH_NO_PREEMPT). Moreover, the number of buffer_heads in the
checkpoint list of a single transaction should not be large enough
to trigger such soft lockups. So I suspect this issue should be
unlikely to occur on the latest kernel releases.

[1] 7dadeaa6e851 ("sched: Further restrict the preemption modes")
[2] 7c70cb94d29c ("sched: Add Lazy preemption model")

Thanks,
Yi.

> 
> kjournald2 has already passed the "wait for outstanding handles"
> barrier and is in the metadata write-out loop; it is simply unable to
> acquire j_list_lock for >21s.  The lock holder is the jbd2 checkpoint
> shrinker.
> 
> Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
> jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
> walks a transaction's checkpoint list under j_list_lock.  On this
> workload, the lists are large and dominated by busy buffers (dirty /
> under writeback / attached to the running transaction), and
> journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
> proportional to the whole list length, for two reasons:
> 
> 1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
>    the need_resched() check, so the scan does not yield even when a
>    reschedule is pending.
> 
> 2. The scan budget (nr_to_scan) is decremented only for buffers that
>    are actually freed, so busy buffers do not consume it and the loop
>    is not bounded by the shrinker's batch at all.
> 
> Both were introduced by commit b98dba273a0e ("jbd2: remove
> journal_clean_one_cp_list()"), which folded
> journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
> dropped the per-examined-buffer budget that the shrinker previously
> had.
> 
> The result is a self-reinforcing collapse: shrinker instances across
> many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
> journal fills, CacheFiles lookups block in ext4, fscache cookies get
> stuck in LOOKING_UP, and netfs I/O times out.
> 
> Max Kellermann (2):
>   jbd2: check need_resched() when skipping busy checkpoint buffers
>   jbd2: bound shrinker scans by examined checkpoint buffers
> 
>  fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
> 

Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker

[Max Kellermann]

On Mon, Jul 13, 2026 at 2:53 PM Zhang Yi <yi.zhang@huaweicloud.com> wrote:
> I'm a bit curious about which kernel version and CONFIG you
> encountered this issue on. IIRC, after commits [1] and [2], the
> spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should
> always be effective (unless the hardware does not support it, i.e.,
> ARCH_NO_PREEMPT).

We run 6.18.38, and preemption is disabled. Therefore,
spin_needbreak() is a no-op.
(We used to always run the latest stable, but got burned by way too
many critical Ceph/netfs regressions and settled on the LTS.)

But even with preemption enabled, this would likely be a problem,
though a much smaller one.

> Moreover, the number of buffer_heads in the
> checkpoint list of a single transaction should not be large enough
> to trigger such soft lockups. So I suspect this issue should be
> unlikely to occur on the latest kernel releases.

Unfortunately, I don't have any numbers for you, because this problem
occurs only every other week, but we have been haunted by this for a
long time.

The setup is a 7 TB ext4 partition dedicated to fscache/Ceph. It's
always full, and culling is running quite often.

For culling, we don't use cachefilesd but our own reimplementation
(https://github.com/CM4all/cash) because cachefilesd scales
quadratically and never makes any progress while burning CPU cycles
and I/O forever; ours is massively parallel thanks to io_uring. That
puts a lot of pressure on the ext4 filesystem, and a cgroup
memory.high setting means it's always under memory pressure (or else
the kernel will accumulate many gigabytes of RAM usage very quickly
during the filesystem scan that is necessary for culling). Which means
the shrinker is running all the time. Plus the usual kernel-initiated
I/O on that partition.
Every time this problem occurred, fscache culling was active.

-- 
Max Kellermann
Principal Architect
Hosting Technology

cm4all | Im Mediapark 6a | 50670 Köln | Germany
General information about the company can be found here:
https://www.cm4all.com/impressum
A member of the IONOS Group