* [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker
@ 2026-07-13 10:22 Max Kellermann
2026-07-13 10:22 ` [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers Max Kellermann
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Max Kellermann @ 2026-07-13 10:22 UTC (permalink / raw)
To: tytso, jack, linux-ext4, linux-kernel; +Cc: Max Kellermann
We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem
that serves as the CacheFiles backing store for a Ceph (fscache)
client. The stall is always the journal commit thread spinning on
journal->j_list_lock:
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 106-....: (2099 ticks this GP) ... (t=2100 jiffies)
CPU: 106 Comm: jbd2/nvme1n1p1-
RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240
Call Trace:
jbd2_journal_write_metadata_buffer+0x1c0/0x310
jbd2_journal_commit_transaction+0x5e2/0x16e0
kjournald2+0xa1/0x220
kthread+0xe4/0x1d0
kjournald2 has already passed the "wait for outstanding handles"
barrier and is in the metadata write-out loop; it is simply unable to
acquire j_list_lock for >21s. The lock holder is the jbd2 checkpoint
shrinker.
Under memory pressure, the shrinker (jbd2_journal_shrink_scan ->
jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list)
walks a transaction's checkpoint list under j_list_lock. On this
workload, the lists are large and dominated by busy buffers (dirty /
under writeback / attached to the running transaction), and
journal_shrink_one_cp_list() can therefore hold j_list_lock for a time
proportional to the whole list length, for two reasons:
1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips
the need_resched() check, so the scan does not yield even when a
reschedule is pending.
2. The scan budget (nr_to_scan) is decremented only for buffers that
are actually freed, so busy buffers do not consume it and the loop
is not bounded by the shrinker's batch at all.
Both were introduced by commit b98dba273a0e ("jbd2: remove
journal_clean_one_cp_list()"), which folded
journal_clean_one_cp_list() into journal_shrink_one_cp_list() and
dropped the per-examined-buffer budget that the shrinker previously
had.
The result is a self-reinforcing collapse: shrinker instances across
many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the
journal fills, CacheFiles lookups block in ext4, fscache cookies get
stuck in LOOKING_UP, and netfs I/O times out.
Max Kellermann (2):
jbd2: check need_resched() when skipping busy checkpoint buffers
jbd2: bound shrinker scans by examined checkpoint buffers
fs/jbd2/checkpoint.c | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)
--
2.47.3
^ permalink raw reply [flat|nested] 8+ messages in thread* [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers 2026-07-13 10:22 [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Max Kellermann @ 2026-07-13 10:22 ` Max Kellermann 2026-07-13 12:16 ` Jan Kara 2026-07-13 10:22 ` [PATCH 2/2] jbd2: bound shrinker scans by examined " Max Kellermann 2026-07-13 12:53 ` [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Zhang Yi 2 siblings, 1 reply; 8+ messages in thread From: Max Kellermann @ 2026-07-13 10:22 UTC (permalink / raw) To: tytso, jack, linux-ext4, linux-kernel; +Cc: Max Kellermann, stable journal_shrink_one_cp_list() skips busy checkpoint buffers when called with JBD2_SHRINK_BUSY_SKIP. The continue statement on this path also skips the need_resched() check at the end of the loop body. Consequently, when a checkpoint list contains mostly busy buffers, the shrinker can walk the entire list while holding journal->j_list_lock, even when a reschedule has been requested. Large checkpoint lists under memory pressure can therefore cause long lock hold times and leave other CPUs spinning on j_list_lock, resulting in soft lockups or RCU stalls. Route the busy-buffer path through the need_resched() check so that the shrinker can release j_list_lock and reschedule promptly, restoring parity with the clean-buffer path, which already checks need_resched(). This does not change which checkpoint buffers are eligible for removal. Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()") Cc: stable@vger.kernel.org Signed-off-by: Max Kellermann <max.kellermann@ionos.com> --- fs/jbd2/checkpoint.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index 1508e2f54462..5266017565ac 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -389,7 +389,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, ret = jbd2_journal_try_remove_checkpoint(jh); if (ret < 0) { if (type == JBD2_SHRINK_BUSY_SKIP) - continue; + goto next; break; } } @@ -400,6 +400,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, break; } +next: if (need_resched()) break; } while (jh != last_jh); -- 2.47.3 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers 2026-07-13 10:22 ` [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers Max Kellermann @ 2026-07-13 12:16 ` Jan Kara 0 siblings, 0 replies; 8+ messages in thread From: Jan Kara @ 2026-07-13 12:16 UTC (permalink / raw) To: Max Kellermann; +Cc: tytso, jack, linux-ext4, linux-kernel, stable On Mon 13-07-26 12:22:28, Max Kellermann wrote: > journal_shrink_one_cp_list() skips busy checkpoint buffers when called > with JBD2_SHRINK_BUSY_SKIP. The continue statement on this path also > skips the need_resched() check at the end of the loop body. > > Consequently, when a checkpoint list contains mostly busy buffers, the > shrinker can walk the entire list while holding journal->j_list_lock, > even when a reschedule has been requested. Large checkpoint lists under > memory pressure can therefore cause long lock hold times and leave other > CPUs spinning on j_list_lock, resulting in soft lockups or RCU stalls. > > Route the busy-buffer path through the need_resched() check so that the > shrinker can release j_list_lock and reschedule promptly, restoring > parity with the clean-buffer path, which already checks need_resched(). > This does not change which checkpoint buffers are eligible for removal. > > Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()") > Cc: stable@vger.kernel.org > Signed-off-by: Max Kellermann <max.kellermann@ionos.com> Good catch! Thanks for fixing this. Feel free to add: Reviewed-by: Jan Kara <jack@suse.cz> Honza > --- > fs/jbd2/checkpoint.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c > index 1508e2f54462..5266017565ac 100644 > --- a/fs/jbd2/checkpoint.c > +++ b/fs/jbd2/checkpoint.c > @@ -389,7 +389,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, > ret = jbd2_journal_try_remove_checkpoint(jh); > if (ret < 0) { > if (type == JBD2_SHRINK_BUSY_SKIP) > - continue; > + goto next; > break; > } > } > @@ -400,6 +400,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, > break; > } > > +next: > if (need_resched()) > break; > } while (jh != last_jh); > -- > 2.47.3 > -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 2/2] jbd2: bound shrinker scans by examined checkpoint buffers 2026-07-13 10:22 [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Max Kellermann 2026-07-13 10:22 ` [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers Max Kellermann @ 2026-07-13 10:22 ` Max Kellermann 2026-07-13 12:24 ` Jan Kara 2026-07-13 12:53 ` [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Zhang Yi 2 siblings, 1 reply; 8+ messages in thread From: Max Kellermann @ 2026-07-13 10:22 UTC (permalink / raw) To: tytso, jack, linux-ext4, linux-kernel; +Cc: Max Kellermann, stable The jbd2 shrinker currently accounts only checkpoint buffers that it successfully releases against nr_to_scan. Busy buffers therefore do not consume the scan budget. If a checkpoint transaction contains mostly busy buffers, the shrinker can scan its entire checkpoint list while holding journal->j_list_lock. Large checkpoint lists can result in excessive lock hold times and leave other CPUs spinning on j_list_lock, causing soft lockups or RCU stalls. Pass nr_to_scan into journal_shrink_one_cp_list() and decrement it for every buffer examined, including busy buffers. Pass NULL from checkpoint cleanup paths so their existing full-list behavior is preserved. This restores the scan-budget semantics that existed before journal_shrink_one_cp_list() was changed to always scan a complete checkpoint list. Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()") Cc: stable@vger.kernel.org Signed-off-by: Max Kellermann <max.kellermann@ionos.com> --- fs/jbd2/checkpoint.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index 5266017565ac..513273712010 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -358,15 +358,16 @@ int jbd2_cleanup_journal_tail(journal_t *journal) /* * journal_shrink_one_cp_list * - * Find all the written-back checkpoint buffers in the given list - * and try to release them. If the whole transaction is released, set - * the 'released' parameter. Return the number of released checkpointed - * buffers. + * Find written-back checkpoint buffers in the given list and try to release + * them. If 'nr_to_scan' is set, scan at most that many buffers. If the whole + * transaction is released, set the 'released' parameter. Return the number of + * released checkpointed buffers. * * Called with j_list_lock held. */ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, enum jbd2_shrink_type type, + unsigned long *nr_to_scan, bool *released) { struct journal_head *last_jh; @@ -375,13 +376,15 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, int ret; *released = false; - if (!jh) + if (!jh || (nr_to_scan && !*nr_to_scan)) return 0; last_jh = jh->b_cpprev; do { jh = next_jh; next_jh = jh->b_cpnext; + if (nr_to_scan) + (*nr_to_scan)--; if (type == JBD2_SHRINK_DESTROY) { ret = __jbd2_journal_remove_checkpoint(jh); @@ -403,7 +406,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, next: if (need_resched()) break; - } while (jh != last_jh); + } while (jh != last_jh && (!nr_to_scan || *nr_to_scan)); return nr_freed; } @@ -425,7 +428,6 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, tid_t first_tid = 0, last_tid = 0, next_tid = 0; tid_t tid = 0; unsigned long nr_freed = 0; - unsigned long freed; bool first_set = false; again: @@ -458,10 +460,9 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, next_transaction = transaction->t_cpnext; tid = transaction->t_tid; - freed = journal_shrink_one_cp_list(transaction->t_checkpoint_list, - JBD2_SHRINK_BUSY_SKIP, &released); - nr_freed += freed; - (*nr_to_scan) -= min(*nr_to_scan, freed); + nr_freed += journal_shrink_one_cp_list(transaction->t_checkpoint_list, + JBD2_SHRINK_BUSY_SKIP, + nr_to_scan, &released); if (*nr_to_scan == 0) break; if (need_resched() || spin_needbreak(&journal->j_list_lock)) @@ -517,7 +518,7 @@ void __jbd2_journal_clean_checkpoint_list(journal_t *journal, transaction = next_transaction; next_transaction = transaction->t_cpnext; journal_shrink_one_cp_list(transaction->t_checkpoint_list, - type, &released); + type, NULL, &released); /* * This function only frees up some memory if possible so we * dont have an obligation to finish processing. Bail out if -- 2.47.3 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] jbd2: bound shrinker scans by examined checkpoint buffers 2026-07-13 10:22 ` [PATCH 2/2] jbd2: bound shrinker scans by examined " Max Kellermann @ 2026-07-13 12:24 ` Jan Kara 0 siblings, 0 replies; 8+ messages in thread From: Jan Kara @ 2026-07-13 12:24 UTC (permalink / raw) To: Max Kellermann; +Cc: tytso, jack, linux-ext4, linux-kernel, stable On Mon 13-07-26 12:22:29, Max Kellermann wrote: > The jbd2 shrinker currently accounts only checkpoint buffers that it > successfully releases against nr_to_scan. Busy buffers therefore do not > consume the scan budget. > > If a checkpoint transaction contains mostly busy buffers, the shrinker > can scan its entire checkpoint list while holding journal->j_list_lock. > Large checkpoint lists can result in excessive lock hold times and leave > other CPUs spinning on j_list_lock, causing soft lockups or RCU stalls. > > Pass nr_to_scan into journal_shrink_one_cp_list() and decrement it for > every buffer examined, including busy buffers. Pass NULL from checkpoint > cleanup paths so their existing full-list behavior is preserved. > > This restores the scan-budget semantics that existed before > journal_shrink_one_cp_list() was changed to always scan a complete > checkpoint list. > > Fixes: b98dba273a0e ("jbd2: remove journal_clean_one_cp_list()") > Cc: stable@vger.kernel.org > Signed-off-by: Max Kellermann <max.kellermann@ionos.com> Looks good. Feel free to add: Reviewed-by: Jan Kara <jack@suse.cz> Honza > --- > fs/jbd2/checkpoint.c | 25 +++++++++++++------------ > 1 file changed, 13 insertions(+), 12 deletions(-) > > diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c > index 5266017565ac..513273712010 100644 > --- a/fs/jbd2/checkpoint.c > +++ b/fs/jbd2/checkpoint.c > @@ -358,15 +358,16 @@ int jbd2_cleanup_journal_tail(journal_t *journal) > /* > * journal_shrink_one_cp_list > * > - * Find all the written-back checkpoint buffers in the given list > - * and try to release them. If the whole transaction is released, set > - * the 'released' parameter. Return the number of released checkpointed > - * buffers. > + * Find written-back checkpoint buffers in the given list and try to release > + * them. If 'nr_to_scan' is set, scan at most that many buffers. If the whole > + * transaction is released, set the 'released' parameter. Return the number of > + * released checkpointed buffers. > * > * Called with j_list_lock held. > */ > static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, > enum jbd2_shrink_type type, > + unsigned long *nr_to_scan, > bool *released) > { > struct journal_head *last_jh; > @@ -375,13 +376,15 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, > int ret; > > *released = false; > - if (!jh) > + if (!jh || (nr_to_scan && !*nr_to_scan)) > return 0; > > last_jh = jh->b_cpprev; > do { > jh = next_jh; > next_jh = jh->b_cpnext; > + if (nr_to_scan) > + (*nr_to_scan)--; > > if (type == JBD2_SHRINK_DESTROY) { > ret = __jbd2_journal_remove_checkpoint(jh); > @@ -403,7 +406,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, > next: > if (need_resched()) > break; > - } while (jh != last_jh); > + } while (jh != last_jh && (!nr_to_scan || *nr_to_scan)); > > return nr_freed; > } > @@ -425,7 +428,6 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, > tid_t first_tid = 0, last_tid = 0, next_tid = 0; > tid_t tid = 0; > unsigned long nr_freed = 0; > - unsigned long freed; > bool first_set = false; > > again: > @@ -458,10 +460,9 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, > next_transaction = transaction->t_cpnext; > tid = transaction->t_tid; > > - freed = journal_shrink_one_cp_list(transaction->t_checkpoint_list, > - JBD2_SHRINK_BUSY_SKIP, &released); > - nr_freed += freed; > - (*nr_to_scan) -= min(*nr_to_scan, freed); > + nr_freed += journal_shrink_one_cp_list(transaction->t_checkpoint_list, > + JBD2_SHRINK_BUSY_SKIP, > + nr_to_scan, &released); > if (*nr_to_scan == 0) > break; > if (need_resched() || spin_needbreak(&journal->j_list_lock)) > @@ -517,7 +518,7 @@ void __jbd2_journal_clean_checkpoint_list(journal_t *journal, > transaction = next_transaction; > next_transaction = transaction->t_cpnext; > journal_shrink_one_cp_list(transaction->t_checkpoint_list, > - type, &released); > + type, NULL, &released); > /* > * This function only frees up some memory if possible so we > * dont have an obligation to finish processing. Bail out if > -- > 2.47.3 > -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker 2026-07-13 10:22 [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Max Kellermann 2026-07-13 10:22 ` [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers Max Kellermann 2026-07-13 10:22 ` [PATCH 2/2] jbd2: bound shrinker scans by examined " Max Kellermann @ 2026-07-13 12:53 ` Zhang Yi 2026-07-13 15:03 ` Max Kellermann 2 siblings, 1 reply; 8+ messages in thread From: Zhang Yi @ 2026-07-13 12:53 UTC (permalink / raw) To: Max Kellermann, tytso, jack, linux-ext4, linux-kernel On 7/13/2026 6:22 PM, Max Kellermann wrote: > We hit recurring RCU stalls / soft lockups on a busy ext4 filesystem > that serves as the CacheFiles backing store for a Ceph (fscache) > client. The stall is always the journal commit thread spinning on > journal->j_list_lock: > > rcu: INFO: rcu_sched self-detected stall on CPU > rcu: 106-....: (2099 ticks this GP) ... (t=2100 jiffies) > CPU: 106 Comm: jbd2/nvme1n1p1- > RIP: 0010:queued_spin_lock_slowpath+0x20a/0x240 > Call Trace: > jbd2_journal_write_metadata_buffer+0x1c0/0x310 > jbd2_journal_commit_transaction+0x5e2/0x16e0 > kjournald2+0xa1/0x220 > kthread+0xe4/0x1d0 Hi, Max! Thanks for the report! I'm a bit curious about which kernel version and CONFIG you encountered this issue on. IIRC, after commits [1] and [2], the spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should always be effective (unless the hardware does not support it, i.e., ARCH_NO_PREEMPT). Moreover, the number of buffer_heads in the checkpoint list of a single transaction should not be large enough to trigger such soft lockups. So I suspect this issue should be unlikely to occur on the latest kernel releases. [1] 7dadeaa6e851 ("sched: Further restrict the preemption modes") [2] 7c70cb94d29c ("sched: Add Lazy preemption model") Thanks, Yi. > > kjournald2 has already passed the "wait for outstanding handles" > barrier and is in the metadata write-out loop; it is simply unable to > acquire j_list_lock for >21s. The lock holder is the jbd2 checkpoint > shrinker. > > Under memory pressure, the shrinker (jbd2_journal_shrink_scan -> > jbd2_journal_shrink_checkpoint_list -> journal_shrink_one_cp_list) > walks a transaction's checkpoint list under j_list_lock. On this > workload, the lists are large and dominated by busy buffers (dirty / > under writeback / attached to the running transaction), and > journal_shrink_one_cp_list() can therefore hold j_list_lock for a time > proportional to the whole list length, for two reasons: > > 1. The JBD2_SHRINK_BUSY_SKIP path uses "continue", which also skips > the need_resched() check, so the scan does not yield even when a > reschedule is pending. > > 2. The scan budget (nr_to_scan) is decremented only for buffers that > are actually freed, so busy buffers do not consume it and the loop > is not bounded by the shrinker's batch at all. > > Both were introduced by commit b98dba273a0e ("jbd2: remove > journal_clean_one_cp_list()"), which folded > journal_clean_one_cp_list() into journal_shrink_one_cp_list() and > dropped the per-examined-buffer budget that the shrinker previously > had. > > The result is a self-reinforcing collapse: shrinker instances across > many CPUs hold/contend j_list_lock, kjournald2 cannot commit, the > journal fills, CacheFiles lookups block in ext4, fscache cookies get > stuck in LOOKING_UP, and netfs I/O times out. > > Max Kellermann (2): > jbd2: check need_resched() when skipping busy checkpoint buffers > jbd2: bound shrinker scans by examined checkpoint buffers > > fs/jbd2/checkpoint.c | 28 +++++++++++++++------------- > 1 file changed, 15 insertions(+), 13 deletions(-) > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker 2026-07-13 12:53 ` [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Zhang Yi @ 2026-07-13 15:03 ` Max Kellermann 2026-07-14 3:41 ` Zhang Yi 0 siblings, 1 reply; 8+ messages in thread From: Max Kellermann @ 2026-07-13 15:03 UTC (permalink / raw) To: Zhang Yi; +Cc: tytso, jack, linux-ext4, linux-kernel On Mon, Jul 13, 2026 at 2:53 PM Zhang Yi <yi.zhang@huaweicloud.com> wrote: > I'm a bit curious about which kernel version and CONFIG you > encountered this issue on. IIRC, after commits [1] and [2], the > spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should > always be effective (unless the hardware does not support it, i.e., > ARCH_NO_PREEMPT). We run 6.18.38, and preemption is disabled. Therefore, spin_needbreak() is a no-op. (We used to always run the latest stable, but got burned by way too many critical Ceph/netfs regressions and settled on the LTS.) But even with preemption enabled, this would likely be a problem, though a much smaller one. > Moreover, the number of buffer_heads in the > checkpoint list of a single transaction should not be large enough > to trigger such soft lockups. So I suspect this issue should be > unlikely to occur on the latest kernel releases. Unfortunately, I don't have any numbers for you, because this problem occurs only every other week, but we have been haunted by this for a long time. The setup is a 7 TB ext4 partition dedicated to fscache/Ceph. It's always full, and culling is running quite often. For culling, we don't use cachefilesd but our own reimplementation (https://github.com/CM4all/cash) because cachefilesd scales quadratically and never makes any progress while burning CPU cycles and I/O forever; ours is massively parallel thanks to io_uring. That puts a lot of pressure on the ext4 filesystem, and a cgroup memory.high setting means it's always under memory pressure (or else the kernel will accumulate many gigabytes of RAM usage very quickly during the filesystem scan that is necessary for culling). Which means the shrinker is running all the time. Plus the usual kernel-initiated I/O on that partition. Every time this problem occurred, fscache culling was active. -- Max Kellermann Principal Architect Hosting Technology cm4all | Im Mediapark 6a | 50670 Köln | Germany General information about the company can be found here: https://www.cm4all.com/impressum A member of the IONOS Group ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker 2026-07-13 15:03 ` Max Kellermann @ 2026-07-14 3:41 ` Zhang Yi 0 siblings, 0 replies; 8+ messages in thread From: Zhang Yi @ 2026-07-14 3:41 UTC (permalink / raw) To: Max Kellermann; +Cc: tytso, jack, linux-ext4, linux-kernel On 7/13/2026 11:03 PM, Max Kellermann wrote: > On Mon, Jul 13, 2026 at 2:53 PM Zhang Yi <yi.zhang@huaweicloud.com> wrote: >> I'm a bit curious about which kernel version and CONFIG you >> encountered this issue on. IIRC, after commits [1] and [2], the >> spin_needbreak() in jbd2_journal_shrink_checkpoint_list() should >> always be effective (unless the hardware does not support it, i.e., >> ARCH_NO_PREEMPT). > > We run 6.18.38, and preemption is disabled. Therefore, > spin_needbreak() is a no-op. > (We used to always run the latest stable, but got burned by way too > many critical Ceph/netfs regressions and settled on the LTS.) > > But even with preemption enabled, this would likely be a problem, > though a much smaller one. > >> Moreover, the number of buffer_heads in the >> checkpoint list of a single transaction should not be large enough >> to trigger such soft lockups. So I suspect this issue should be >> unlikely to occur on the latest kernel releases. > > Unfortunately, I don't have any numbers for you, because this problem > occurs only every other week, but we have been haunted by this for a > long time. > > The setup is a 7 TB ext4 partition dedicated to fscache/Ceph. It's > always full, and culling is running quite often. > > For culling, we don't use cachefilesd but our own reimplementation > (https://github.com/CM4all/cash) because cachefilesd scales > quadratically and never makes any progress while burning CPU cycles > and I/O forever; ours is massively parallel thanks to io_uring. That > puts a lot of pressure on the ext4 filesystem, and a cgroup > memory.high setting means it's always under memory pressure (or else > the kernel will accumulate many gigabytes of RAM usage very quickly > during the filesystem scan that is necessary for culling). Which means > the shrinker is running all the time. Plus the usual kernel-initiated > I/O on that partition. > Every time this problem occurred, fscache culling was active. > Thank you for the details! We recently encountered the same issue on older kernels (up to v6.6) during extreme memory pressure stress testing. Interestingly, we were unable to reproduce it on the latest v7.1 kernel. After investigation, we identified that commit 7dadeaa6e851 ("sched: Further restrict the preemption modes") effectively prevents disabling preemption entirely, which makes spin_needbreak() always work in practice. So I was previously curious whether you discovered this issue in the latest kernel. However, let's come back to the point. I agree this is a real issue that deserves a proper fix, independent of the preemption model changes. Thanks, Yi. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-07-14 3:42 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-07-13 10:22 [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Max Kellermann 2026-07-13 10:22 ` [PATCH 1/2] jbd2: check need_resched() when skipping busy checkpoint buffers Max Kellermann 2026-07-13 12:16 ` Jan Kara 2026-07-13 10:22 ` [PATCH 2/2] jbd2: bound shrinker scans by examined " Max Kellermann 2026-07-13 12:24 ` Jan Kara 2026-07-13 12:53 ` [PATCH 0/2] jbd2: bound j_list_lock hold time in the checkpoint shrinker Zhang Yi 2026-07-13 15:03 ` Max Kellermann 2026-07-14 3:41 ` Zhang Yi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox