Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2026-10-09 15:34:02 [+0000], Naulls, Peter wrote:
> Long term kernel hacker, but very infrequent contributor. 
Hi Peter,

> We are using the RT kernel patch on x86 under kernel 6.6.78-rt51.
> Specifically, this is the kernel from meta-intel in Yocto Scarthgap in
> the intel-kernel-rt recipe. 
…
> sync (PID 52371)
>   btrfs_commit_transaction+0xc48
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   btrfs_sync_fs
>   sync_fs_one_sb
>   ksys_sync

This sort of sounds like the state is TRANS_STATE_SUPER_COMMITTED but is
not making progress after that. Are there any tasks involved that run at
realtime priority and would block other tasks from making progress?

…
> This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem. 
…
> # Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
> # livelock. wait_for_commit() can reselect a completed transaction that remains
> # on trans_list after its waiters are woken. Skip completed entries while still
> # waiting for older incomplete transactions. This is a diagnostic hypothesis,
> # not a confirmed fix for the observed lockup.
> # Review notes: the scan is O(n) under trans_lock. Completion state is published
> # locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
> # A multi-hour run did not reproduce the issue, but the lock owner and root cause
> # remain unknown.
> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index e15e1855fa0a..0f05fabe451c 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                                      const enum btrfs_trans_state min_state)
>  {
>         struct btrfs_fs_info *fs_info = commit->fs_info;
> +       struct btrfs_transaction *next;
>         u64 transid = commit->transid;
>         bool put = false;
> 
> @@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                  */
> 
>                 spin_lock(&fs_info->trans_lock);
> -               commit = list_first_entry_or_null(&fs_info->trans_list,
> -                                                 struct btrfs_transaction,
> -                                                 list);
> -               if (!commit || commit->transid > transid) {
> +               commit = NULL;
> +               list_for_each_entry(next, &fs_info->trans_list, list) {
> +                       if (next->transid > transid)
> +                               break;
> +                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
> +                               continue;
> +                       commit = next;
> +                       break;
> +               }
> +               if (!commit) {
>                         spin_unlock(&fs_info->trans_lock);
>                         break;
>                 }
> 
> 
> Note the O(N) time here. To that end, this is very unlikely to be the
> correct fix, but I hope it highlights the core problem. I ran this
> patch for several hours with no apparent problem (versus seconds of a
> lockup in previous testing). 
> 
> If there's a later kernel patch I've missed, or something else I ought
> to be looking at or test or need more info, please let me know. 

PREEMPT_RT shouldn't do much different than !PREEMPT_RT. Do you have
tasks with different priorities? If that is the case then it might be
that one thread is blocking another thread that should make progress.
Your changes iterates over TRANS_STATE_COMPLETED so it seems that it is
not cleaned up for some reason.

Could you try if you can reproduce this with v7.3-rc7? This one does not
require any additional patches.
I'm not sure if this report would help the btrfs folks to recognize if
this is a solved problem or where the issue might be.

Sebastian