Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Naulls, Peter]

 
Originally posted to RT mailing list, see post and replies there:

https://lore.kernel.org/linux-rt-users/PH0PR17MB4277167A37B20340B231443FBB922@PH0PR17MB4277.namprd17.prod.outlook.com/T/#u

________________________________________
From: Naulls, Peter <PeterNaulls@eaton.com>
Sent: Friday, October 9, 2026 11:34 AM
To: linux-rt-users@vger.kernel.org <linux-rt-users@vger.kernel.org>
Subject: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78
 
Long term kernel hacker, but very infrequent contributor. 

We are using the RT kernel patch on x86 under kernel 6.6.78-rt51. Specifically, this is the kernel from meta-intel in Yocto Scarthgap in the intel-kernel-rt recipe. 

It appears under some high load conditions, and especially triggered by calls to sync(), there is a live lock condition in the btrfs code. I used copilot for this analysis, with apologies due here where relevant. I'm familiar with kernel code, and I'm well aware how complex filesystem interactions can be. 

Here is the lock we observe:


sync (PID 52371)
  btrfs_commit_transaction+0xc48
    [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
  btrfs_sync_fs
  sync_fs_one_sb
  ksys_sync

ordered-I/O completion workers
  join_transaction+0x2b
    [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
  start_transaction
  btrfs_join_transaction
  btrfs_finish_one_ordered
  btrfs_work_helper

journal/fsync callers
  wait_current_trans+0x33
    [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
  start_transaction
  btrfs_start_transaction
  btrfs_sync_file
  vfs_fsync_range
  fsync

btrfs-transaction thread
  transaction_kthread+0x76
    [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
  kthread


The processes are then not killable, indicting a kernel thread problem. 

This was largely found because our userspace code calls sync() excessively (being separately addressed), although that's not the proximate cause. We ended up writing shell script which writes blocks of data to a brtfs partition over several threads and also calls sync and were able to repeat it within 10s of seconds. It is not repeatable in the non-RT kernel.

To add:

I have been extensively through the later kernels in 6.6, and later series and have not found any direct fixe or references for this. I've also been through the latest RT changes and btrfs mailing lists/changes (and this post might be served better there) and cannot find any direct reference to this. The only close match is:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=99da896614d17e8a84aeb2b2d464ac046cc8633d

This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem. 

With that in mind, I had copilot (again, apologies) have an attempt. It came up with this:

# Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
# livelock. wait_for_commit() can reselect a completed transaction that remains
# on trans_list after its waiters are woken. Skip completed entries while still
# waiting for older incomplete transactions. This is a diagnostic hypothesis,
# not a confirmed fix for the observed lockup.
# Review notes: the scan is O(n) under trans_lock. Completion state is published
# locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
# A multi-hour run did not reproduce the issue, but the lock owner and root cause
# remain unknown.
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index e15e1855fa0a..0f05fabe451c 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
                                     const enum btrfs_trans_state min_state)
 {
        struct btrfs_fs_info *fs_info = commit->fs_info;
+       struct btrfs_transaction *next;
        u64 transid = commit->transid;
        bool put = false;

@@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
                 */

                spin_lock(&fs_info->trans_lock);
-               commit = list_first_entry_or_null(&fs_info->trans_list,
-                                                 struct btrfs_transaction,
-                                                 list);
-               if (!commit || commit->transid > transid) {
+               commit = NULL;
+               list_for_each_entry(next, &fs_info->trans_list, list) {
+                       if (next->transid > transid)
+                               break;
+                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
+                               continue;
+                       commit = next;
+                       break;
+               }
+               if (!commit) {
                        spin_unlock(&fs_info->trans_lock);
                        break;
                }


Note the O(N) time here. To that end, this is very unlikely to be the correct fix, but I hope it highlights the core problem. I ran this patch for several hours with no apparent problem (versus seconds of a lockup in previous testing).

If there's a later kernel patch I've missed, or something else I ought to be looking at or test or need more info, please let me know.

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Daniel Vacek]

On Tue, 13 Oct 2026 at 19:46, Naulls, Peter <PeterNaulls@eaton.com> wrote:
>
> Originally posted to RT mailing list, see post and replies there:
>
> https://lore.kernel.org/linux-rt-users/PH0PR17MB4277167A37B20340B231443FBB922@PH0PR17MB4277.namprd17.prod.outlook.com/T/#u
>
> ________________________________________
> From: Naulls, Peter <PeterNaulls@eaton.com>
> Sent: Friday, October 9, 2026 11:34 AM
> To: linux-rt-users@vger.kernel.org <linux-rt-users@vger.kernel.org>
> Subject: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78
>
> Long term kernel hacker, but very infrequent contributor.
>
> We are using the RT kernel patch on x86 under kernel 6.6.78-rt51. Specifically, this is the kernel from meta-intel in Yocto Scarthgap in the intel-kernel-rt recipe.
>
> It appears under some high load conditions, and especially triggered by calls to sync(), there is a live lock condition in the btrfs code. I used copilot for this analysis, with apologies due here where relevant. I'm familiar with kernel code, and I'm well aware how complex filesystem interactions can be.
>
> Here is the lock we observe:
>
>
> sync (PID 52371)
>   btrfs_commit_transaction+0xc48
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   btrfs_sync_fs
>   sync_fs_one_sb
>   ksys_sync
>
> ordered-I/O completion workers
>   join_transaction+0x2b
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_join_transaction
>   btrfs_finish_one_ordered
>   btrfs_work_helper
>
> journal/fsync callers
>   wait_current_trans+0x33
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_start_transaction
>   btrfs_sync_file
>   vfs_fsync_range
>   fsync
>
> btrfs-transaction thread
>   transaction_kthread+0x76
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   kthread
>
>
> The processes are then not killable, indicting a kernel thread problem.
>
> This was largely found because our userspace code calls sync() excessively (being separately addressed), although that's not the proximate cause. We ended up writing shell script which writes blocks of data to a brtfs partition over several threads and also calls sync and were able to repeat it within 10s of seconds. It is not repeatable in the non-RT kernel.
>
> To add:
>
> I have been extensively through the later kernels in 6.6, and later series and have not found any direct fixe or references for this. I've also been through the latest RT changes and btrfs mailing lists/changes (and this post might be served better there) and cannot find any direct reference to this. The only close match is:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=99da896614d17e8a84aeb2b2d464ac046cc8633d
>
> This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem.
>
> With that in mind, I had copilot (again, apologies) have an attempt. It came up with this:
>
> # Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
> # livelock. wait_for_commit() can reselect a completed transaction that remains
> # on trans_list after its waiters are woken. Skip completed entries while still
> # waiting for older incomplete transactions. This is a diagnostic hypothesis,
> # not a confirmed fix for the observed lockup.
> # Review notes: the scan is O(n) under trans_lock. Completion state is published
> # locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
> # A multi-hour run did not reproduce the issue, but the lock owner and root cause
> # remain unknown.

Right, the above does not explain why this only happens on RT.

Though the lock owner can be easily found using crash or drgn. One
could also expect it's task->sched_info.last_arrival timestamp would
be about close to those blocked tasks you found.

Would you mind sharing a vmcore (and (a link to) the debug symbols)?
Or eventualy the root access to the reproducer machine? (in case you
have an ephemeral VM)

I'll try to review the related code in the meantime.

--nX

> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index e15e1855fa0a..0f05fabe451c 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                                      const enum btrfs_trans_state min_state)
>  {
>         struct btrfs_fs_info *fs_info = commit->fs_info;
> +       struct btrfs_transaction *next;
>         u64 transid = commit->transid;
>         bool put = false;
>
> @@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                  */
>
>                 spin_lock(&fs_info->trans_lock);
> -               commit = list_first_entry_or_null(&fs_info->trans_list,
> -                                                 struct btrfs_transaction,
> -                                                 list);
> -               if (!commit || commit->transid > transid) {
> +               commit = NULL;
> +               list_for_each_entry(next, &fs_info->trans_list, list) {
> +                       if (next->transid > transid)
> +                               break;
> +                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
> +                               continue;
> +                       commit = next;
> +                       break;
> +               }
> +               if (!commit) {
>                         spin_unlock(&fs_info->trans_lock);
>                         break;
>                 }
>
>
> Note the O(N) time here. To that end, this is very unlikely to be the correct fix, but I hope it highlights the core problem. I ran this patch for several hours with no apparent problem (versus seconds of a lockup in previous testing).
>
> If there's a later kernel patch I've missed, or something else I ought to be looking at or test or need more info, please let me know.
>

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Daniel Vacek]

On Tue, 13 Oct 2026 at 19:46, Naulls, Peter <PeterNaulls@eaton.com> wrote:
>
> Originally posted to RT mailing list, see post and replies there:
>
> https://lore.kernel.org/linux-rt-users/PH0PR17MB4277167A37B20340B231443FBB922@PH0PR17MB4277.namprd17.prod.outlook.com/T/#u
>
> ________________________________________
> From: Naulls, Peter <PeterNaulls@eaton.com>
> Sent: Friday, October 9, 2026 11:34 AM
> To: linux-rt-users@vger.kernel.org <linux-rt-users@vger.kernel.org>
> Subject: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78
>
> Long term kernel hacker, but very infrequent contributor.
>
> We are using the RT kernel patch on x86 under kernel 6.6.78-rt51. Specifically, this is the kernel from meta-intel in Yocto Scarthgap in the intel-kernel-rt recipe.
>
> It appears under some high load conditions, and especially triggered by calls to sync(), there is a live lock condition in the btrfs code. I used copilot for this analysis, with apologies due here where relevant. I'm familiar with kernel code, and I'm well aware how complex filesystem interactions can be.
>
> Here is the lock we observe:
>
>
> sync (PID 52371)
>   btrfs_commit_transaction+0xc48
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   btrfs_sync_fs
>   sync_fs_one_sb
>   ksys_sync
>
> ordered-I/O completion workers
>   join_transaction+0x2b
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_join_transaction
>   btrfs_finish_one_ordered
>   btrfs_work_helper
>
> journal/fsync callers
>   wait_current_trans+0x33
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_start_transaction
>   btrfs_sync_file
>   vfs_fsync_range
>   fsync
>
> btrfs-transaction thread
>   transaction_kthread+0x76
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   kthread

Neither of these seems to come from an interrupt. By any chance, have
you seen a "sleeping while atomic" error in dmesg?

--nX

> The processes are then not killable, indicting a kernel thread problem.
>
> This was largely found because our userspace code calls sync() excessively (being separately addressed), although that's not the proximate cause. We ended up writing shell script which writes blocks of data to a brtfs partition over several threads and also calls sync and were able to repeat it within 10s of seconds. It is not repeatable in the non-RT kernel.
>
> To add:
>
> I have been extensively through the later kernels in 6.6, and later series and have not found any direct fixe or references for this. I've also been through the latest RT changes and btrfs mailing lists/changes (and this post might be served better there) and cannot find any direct reference to this. The only close match is:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=99da896614d17e8a84aeb2b2d464ac046cc8633d
>
> This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem.
>
> With that in mind, I had copilot (again, apologies) have an attempt. It came up with this:
>
> # Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
> # livelock. wait_for_commit() can reselect a completed transaction that remains
> # on trans_list after its waiters are woken. Skip completed entries while still
> # waiting for older incomplete transactions. This is a diagnostic hypothesis,
> # not a confirmed fix for the observed lockup.
> # Review notes: the scan is O(n) under trans_lock. Completion state is published
> # locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
> # A multi-hour run did not reproduce the issue, but the lock owner and root cause
> # remain unknown.
> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index e15e1855fa0a..0f05fabe451c 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                                      const enum btrfs_trans_state min_state)
>  {
>         struct btrfs_fs_info *fs_info = commit->fs_info;
> +       struct btrfs_transaction *next;
>         u64 transid = commit->transid;
>         bool put = false;
>
> @@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                  */
>
>                 spin_lock(&fs_info->trans_lock);
> -               commit = list_first_entry_or_null(&fs_info->trans_list,
> -                                                 struct btrfs_transaction,
> -                                                 list);
> -               if (!commit || commit->transid > transid) {
> +               commit = NULL;
> +               list_for_each_entry(next, &fs_info->trans_list, list) {
> +                       if (next->transid > transid)
> +                               break;
> +                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
> +                               continue;
> +                       commit = next;
> +                       break;
> +               }
> +               if (!commit) {
>                         spin_unlock(&fs_info->trans_lock);
>                         break;
>                 }
>
>
> Note the O(N) time here. To that end, this is very unlikely to be the correct fix, but I hope it highlights the core problem. I ran this patch for several hours with no apparent problem (versus seconds of a lockup in previous testing).
>
> If there's a later kernel patch I've missed, or something else I ought to be looking at or test or need more info, please let me know.
>

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Daniel Vacek]

On Wed, 14 Oct 2026 at 16:33, Daniel Vacek <neelx@suse.com> wrote:
> On Tue, 13 Oct 2026 at 19:46, Naulls, Peter <PeterNaulls@eaton.com> wrote:
> >
> > Originally posted to RT mailing list, see post and replies there:
> >
> > https://lore.kernel.org/linux-rt-users/PH0PR17MB4277167A37B20340B231443FBB922@PH0PR17MB4277.namprd17.prod.outlook.com/T/#u
> >
> > ________________________________________
> > From: Naulls, Peter <PeterNaulls@eaton.com>
> > Sent: Friday, October 9, 2026 11:34 AM
> > To: linux-rt-users@vger.kernel.org <linux-rt-users@vger.kernel.org>
> > Subject: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78
> >
> > Long term kernel hacker, but very infrequent contributor.
> >
> > We are using the RT kernel patch on x86 under kernel 6.6.78-rt51. Specifically, this is the kernel from meta-intel in Yocto Scarthgap in the intel-kernel-rt recipe.
> >
> > It appears under some high load conditions, and especially triggered by calls to sync(), there is a live lock condition in the btrfs code. I used copilot for this analysis, with apologies due here where relevant. I'm familiar with kernel code, and I'm well aware how complex filesystem interactions can be.
> >
> > Here is the lock we observe:
> >
> >
> > sync (PID 52371)
> >   btrfs_commit_transaction+0xc48
> >     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
> >   btrfs_sync_fs
> >   sync_fs_one_sb
> >   ksys_sync
> >
> > ordered-I/O completion workers
> >   join_transaction+0x2b
> >     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
> >   start_transaction
> >   btrfs_join_transaction
> >   btrfs_finish_one_ordered
> >   btrfs_work_helper
> >
> > journal/fsync callers
> >   wait_current_trans+0x33
> >     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
> >   start_transaction
> >   btrfs_start_transaction
> >   btrfs_sync_file
> >   vfs_fsync_range
> >   fsync
> >
> > btrfs-transaction thread
> >   transaction_kthread+0x76
> >     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
> >   kthread
>
> Neither of these seems to come from an interrupt. By any chance, have
> you seen a "sleeping while atomic" error in dmesg?

I reviewed the code from tag
lts-v6.6.78-rt51-preempt-rt-250304T022441Z in
https://github.com/intel/linux-intel-lts/commits/6.6/preempt-rt/ repo,
which I believe is the version you are using. Specifically I audited
all the usages of the `trans_lock`. Though I do not see any code which
could potentially block while holding the lock. There must be
something missing.

Anyways, perhaps you could get a hint from lockdep. Have you tried
enabling it and checking the log while the reproducer hangs?
Possibly, could you share the reproduces shell script you mentioned so
that I can confirm and debug locally, please?

--nX

> --nX
>
> > The processes are then not killable, indicting a kernel thread problem.
> >
> > This was largely found because our userspace code calls sync() excessively (being separately addressed), although that's not the proximate cause. We ended up writing shell script which writes blocks of data to a brtfs partition over several threads and also calls sync and were able to repeat it within 10s of seconds. It is not repeatable in the non-RT kernel.
> >
> > To add:
> >
> > I have been extensively through the later kernels in 6.6, and later series and have not found any direct fixe or references for this. I've also been through the latest RT changes and btrfs mailing lists/changes (and this post might be served better there) and cannot find any direct reference to this. The only close match is:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=99da896614d17e8a84aeb2b2d464ac046cc8633d
> >
> > This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem.
> >
> > With that in mind, I had copilot (again, apologies) have an attempt. It came up with this:
> >
> > # Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
> > # livelock. wait_for_commit() can reselect a completed transaction that remains
> > # on trans_list after its waiters are woken. Skip completed entries while still
> > # waiting for older incomplete transactions. This is a diagnostic hypothesis,
> > # not a confirmed fix for the observed lockup.
> > # Review notes: the scan is O(n) under trans_lock. Completion state is published
> > # locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
> > # A multi-hour run did not reproduce the issue, but the lock owner and root cause
> > # remain unknown.
> > diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> > index e15e1855fa0a..0f05fabe451c 100644
> > --- a/fs/btrfs/transaction.c
> > +++ b/fs/btrfs/transaction.c
> > @@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
> >                                      const enum btrfs_trans_state min_state)
> >  {
> >         struct btrfs_fs_info *fs_info = commit->fs_info;
> > +       struct btrfs_transaction *next;
> >         u64 transid = commit->transid;
> >         bool put = false;
> >
> > @@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
> >                  */
> >
> >                 spin_lock(&fs_info->trans_lock);
> > -               commit = list_first_entry_or_null(&fs_info->trans_list,
> > -                                                 struct btrfs_transaction,
> > -                                                 list);
> > -               if (!commit || commit->transid > transid) {
> > +               commit = NULL;
> > +               list_for_each_entry(next, &fs_info->trans_list, list) {
> > +                       if (next->transid > transid)
> > +                               break;
> > +                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
> > +                               continue;
> > +                       commit = next;
> > +                       break;
> > +               }
> > +               if (!commit) {
> >                         spin_unlock(&fs_info->trans_lock);
> >                         break;
> >                 }
> >
> >
> > Note the O(N) time here. To that end, this is very unlikely to be the correct fix, but I hope it highlights the core problem. I ran this patch for several hours with no apparent problem (versus seconds of a lockup in previous testing).
> >
> > If there's a later kernel patch I've missed, or something else I ought to be looking at or test or need more info, please let me know.
> >

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Daniel Vacek]

On Tue, 13 Oct 2026 at 19:46, Naulls, Peter <PeterNaulls@eaton.com> wrote:
>
> Originally posted to RT mailing list, see post and replies there:
>
> https://lore.kernel.org/linux-rt-users/PH0PR17MB4277167A37B20340B231443FBB922@PH0PR17MB4277.namprd17.prod.outlook.com/T/#u
>
> ________________________________________
> From: Naulls, Peter <PeterNaulls@eaton.com>
> Sent: Friday, October 9, 2026 11:34 AM
> To: linux-rt-users@vger.kernel.org <linux-rt-users@vger.kernel.org>
> Subject: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78
>
> Long term kernel hacker, but very infrequent contributor.
>
> We are using the RT kernel patch on x86 under kernel 6.6.78-rt51. Specifically, this is the kernel from meta-intel in Yocto Scarthgap in the intel-kernel-rt recipe.
>
> It appears under some high load conditions, and especially triggered by calls to sync(), there is a live lock condition in the btrfs code. I used copilot for this analysis, with apologies due here where relevant. I'm familiar with kernel code, and I'm well aware how complex filesystem interactions can be.
>
> Here is the lock we observe:
>
>
> sync (PID 52371)
>   btrfs_commit_transaction+0xc48
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   btrfs_sync_fs
>   sync_fs_one_sb
>   ksys_sync
>
> ordered-I/O completion workers
>   join_transaction+0x2b
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_join_transaction
>   btrfs_finish_one_ordered
>   btrfs_work_helper
>
> journal/fsync callers
>   wait_current_trans+0x33
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   start_transaction
>   btrfs_start_transaction
>   btrfs_sync_file
>   vfs_fsync_range
>   fsync
>
> btrfs-transaction thread
>   transaction_kthread+0x76
>     [return address immediately after rt_spin_lock(&fs_info->trans_lock)]
>   kthread
>
>
> The processes are then not killable, indicting a kernel thread problem.
>
> This was largely found because our userspace code calls sync() excessively (being separately addressed), although that's not the proximate cause. We ended up writing shell script which writes blocks of data to a brtfs partition over several threads and also calls sync and were able to repeat it within 10s of seconds. It is not repeatable in the non-RT kernel.

Thinking about it again, I don't think this is a livelock situation. I
think it's just a severe lock contention due to the number of
threads/CPUs fighting in a tight loop (due to the
btrfs_commit_transaction() and wait_for_commit() design) and as a
result the overal progress and throughput goes south.

Would you mind testing the below change? It passes my tests though I
was not able to reproduce the exact situation. I still believe it's
going to address it for you and restore the expected performance.

diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index aa03db69a016..569f5a5c9b91 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -2536,6 +2536,10 @@ int btrfs_commit_transaction(struct
btrfs_trans_handle *trans)
  btrfs_clear_space_info_full(fs_info);

  fs_info->last_trans_committed = cur_trans->transid;
+
+ spin_lock(&fs_info->trans_lock);
+ list_del_init(&cur_trans->list);
+ spin_unlock(&fs_info->trans_lock);
  /*
  * We needn't acquire the lock here because there is no other task
  * which can change it.
@@ -2544,10 +2548,6 @@ int btrfs_commit_transaction(struct
btrfs_trans_handle *trans)
  wake_up(&cur_trans->commit_wait);
  btrfs_trans_state_lockdep_release(fs_info, BTRFS_LOCKDEP_TRANS_COMPLETED);

- spin_lock(&fs_info->trans_lock);
- list_del_init(&cur_trans->list);
- spin_unlock(&fs_info->trans_lock);
-
  btrfs_put_transaction(cur_trans);
  btrfs_put_transaction(cur_trans);


--nX

> To add:
>
> I have been extensively through the later kernels in 6.6, and later series and have not found any direct fixe or references for this. I've also been through the latest RT changes and btrfs mailing lists/changes (and this post might be served better there) and cannot find any direct reference to this. The only close match is:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=99da896614d17e8a84aeb2b2d464ac046cc8633d
>
> This is in related code. I put this patch into our 6.6.78 kernel and it did not resolve the problem.
>
> With that in mind, I had copilot (again, apologies) have an attempt. It came up with this:
>
> # Experimental mitigation for a suspected PREEMPT_RT Btrfs transaction-waiter
> # livelock. wait_for_commit() can reselect a completed transaction that remains
> # on trans_list after its waiters are woken. Skip completed entries while still
> # waiting for older incomplete transactions. This is a diagnostic hypothesis,
> # not a confirmed fix for the observed lockup.
> # Review notes: the scan is O(n) under trans_lock. Completion state is published
> # locklessly elsewhere; READ_ONCE provides a single load, not acquire ordering.
> # A multi-hour run did not reproduce the issue, but the lock owner and root cause
> # remain unknown.
> diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
> index e15e1855fa0a..0f05fabe451c 100644
> --- a/fs/btrfs/transaction.c
> +++ b/fs/btrfs/transaction.c
> @@ -865,6 +865,7 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                                      const enum btrfs_trans_state min_state)
>  {
>         struct btrfs_fs_info *fs_info = commit->fs_info;
> +       struct btrfs_transaction *next;
>         u64 transid = commit->transid;
>         bool put = false;
>
> @@ -893,10 +894,16 @@ static noinline void wait_for_commit(struct btrfs_transaction *commit,
>                  */
>
>                 spin_lock(&fs_info->trans_lock);
> -               commit = list_first_entry_or_null(&fs_info->trans_list,
> -                                                 struct btrfs_transaction,
> -                                                 list);
> -               if (!commit || commit->transid > transid) {
> +               commit = NULL;
> +               list_for_each_entry(next, &fs_info->trans_list, list) {
> +                       if (next->transid > transid)
> +                               break;
> +                       if (READ_ONCE(next->state) == TRANS_STATE_COMPLETED)
> +                               continue;
> +                       commit = next;
> +                       break;
> +               }
> +               if (!commit) {
>                         spin_unlock(&fs_info->trans_lock);
>                         break;
>                 }
>
>
> Note the O(N) time here. To that end, this is very unlikely to be the correct fix, but I hope it highlights the core problem. I ran this patch for several hours with no apparent problem (versus seconds of a lockup in previous testing).
>
> If there's a later kernel patch I've missed, or something else I ought to be looking at or test or need more info, please let me know.
>

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Naulls, Peter]

Sorry for the delay, I had some other tasks and wanted to do some more rigorous testing on a recent kernel and address pieces from both mailing lists. I'm going to mix up replies here in Outlook, so bear with me. 


From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Could you try if you can reproduce this with v7.3-rc7? This one does not require any additional patches.

For the testing below, I'm now using 7.3 which was released just a few days ago. I haven't gone back to the 6.6 kernel with the suggested patch. 


> From: Daniel Vacek <neelx@suse.com
> Thinking about it again, I don't think this is a livelock situation. I
> think it's just a severe lock contention due to the number of
> threads/CPUs fighting in a tight loop (due to the
> btrfs_commit_transaction() and wait_for_commit() design) and as a
> result the overall progress and throughput goes south.

Maybe. In the isolated testing I did, it certainly looks like that - the contention eventually clears, but in the meantime there's
very high CPU usage, apparently from the sync command, but traced to the sync() handling in brtfs.  Copilot believes that might not
be the full story, but I'll get to that in a moment. 

> Would you mind testing the below change? It passes my tests though I
> was not able to reproduce the exact situation. I still believe it's
> going to address it for you and restore the expected performance.

In limited testing, this appears to resolve the problem under 7.3, where as I was able to repeat it within a few seconds otherwise. 

[snip]

So to clarify, this is on 7.3 release, on a btrfs loopback mount in /tmp, it doesn't need actual flash backing:

-----
#!/bin/sh
# Create and mount the RAM-backed (tmpfs /tmp) Btrfs loopback test filesystem.
set -eu

image=/tmp/btrfs-loop-test.img
target=/tmp/btrfs-loop-test

[ "$(id -u)" -eq 0 ] || { echo "Must run as root" >&2; exit 1; }
for path in "$image" "$target"; do
    if [ -e "$path" ] || [ -L "$path" ]; then
        echo "$path already exists; reboot or remove it first" >&2
        exit 1
    fi
done

( set -C; : > "$image" )
dd if=/dev/zero of="$image" bs=1048576 count=256 2>/dev/null
mkfs.btrfs -q -n 16384 -s 4096 -d single -m dup -O ^block-group-tree "$image"
mkdir "$target"
mount -t btrfs -o loop,noatime,compress=zstd:3,ssd,discard=async,noacl,space_cache=v2 \
    "$image" "$target"

resolved=$(readlink -f "$target")
awk -v t="$resolved" '$2 == t {print; found=1} END {if (!found) exit 1}' /proc/mounts
df -h "$target"
----

This is the minimal test case I came up with below. 

Typically run with (after creating the mount of course):

/home/root/btrfs-minimal.sh /tmp/btrfs-loop-test 4 1000 4

Failure is typically at iteration 130 with these parameters. It's possible to reproduce with "2 1000 2", although that particular case seems intermittent. 
Observe "sync" in top using CPU. 

----
#!/bin/sh
set -eu

target=${1:?Usage: btrfs-minimal.sh /mount/point [workers] [rounds] [rt_sync_helpers]}
workers=${2:-4}
rounds=${3:-1000}
helpers=${4:-4}
for value in "$workers" "$rounds" "$helpers"; do
    case "$value" in
        ''|*[!0-9]*) echo "Counts must be positive integers" >&2; exit 1 ;;
    esac
    [ "$value" -gt 0 ] || exit 1
done
[ -d "$target" ] && [ "$(id -u)" -eq 0 ] || exit 1
chrt -p $$ >/dev/null
pgrep -V >/dev/null
chrt -r 25 /bin/true
[ -x /usr/bin/sync ] || exit 1

testdir=$(mktemp -d "$target/btrfs-minimal.XXXXXX")
writer_pids=
helper_pids=

cleanup()
{
    trap - EXIT INT TERM
    for pid in $helper_pids; do
        chrt -o -p 0 "$pid" >/dev/null 2>&1 || true
        for child in $(pgrep -P "$pid" 2>/dev/null || true); do
            chrt -o -p 0 "$child" >/dev/null 2>&1 || true
        done
    done
    for pid in $helper_pids $writer_pids; do
        for child in $(pgrep -P "$pid" 2>/dev/null || true); do
            kill "$child" 2>/dev/null || true
        done
        kill "$pid" 2>/dev/null || true
    done
    for pid in $helper_pids $writer_pids; do
        wait "$pid" 2>/dev/null || true
    done
    rm -rf "$testdir"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

printf 'WARNING: global sync can stall any mounted filesystem; recovery may require a reboot.\n'
printf 'Directory: %s; normal writers: %s; rounds: %s; RR/25 sync helpers: %s\n' \
    "$testdir" "$workers" "$rounds" "$helpers"

helper=1
while [ "$helper" -le "$helpers" ]; do
    chrt -r 25 /bin/sh -c 'set -e; while :; do /usr/bin/sync; done' &
    helper_pids="$helper_pids $!"
    helper=$((helper + 1))
done

worker=1
while [ "$worker" -le "$workers" ]; do
    chrt -o 0 /bin/sh -c '
        set -eu
        round=1
        while [ "$round" -le "$2" ]; do
            if ! dd if=/dev/urandom of="$1" bs=262144 count=16 2>/dev/null; then
                printf "Write failed: %s, round %s\n" "$1" "$round" >&2
                exit 1
            fi
            /usr/bin/sync
            if [ $((round % 10)) -eq 0 ]; then
                printf "%s: round %s/%s\n" "$1" "$round" "$2"
            fi
            round=$((round + 1))
        done
    ' writer "$testdir/worker-$worker" "$rounds" &
    writer_pids="$writer_pids $!"
    worker=$((worker + 1))
done

status=0
for pid in $writer_pids; do
    wait "$pid" || status=1
done
---

So yes, your suggested patch appears to resolve it, I ran with 10000 runs and was not able to repeat. I will repeat this on our 6.6 kernel where I think there are some RT differences. 

As to why this needed an RT kernel, from copilot (so handfuls of salt):

"The RT kernel changes how trans_lock behaves. In btrfs it's an ordinary kernel spinlock, and PREEMPT_RT turns kernel spinlocks into priority-ordered sleeping locks. That change turns the loop into a starvation.

Without PREEMPT_RT, trans_lock is a real spinlock:

Tasks waiting for it are served in arrival order, regardless of priority. Once the committer asks for the lock, it gets it after the tasks already ahead of it.
Taking the lock disables preemption, so the committer can't be preempted between getting the lock and removing the transaction from the list.
The SCHED_RR waiters still loop, re-finding the finished transaction, until the committer removes it. But the committer gets through quickly, so the loop is short. The worst case is the contention and lost throughput the maintainer describes.

With PREEMPT_RT, spin_lock() uses an rtmutex. Three things change:

Waiters queue by priority, not arrival. Each looping SCHED_RR/25 waiter goes ahead of the SCHED_OTHER committer.
A higher-priority task can take the lock from a woken waiter. When the lock is released and the committer is woken to take it, an RT waiter that asks before the committer runs can grab it first. The RT waiters ask constantly, because their loop never sleeps: each finds the finished transaction still listed, sees it's complete, and goes back for the lock.
Everything is preemptible. The committer, and anyone else holding the lock, can be preempted by RT tasks at any point.
The RT waiters keep the lock in their own hands, while the one task that could end their loop waits behind them indefinitely. That's the livelock."


And as to why it thinks this might be an incomplete fix:

"The error cleanup path has the same bug and isn't fixed. btrfs_cleanup_one_transaction sets COMPLETED and wakes waiters. btrfs_cleanup_transaction only removes the transaction from the list afterwards. RT waiters in wait_for_commit can loop there exactly as before. It only runs after a filesystem error, so it's rare, but it's the same livelock."

I will now go and repeat this testing and patch on our 6.6 kernel. 


Regards, and thank you. 

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Sebastian Andrzej Siewior]

nice to see that you have made progress.

On 2026-10-21 14:46:34 [+0000], Naulls, Peter wrote:
> As to why this needed an RT kernel, from copilot (so handfuls of salt):
> 
> "The RT kernel changes how trans_lock behaves. In btrfs it's an
> ordinary kernel spinlock, and PREEMPT_RT turns kernel spinlocks into
> priority-ordered sleeping locks. That change turns the loop into a
> starvation.
> 
> Without PREEMPT_RT, trans_lock is a real spinlock:
> 
> Tasks waiting for it are served in arrival order, regardless of
> priority. Once the committer asks for the lock, it gets it after the
> tasks already ahead of it.
> Taking the lock disables preemption, so the committer can't be
> preempted between getting the lock and removing the transaction from
> the list.
> The SCHED_RR waiters still loop, re-finding the finished transaction,
> until the committer removes it. But the committer gets through
> quickly, so the loop is short. The worst case is the contention and
> lost throughput the maintainer describes.
> 
> With PREEMPT_RT, spin_lock() uses an rtmutex. Three things change:
> 
> Waiters queue by priority, not arrival. Each looping SCHED_RR/25
> waiter goes ahead of the SCHED_OTHER committer.
This is correct.

> A higher-priority task can take the lock from a woken waiter. When the
> lock is released and the committer is woken to take it, an RT waiter
> that asks before the committer runs can grab it first. The RT waiters
> ask constantly, because their loop never sleeps: each finds the
> finished transaction still listed, sees it's complete, and goes back
> for the lock.
> Everything is preemptible. The committer, and anyone else holding the
> lock, can be preempted by RT tasks at any point.
> The RT waiters keep the lock in their own hands, while the one task
> that could end their loop waits behind them indefinitely. That's the
> livelock."

That sounds plausible. I did ask if RT tasks are involved and I think
the answer as no. With just SCHED_OTHER tasks it behaves as spinlock_t
on !PREEMPT_RT (except for the preemption which should have no impact).

Sebastian

Re: Btrfs PREEMPT_RT transaction-waiter livelock: experimental mitigation in kernel 6.6.78

[Naulls, Peter]

From: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

> That sounds plausible. I did ask if RT tasks are involved and I think
> the answer as no. With just SCHED_OTHER tasks it behaves as spinlock_t
> on !PREEMPT_RT (except for the preemption which should have no impact).

Yes, we are not doing anything special RT wise apart from the kernel enablement. 

I put Daniel's patch into 6.6 and ran it 100000 times against a ramfs btrfs image
and then thousands more times against a flash backed btrfs mount and no problems. 

I'll await the final word from him on this before committing to it for our work. 

As some background, it appears the bug was introduced in early 2022:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5fd76bf31ccfecc06e2e6b29f8c809e934085b99

And is present in all kernels from 5.15 on.