All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] livepatch: Avoid blocking tasklist_lock too long
@ 2026-11-17 13:08 Yafang Shao
  2026-11-17 13:29 ` sashiko-bot
  0 siblings, 1 reply; 3+ messages in thread
From: Yafang Shao @ 2026-11-17 13:08 UTC (permalink / raw)
  To: jpoimboe, mbenes, pmladek, joe.lawrence, song; +Cc: live-patching, Yafang Shao

Problem
=======

As more and more patches accumulate in the cumulative livepatch, we see an
increasing number of latency spikes from latency-sensitive services during
livepatch upgrades. The most recent case is the redis data_server [0].

With some tracing added to the userspace redis code and a hard trace in the
kernel, we finally tracked the root cause down to tasklist_lock contention
during livepatch transition.

When redis flushes data, it clones a new task, which takes the write side of
tasklist_lock. If a livepatch upgrade is in progress at the same time,
tasklist_lock is held for the whole transition. Below is the trace we
captured while reproducing the issue:

=== [2026-11-15 15:18:06] tid=1635309 comm=data_server  WRITE wait=79372 us

        queued_write_lock_slowpath+276
        queued_write_lock_slowpath+276
        _raw_write_lock_irq+69
        copy_process+3474
        kernel_clone+157
        __do_sys_clone+102
        __x64_sys_clone+37
        do_syscall_64+56
        entry_SYSCALL_64_after_hwframe+100

=== [2026-11-15 15:18:06] klp scan held tasklist_lock(read) ~ 96864 us (comm=insmod)

        klp_enable_patch.cold+247
        init_module+477
        do_one_initcall+70
        do_init_module+82
        load_module+2526
        __do_sys_finit_module+200
        __x64_sys_finit_module+24
        do_syscall_64+56
        entry_SYSCALL_64_after_hwframe+100

=== [2026-11-15 15:18:08] klp scan held tasklist_lock(read) ~ 3504145 us (comm=kworker/8:2)

        klp_transition_work_fn+36
        process_one_work+492
        worker_thread+83
        kthread+235
        ret_from_fork+31

  crash> p klp_signals_cnt
  klp_signals_cnt = $1 = 1

The latency budget of our redis workload is 50 ms, while the clone() call
observed during the livepatch transition took ~79 ms.

Environment used for the reproduction
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  server: 256C AMD EPYC server
  tasks: 1000~2000
  number of patched functions: around 80

Solution
========

The redis application can work around this issue by removing the clone()
from its data flush path, but I believe we should improve this in the kernel
if we can.

In 2025, we discussed completely replacing tasklist_lock with RCU [1], but
Petr pointed out that it is far more complicated than we expected [2], so we
ended up abandoning that idea.

We still have a more lightweight way to partially replace this heavy
tasklist_lock with RCU, as Josh suggested [3]:

  : Maybe klp_try_complete_transition() could iterate the tasks in two
  : passes?  The first pass would use rcu_read_lock().  Then if all tasks
  : appear to be patched, try again with tasklist_lock.

This patch implements that suggestion.

Verification
============

I verified this solution on my test server:

  server: 64C AMD virtual server
  tasks: ~2000 processes with ~8000 threads, ~30 running
  number of patched functions: 200

- Before this patch
  the maximum clone() latency during the livepatch transition exceeded
  500 ms.

- After this patch
  the maximum clone() latency during the livepatch transition is below
  20 ms, which is within the range expected by the redis side.

Link: https://github.com/redis/redis [0]
Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]
Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 kernel/livepatch/transition.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 2351a19ac2a9..e058d21d84b7 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
 	 *
 	 * Usually this will transition most (or all) of the tasks on a system
 	 * unless the patch includes changes to a very common function.
+	 *
+	 * To avoid holding the tasklist_lock for too long, we iterate over the
+	 * tasks in two passes. In the first pass, we use the lightweight RCU
+	 * read lock to iterate through most tasks. This pass will cover the
+	 * majority of tasks, except for some newly forked tasks which may be
+	 * added into the tasklist concurrently. In the second pass, we acquire
+	 * the tasklist_lock to handle the tasks missed in the first pass.
 	 */
+
+	/* First pass: Iterate over tasks using the lightweight RCU read lock. */
+	rcu_read_lock();
+	for_each_process_thread(g, task) {
+		/* klp_check_stack() requires preemption to be disabled */
+		if (task == current)
+			continue;
+
+		klp_try_switch_task(task);
+	}
+	rcu_read_unlock();
+
+	/* Second pass: Iterate over missed tasks, holding tasklist_lock. */
 	read_lock(&tasklist_lock);
 	for_each_process_thread(g, task)
 		if (!klp_try_switch_task(task))
-- 
2.52.0


^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH] livepatch: Avoid blocking tasklist_lock too long
  2026-11-17 13:08 [PATCH] livepatch: Avoid blocking tasklist_lock too long Yafang Shao
@ 2026-11-17 13:29 ` sashiko-bot
  2026-11-17 14:26   ` Yafang Shao
  0 siblings, 1 reply; 3+ messages in thread
From: sashiko-bot @ 2026-11-17 13:29 UTC (permalink / raw)
  To: Yafang Shao, jpoimboe, mbenes, pmladek, joe.lawrence, song; +Cc: live-patching

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] livepatch: race between klp_try_complete_transition() and
  klp_copy_process()
--

commit ceb01fc48f77409ab4a5313629520f441760c90c
Author: Yafang Shao <laoar.shao@gmail.com>

livepatch: Avoid blocking tasklist_lock too long

This commit splits the task iteration in klp_try_complete_transition() into
a first pass under rcu_read_lock() and a second pass under tasklist_lock to
reduce tasklist_lock hold times during livepatch transitions.

Link: https://github.com/redis/redis [0]
Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]

> diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> index 2351a19ac2a9e..e058d21d84b75 100644
> --- a/kernel/livepatch/transition.c
> +++ b/kernel/livepatch/transition.c
> @@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
>  	 *
>  	 * Usually this will transition most (or all) of the tasks on a system
>  	 * unless the patch includes changes to a very common function.
> +	 *
> +	 * To avoid holding the tasklist_lock for too long, we iterate over the
> +	 * tasks in two passes. In the first pass, we use the lightweight RCU
> +	 * read lock to iterate through most tasks. This pass will cover the
> +	 * majority of tasks, except for some newly forked tasks which may be
> +	 * added into the tasklist concurrently. In the second pass, we acquire
> +	 * the tasklist_lock to handle the tasks missed in the first pass.
>  	 */
> +
> +	/* First pass: Iterate over tasks using the lightweight RCU read lock. */
> +	rcu_read_lock();
> +	for_each_process_thread(g, task) {
> +		/* klp_check_stack() requires preemption to be disabled */
> +		if (task == current)
> +			continue;
> +
> +		klp_try_switch_task(task);
> +	}
> +	rcu_read_unlock();

[Severity: Medium]
Can this RCU pass race with klp_copy_process() on CONFIG_PREEMPT_RT kernels
and leave a newly forked child with TIF_PATCH_PENDING still set when the
transition completes?

klp_copy_process() relies on tasklist_lock to serialize against transition
updates while copying TIF_PATCH_PENDING and patch_state from current to
child:

kernel/livepatch/transition.c:klp_copy_process() {
	if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
		set_tsk_thread_flag(child, TIF_PATCH_PENDING);
	else
		clear_tsk_thread_flag(child, TIF_PATCH_PENDING);

	child->patch_state = current->patch_state;
}

On CONFIG_PREEMPT_RT, write_lock_irq(&tasklist_lock) in copy_process() is a
preemptible rtmutex lock, and unwinders such as x86_64 ORC unwind reliably
across in-kernel preemption frames:

CPU0 (parent in fork)             CPU1 (klp_try_complete_transition)
copy_process()
  write_lock_irq(&tasklist_lock)
  klp_copy_process(child)
    set_tsk_thread_flag(child,
      TIF_PATCH_PENDING)
    <preempted>
                                  /* First pass under rcu_read_lock() */
                                  klp_try_switch_task(parent)
                                    ...
                                    clear_tsk_thread_flag(parent,
                                      TIF_PATCH_PENDING)
                                    parent->patch_state = klp_target_state
    child->patch_state =
      current->patch_state
  write_unlock_irq(&tasklist_lock)
                                  /* Second pass under tasklist_lock */
                                  klp_try_switch_task(child)
                                    if (task->patch_state == klp_target_state)
                                      return true;
                                  klp_complete_transition()
                                    WARN_ON_ONCE(test_tsk_thread_flag(child,
                                      TIF_PATCH_PENDING))  <-- stale flag

Because child->patch_state already matches klp_target_state, the second pass
under tasklist_lock returns early in klp_try_switch_task() without clearing
TIF_PATCH_PENDING on child.  klp_complete_transition() then triggers
WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING)) and leaves the
flag set on the child task.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117130848.65142-1-laoar.shao@gmail.com?part=1

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] livepatch: Avoid blocking tasklist_lock too long
  2026-11-17 13:29 ` sashiko-bot
@ 2026-11-17 14:26   ` Yafang Shao
  0 siblings, 0 replies; 3+ messages in thread
From: Yafang Shao @ 2026-11-17 14:26 UTC (permalink / raw)
  To: sashiko-reviews
  Cc: jpoimboe, mbenes, pmladek, joe.lawrence, song, live-patching

On Tue, Nov 17, 2026 at 9:29 PM <sashiko-bot@kernel.org> wrote:
>
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] livepatch: race between klp_try_complete_transition() and
>   klp_copy_process()
> --
>
> commit ceb01fc48f77409ab4a5313629520f441760c90c
> Author: Yafang Shao <laoar.shao@gmail.com>
>
> livepatch: Avoid blocking tasklist_lock too long
>
> This commit splits the task iteration in klp_try_complete_transition() into
> a first pass under rcu_read_lock() and a second pass under tasklist_lock to
> reduce tasklist_lock hold times during livepatch transitions.
>
> Link: https://github.com/redis/redis [0]
> Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
> Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
> Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]
>
> > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > index 2351a19ac2a9e..e058d21d84b75 100644
> > --- a/kernel/livepatch/transition.c
> > +++ b/kernel/livepatch/transition.c
> > @@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
> >        *
> >        * Usually this will transition most (or all) of the tasks on a system
> >        * unless the patch includes changes to a very common function.
> > +      *
> > +      * To avoid holding the tasklist_lock for too long, we iterate over the
> > +      * tasks in two passes. In the first pass, we use the lightweight RCU
> > +      * read lock to iterate through most tasks. This pass will cover the
> > +      * majority of tasks, except for some newly forked tasks which may be
> > +      * added into the tasklist concurrently. In the second pass, we acquire
> > +      * the tasklist_lock to handle the tasks missed in the first pass.
> >        */
> > +
> > +     /* First pass: Iterate over tasks using the lightweight RCU read lock. */
> > +     rcu_read_lock();
> > +     for_each_process_thread(g, task) {
> > +             /* klp_check_stack() requires preemption to be disabled */
> > +             if (task == current)
> > +                     continue;
> > +
> > +             klp_try_switch_task(task);
> > +     }
> > +     rcu_read_unlock();
>
> [Severity: Medium]
> Can this RCU pass race with klp_copy_process() on CONFIG_PREEMPT_RT kernels
> and leave a newly forked child with TIF_PATCH_PENDING still set when the
> transition completes?
>
> klp_copy_process() relies on tasklist_lock to serialize against transition
> updates while copying TIF_PATCH_PENDING and patch_state from current to
> child:
>
> kernel/livepatch/transition.c:klp_copy_process() {
>         if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
>                 set_tsk_thread_flag(child, TIF_PATCH_PENDING);
>         else
>                 clear_tsk_thread_flag(child, TIF_PATCH_PENDING);
>
>         child->patch_state = current->patch_state;
> }
>
> On CONFIG_PREEMPT_RT, write_lock_irq(&tasklist_lock) in copy_process() is a
> preemptible rtmutex lock, and unwinders such as x86_64 ORC unwind reliably
> across in-kernel preemption frames:
>
> CPU0 (parent in fork)             CPU1 (klp_try_complete_transition)
> copy_process()
>   write_lock_irq(&tasklist_lock)
>   klp_copy_process(child)
>     set_tsk_thread_flag(child,
>       TIF_PATCH_PENDING)
>     <preempted>
>                                   /* First pass under rcu_read_lock() */
>                                   klp_try_switch_task(parent)
>                                     ...
>                                     clear_tsk_thread_flag(parent,
>                                       TIF_PATCH_PENDING)
>                                     parent->patch_state = klp_target_state
>     child->patch_state =
>       current->patch_state
>   write_unlock_irq(&tasklist_lock)
>                                   /* Second pass under tasklist_lock */
>                                   klp_try_switch_task(child)
>                                     if (task->patch_state == klp_target_state)
>                                       return true;
>                                   klp_complete_transition()
>                                     WARN_ON_ONCE(test_tsk_thread_flag(child,
>                                       TIF_PATCH_PENDING))  <-- stale flag
>
> Because child->patch_state already matches klp_target_state, the second pass
> under tasklist_lock returns early in klp_try_switch_task() without clearing
> TIF_PATCH_PENDING on child.  klp_complete_transition() then triggers
> WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING)) and leaves the
> flag set on the child task.

Nice catch!

Perhaps it can be resolved by explicitly disabling the preemption?

diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index e058d21d84b7..81c3afa69f03 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -706,12 +706,14 @@ void klp_copy_process(struct task_struct *child)
         * klp_update_patch_state(current) and __klp_sched_try_switch(), but we
         * cannot race with them because we are current.
         */
+       preempt_disable();
        if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
                set_tsk_thread_flag(child, TIF_PATCH_PENDING);
        else
                clear_tsk_thread_flag(child, TIF_PATCH_PENDING);

        child->patch_state = current->patch_state;
+       preempt_enable();
 }

 /*

-- 
Regards
Yafang

^ permalink raw reply related	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2026-11-17 14:27 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 13:08 [PATCH] livepatch: Avoid blocking tasklist_lock too long Yafang Shao
2026-11-17 13:29 ` sashiko-bot
2026-11-17 14:26   ` Yafang Shao

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.