* [PATCH] livepatch: Avoid blocking tasklist_lock too long
@ 2026-11-17 13:08 Yafang Shao
2026-11-17 13:29 ` sashiko-bot
0 siblings, 1 reply; 3+ messages in thread
From: Yafang Shao @ 2026-11-17 13:08 UTC (permalink / raw)
To: jpoimboe, mbenes, pmladek, joe.lawrence, song; +Cc: live-patching, Yafang Shao
Problem
=======
As more and more patches accumulate in the cumulative livepatch, we see an
increasing number of latency spikes from latency-sensitive services during
livepatch upgrades. The most recent case is the redis data_server [0].
With some tracing added to the userspace redis code and a hard trace in the
kernel, we finally tracked the root cause down to tasklist_lock contention
during livepatch transition.
When redis flushes data, it clones a new task, which takes the write side of
tasklist_lock. If a livepatch upgrade is in progress at the same time,
tasklist_lock is held for the whole transition. Below is the trace we
captured while reproducing the issue:
=== [2026-11-15 15:18:06] tid=1635309 comm=data_server WRITE wait=79372 us
queued_write_lock_slowpath+276
queued_write_lock_slowpath+276
_raw_write_lock_irq+69
copy_process+3474
kernel_clone+157
__do_sys_clone+102
__x64_sys_clone+37
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+100
=== [2026-11-15 15:18:06] klp scan held tasklist_lock(read) ~ 96864 us (comm=insmod)
klp_enable_patch.cold+247
init_module+477
do_one_initcall+70
do_init_module+82
load_module+2526
__do_sys_finit_module+200
__x64_sys_finit_module+24
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+100
=== [2026-11-15 15:18:08] klp scan held tasklist_lock(read) ~ 3504145 us (comm=kworker/8:2)
klp_transition_work_fn+36
process_one_work+492
worker_thread+83
kthread+235
ret_from_fork+31
crash> p klp_signals_cnt
klp_signals_cnt = $1 = 1
The latency budget of our redis workload is 50 ms, while the clone() call
observed during the livepatch transition took ~79 ms.
Environment used for the reproduction
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
server: 256C AMD EPYC server
tasks: 1000~2000
number of patched functions: around 80
Solution
========
The redis application can work around this issue by removing the clone()
from its data flush path, but I believe we should improve this in the kernel
if we can.
In 2025, we discussed completely replacing tasklist_lock with RCU [1], but
Petr pointed out that it is far more complicated than we expected [2], so we
ended up abandoning that idea.
We still have a more lightweight way to partially replace this heavy
tasklist_lock with RCU, as Josh suggested [3]:
: Maybe klp_try_complete_transition() could iterate the tasks in two
: passes? The first pass would use rcu_read_lock(). Then if all tasks
: appear to be patched, try again with tasklist_lock.
This patch implements that suggestion.
Verification
============
I verified this solution on my test server:
server: 64C AMD virtual server
tasks: ~2000 processes with ~8000 threads, ~30 running
number of patched functions: 200
- Before this patch
the maximum clone() latency during the livepatch transition exceeded
500 ms.
- After this patch
the maximum clone() latency during the livepatch transition is below
20 ms, which is within the range expected by the redis side.
Link: https://github.com/redis/redis [0]
Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]
Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
kernel/livepatch/transition.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 2351a19ac2a9..e058d21d84b7 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
*
* Usually this will transition most (or all) of the tasks on a system
* unless the patch includes changes to a very common function.
+ *
+ * To avoid holding the tasklist_lock for too long, we iterate over the
+ * tasks in two passes. In the first pass, we use the lightweight RCU
+ * read lock to iterate through most tasks. This pass will cover the
+ * majority of tasks, except for some newly forked tasks which may be
+ * added into the tasklist concurrently. In the second pass, we acquire
+ * the tasklist_lock to handle the tasks missed in the first pass.
*/
+
+ /* First pass: Iterate over tasks using the lightweight RCU read lock. */
+ rcu_read_lock();
+ for_each_process_thread(g, task) {
+ /* klp_check_stack() requires preemption to be disabled */
+ if (task == current)
+ continue;
+
+ klp_try_switch_task(task);
+ }
+ rcu_read_unlock();
+
+ /* Second pass: Iterate over missed tasks, holding tasklist_lock. */
read_lock(&tasklist_lock);
for_each_process_thread(g, task)
if (!klp_try_switch_task(task))
--
2.52.0
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] livepatch: Avoid blocking tasklist_lock too long
2026-11-17 13:08 [PATCH] livepatch: Avoid blocking tasklist_lock too long Yafang Shao
@ 2026-11-17 13:29 ` sashiko-bot
2026-11-17 14:26 ` Yafang Shao
0 siblings, 1 reply; 3+ messages in thread
From: sashiko-bot @ 2026-11-17 13:29 UTC (permalink / raw)
To: Yafang Shao, jpoimboe, mbenes, pmladek, joe.lawrence, song; +Cc: live-patching
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] livepatch: race between klp_try_complete_transition() and
klp_copy_process()
--
commit ceb01fc48f77409ab4a5313629520f441760c90c
Author: Yafang Shao <laoar.shao@gmail.com>
livepatch: Avoid blocking tasklist_lock too long
This commit splits the task iteration in klp_try_complete_transition() into
a first pass under rcu_read_lock() and a second pass under tasklist_lock to
reduce tasklist_lock hold times during livepatch transitions.
Link: https://github.com/redis/redis [0]
Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]
> diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> index 2351a19ac2a9e..e058d21d84b75 100644
> --- a/kernel/livepatch/transition.c
> +++ b/kernel/livepatch/transition.c
> @@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
> *
> * Usually this will transition most (or all) of the tasks on a system
> * unless the patch includes changes to a very common function.
> + *
> + * To avoid holding the tasklist_lock for too long, we iterate over the
> + * tasks in two passes. In the first pass, we use the lightweight RCU
> + * read lock to iterate through most tasks. This pass will cover the
> + * majority of tasks, except for some newly forked tasks which may be
> + * added into the tasklist concurrently. In the second pass, we acquire
> + * the tasklist_lock to handle the tasks missed in the first pass.
> */
> +
> + /* First pass: Iterate over tasks using the lightweight RCU read lock. */
> + rcu_read_lock();
> + for_each_process_thread(g, task) {
> + /* klp_check_stack() requires preemption to be disabled */
> + if (task == current)
> + continue;
> +
> + klp_try_switch_task(task);
> + }
> + rcu_read_unlock();
[Severity: Medium]
Can this RCU pass race with klp_copy_process() on CONFIG_PREEMPT_RT kernels
and leave a newly forked child with TIF_PATCH_PENDING still set when the
transition completes?
klp_copy_process() relies on tasklist_lock to serialize against transition
updates while copying TIF_PATCH_PENDING and patch_state from current to
child:
kernel/livepatch/transition.c:klp_copy_process() {
if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
set_tsk_thread_flag(child, TIF_PATCH_PENDING);
else
clear_tsk_thread_flag(child, TIF_PATCH_PENDING);
child->patch_state = current->patch_state;
}
On CONFIG_PREEMPT_RT, write_lock_irq(&tasklist_lock) in copy_process() is a
preemptible rtmutex lock, and unwinders such as x86_64 ORC unwind reliably
across in-kernel preemption frames:
CPU0 (parent in fork) CPU1 (klp_try_complete_transition)
copy_process()
write_lock_irq(&tasklist_lock)
klp_copy_process(child)
set_tsk_thread_flag(child,
TIF_PATCH_PENDING)
<preempted>
/* First pass under rcu_read_lock() */
klp_try_switch_task(parent)
...
clear_tsk_thread_flag(parent,
TIF_PATCH_PENDING)
parent->patch_state = klp_target_state
child->patch_state =
current->patch_state
write_unlock_irq(&tasklist_lock)
/* Second pass under tasklist_lock */
klp_try_switch_task(child)
if (task->patch_state == klp_target_state)
return true;
klp_complete_transition()
WARN_ON_ONCE(test_tsk_thread_flag(child,
TIF_PATCH_PENDING)) <-- stale flag
Because child->patch_state already matches klp_target_state, the second pass
under tasklist_lock returns early in klp_try_switch_task() without clearing
TIF_PATCH_PENDING on child. klp_complete_transition() then triggers
WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING)) and leaves the
flag set on the child task.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117130848.65142-1-laoar.shao@gmail.com?part=1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] livepatch: Avoid blocking tasklist_lock too long
2026-11-17 13:29 ` sashiko-bot
@ 2026-11-17 14:26 ` Yafang Shao
0 siblings, 0 replies; 3+ messages in thread
From: Yafang Shao @ 2026-11-17 14:26 UTC (permalink / raw)
To: sashiko-reviews
Cc: jpoimboe, mbenes, pmladek, joe.lawrence, song, live-patching
On Tue, Nov 17, 2026 at 9:29 PM <sashiko-bot@kernel.org> wrote:
>
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] livepatch: race between klp_try_complete_transition() and
> klp_copy_process()
> --
>
> commit ceb01fc48f77409ab4a5313629520f441760c90c
> Author: Yafang Shao <laoar.shao@gmail.com>
>
> livepatch: Avoid blocking tasklist_lock too long
>
> This commit splits the task iteration in klp_try_complete_transition() into
> a first pass under rcu_read_lock() and a second pass under tasklist_lock to
> reduce tasklist_lock hold times during livepatch transitions.
>
> Link: https://github.com/redis/redis [0]
> Link: https://lore.kernel.org/all/20250227024733.16989-1-laoar.shao@gmail.com/ [1]
> Link: https://lore.kernel.org/all/Z8CRXxfI24E9_IHC@pathway.suse.cz/ [2]
> Link: https://lore.kernel.org/all/20250213013603.i6uxtjvc3qxlsqwc@jpoimboe/ [3]
>
> > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > index 2351a19ac2a9e..e058d21d84b75 100644
> > --- a/kernel/livepatch/transition.c
> > +++ b/kernel/livepatch/transition.c
> > @@ -444,7 +444,27 @@ void klp_try_complete_transition(void)
> > *
> > * Usually this will transition most (or all) of the tasks on a system
> > * unless the patch includes changes to a very common function.
> > + *
> > + * To avoid holding the tasklist_lock for too long, we iterate over the
> > + * tasks in two passes. In the first pass, we use the lightweight RCU
> > + * read lock to iterate through most tasks. This pass will cover the
> > + * majority of tasks, except for some newly forked tasks which may be
> > + * added into the tasklist concurrently. In the second pass, we acquire
> > + * the tasklist_lock to handle the tasks missed in the first pass.
> > */
> > +
> > + /* First pass: Iterate over tasks using the lightweight RCU read lock. */
> > + rcu_read_lock();
> > + for_each_process_thread(g, task) {
> > + /* klp_check_stack() requires preemption to be disabled */
> > + if (task == current)
> > + continue;
> > +
> > + klp_try_switch_task(task);
> > + }
> > + rcu_read_unlock();
>
> [Severity: Medium]
> Can this RCU pass race with klp_copy_process() on CONFIG_PREEMPT_RT kernels
> and leave a newly forked child with TIF_PATCH_PENDING still set when the
> transition completes?
>
> klp_copy_process() relies on tasklist_lock to serialize against transition
> updates while copying TIF_PATCH_PENDING and patch_state from current to
> child:
>
> kernel/livepatch/transition.c:klp_copy_process() {
> if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
> set_tsk_thread_flag(child, TIF_PATCH_PENDING);
> else
> clear_tsk_thread_flag(child, TIF_PATCH_PENDING);
>
> child->patch_state = current->patch_state;
> }
>
> On CONFIG_PREEMPT_RT, write_lock_irq(&tasklist_lock) in copy_process() is a
> preemptible rtmutex lock, and unwinders such as x86_64 ORC unwind reliably
> across in-kernel preemption frames:
>
> CPU0 (parent in fork) CPU1 (klp_try_complete_transition)
> copy_process()
> write_lock_irq(&tasklist_lock)
> klp_copy_process(child)
> set_tsk_thread_flag(child,
> TIF_PATCH_PENDING)
> <preempted>
> /* First pass under rcu_read_lock() */
> klp_try_switch_task(parent)
> ...
> clear_tsk_thread_flag(parent,
> TIF_PATCH_PENDING)
> parent->patch_state = klp_target_state
> child->patch_state =
> current->patch_state
> write_unlock_irq(&tasklist_lock)
> /* Second pass under tasklist_lock */
> klp_try_switch_task(child)
> if (task->patch_state == klp_target_state)
> return true;
> klp_complete_transition()
> WARN_ON_ONCE(test_tsk_thread_flag(child,
> TIF_PATCH_PENDING)) <-- stale flag
>
> Because child->patch_state already matches klp_target_state, the second pass
> under tasklist_lock returns early in klp_try_switch_task() without clearing
> TIF_PATCH_PENDING on child. klp_complete_transition() then triggers
> WARN_ON_ONCE(test_tsk_thread_flag(task, TIF_PATCH_PENDING)) and leaves the
> flag set on the child task.
Nice catch!
Perhaps it can be resolved by explicitly disabling the preemption?
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index e058d21d84b7..81c3afa69f03 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -706,12 +706,14 @@ void klp_copy_process(struct task_struct *child)
* klp_update_patch_state(current) and __klp_sched_try_switch(), but we
* cannot race with them because we are current.
*/
+ preempt_disable();
if (test_tsk_thread_flag(current, TIF_PATCH_PENDING))
set_tsk_thread_flag(child, TIF_PATCH_PENDING);
else
clear_tsk_thread_flag(child, TIF_PATCH_PENDING);
child->patch_state = current->patch_state;
+ preempt_enable();
}
/*
--
Regards
Yafang
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-11-17 14:27 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 13:08 [PATCH] livepatch: Avoid blocking tasklist_lock too long Yafang Shao
2026-11-17 13:29 ` sashiko-bot
2026-11-17 14:26 ` Yafang Shao
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.