[PATCH v1 0/2] Debug and fix livepatch issues on LoongArch

[PATCH v1 0/2] Debug and fix livepatch issues on LoongArch

[Tiezhu Yang]

When testing the kernel live patching, there is a timeout over 15 seconds,
debug and fix the issues on LoongArch.

Tiezhu Yang (2):
  livepatch: Add config LIVEPATCH_DEBUG to get debug information
  LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

 arch/loongarch/kernel/stacktrace.c | 10 ++++++++++
 kernel/livepatch/Kconfig           |  8 ++++++++
 kernel/livepatch/transition.c      |  4 ++++
 3 files changed, 22 insertions(+)

-- 
2.42.0

[PATCH v1 1/2] livepatch: Add config LIVEPATCH_DEBUG to get debug information

[Tiezhu Yang]

Add config LIVEPATCH_DEBUG and define DEBUG if CONFIG_LIVEPATCH_DEBUG
is set, then pr_debug() can print a debug level message, it is a easy
way to get debug information without dynamic debugging.

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 kernel/livepatch/Kconfig      | 8 ++++++++
 kernel/livepatch/transition.c | 4 ++++
 2 files changed, 12 insertions(+)

diff --git a/kernel/livepatch/Kconfig b/kernel/livepatch/Kconfig
index 53d51ed619a3..4843665b1939 100644
--- a/kernel/livepatch/Kconfig
+++ b/kernel/livepatch/Kconfig
@@ -18,3 +18,11 @@ config LIVEPATCH
 	  module uses the interface provided by this option to register
 	  a patch, causing calls to patched functions to be redirected
 	  to new function code contained in the patch module.
+
+config LIVEPATCH_DEBUG
+	bool "Kernel Live Patching debug"
+	depends on LIVEPATCH
+	help
+	  Say Y here to print a debug level message with pr_debug() for
+	  the Kernel Live Patching code, it is a easy way to get debug
+	  information without dynamic debugging.
diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 2351a19ac2a9..0ab3e5684680 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -5,6 +5,10 @@
  * Copyright (C) 2015-2016 Josh Poimboeuf <jpoimboe@redhat.com>
  */
 
+#ifdef CONFIG_LIVEPATCH_DEBUG
+#define DEBUG
+#endif
+
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
 #include <linux/cpu.h>
-- 
2.42.0

Re: [PATCH v1 1/2] livepatch: Add config LIVEPATCH_DEBUG to get debug information

[Miroslav Benes]

Hi,

On Tue, 9 Sep 2025, Tiezhu Yang wrote:

> Add config LIVEPATCH_DEBUG and define DEBUG if CONFIG_LIVEPATCH_DEBUG
> is set, then pr_debug() can print a debug level message, it is a easy
> way to get debug information without dynamic debugging.

I do not have a strong opinion but is it really worth it? Configuring 
dynamic debug is not difficult, it is more targetted (you can enable it 
just for a subset of functions in livepatch subsystem) and it can also be 
done on the command line.

Miroslav

Re: [PATCH v1 1/2] livepatch: Add config LIVEPATCH_DEBUG to get debug information

[Tiezhu Yang]

On 2025/9/11 下午9:50, Miroslav Benes wrote:
> Hi,
> 
> On Tue, 9 Sep 2025, Tiezhu Yang wrote:
> 
>> Add config LIVEPATCH_DEBUG and define DEBUG if CONFIG_LIVEPATCH_DEBUG
>> is set, then pr_debug() can print a debug level message, it is a easy
>> way to get debug information without dynamic debugging.
> 
> I do not have a strong opinion but is it really worth it? Configuring

This is an alternative way, there are some similar usages:

drivers/iommu/exynos-iommu.c:
#ifdef CONFIG_EXYNOS_IOMMU_DEBUG
#define DEBUG
#endif

drivers/mtd/nand/raw/s3c2410.c:
#ifdef CONFIG_MTD_NAND_S3C2410_DEBUG
#define DEBUG
#endif

drivers/usb/storage/usb.c:
#ifdef CONFIG_USB_STORAGE_DEBUG
#define DEBUG
#endif

> dynamic debug is not difficult, it is more targetted (you can enable it
> just for a subset of functions in livepatch subsystem) and it can also be
> done on the command line.

Yes, this is true. It is up to the maintainers to apply this patch
or not.

Thanks,
Tiezhu

Re: [PATCH v1 1/2] livepatch: Add config LIVEPATCH_DEBUG to get debug information

[Miroslav Benes]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Fri, 12 Sep 2025, Tiezhu Yang wrote:

> On 2025/9/11 下午9:50, Miroslav Benes wrote:
> > Hi,
> > 
> > On Tue, 9 Sep 2025, Tiezhu Yang wrote:
> > 
> >> Add config LIVEPATCH_DEBUG and define DEBUG if CONFIG_LIVEPATCH_DEBUG
> >> is set, then pr_debug() can print a debug level message, it is a easy
> >> way to get debug information without dynamic debugging.
> > 
> > I do not have a strong opinion but is it really worth it? Configuring
> 
> This is an alternative way, there are some similar usages:
> 
> drivers/iommu/exynos-iommu.c:
> #ifdef CONFIG_EXYNOS_IOMMU_DEBUG
> #define DEBUG
> #endif
> 
> drivers/mtd/nand/raw/s3c2410.c:
> #ifdef CONFIG_MTD_NAND_S3C2410_DEBUG
> #define DEBUG
> #endif
> 
> drivers/usb/storage/usb.c:
> #ifdef CONFIG_USB_STORAGE_DEBUG
> #define DEBUG
> #endif
> 
> > dynamic debug is not difficult, it is more targetted (you can enable it
> > just for a subset of functions in livepatch subsystem) and it can also be
> > done on the command line.
> 
> Yes, this is true. It is up to the maintainers to apply this patch
> or not.

Right and I do not see the point to have it in the tree for the reasons 
above.

Miroslav

[PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Tiezhu Yang]

When testing the kernel live patching with "modprobe livepatch-sample",
there is a timeout over 15 seconds from "starting patching transition"
to "patching complete", dmesg shows "unreliable stack" for user tasks
in debug mode. When executing "rmmod livepatch-sample", there exists
the similar issue.

Like x86, arch_stack_walk_reliable() should return 0 for user tasks.
It is necessary to set regs->csr_prmd as task->thread.csr_prmd first,
then use user_mode() to check whether the task is in userspace.

Here are the call chains:

  klp_enable_patch()
    klp_try_complete_transition()
      klp_try_switch_task()
        klp_check_and_switch_task()
          klp_check_stack()
            stack_trace_save_tsk_reliable()
              arch_stack_walk_reliable()

With this patch, it takes a short time for patching and unpatching.

Before:

  # modprobe livepatch-sample
  # dmesg -T | tail -3
  [Sat Sep  6 11:00:20 2025] livepatch: 'livepatch_sample': starting patching transition
  [Sat Sep  6 11:00:35 2025] livepatch: signaling remaining tasks
  [Sat Sep  6 11:00:36 2025] livepatch: 'livepatch_sample': patching complete

  # echo 0 > /sys/kernel/livepatch/livepatch_sample/enabled
  # rmmod livepatch_sample
  rmmod: ERROR: Module livepatch_sample is in use
  # rmmod livepatch_sample
  # dmesg -T | tail -3
  [Sat Sep  6 11:06:05 2025] livepatch: 'livepatch_sample': starting unpatching transition
  [Sat Sep  6 11:06:20 2025] livepatch: signaling remaining tasks
  [Sat Sep  6 11:06:21 2025] livepatch: 'livepatch_sample': unpatching complete

After:

  # modprobe livepatch-sample
  # dmesg -T | tail -2
  [Sat Sep  6 11:19:00 2025] livepatch: 'livepatch_sample': starting patching transition
  [Sat Sep  6 11:19:01 2025] livepatch: 'livepatch_sample': patching complete

  # echo 0 > /sys/kernel/livepatch/livepatch_sample/enabled
  # rmmod livepatch_sample
  # dmesg -T | tail -2
  [Sat Sep  6 11:21:10 2025] livepatch: 'livepatch_sample': starting unpatching transition
  [Sat Sep  6 11:21:11 2025] livepatch: 'livepatch_sample': unpatching complete

While at it, do the similar thing for arch_stack_walk() to avoid
potential issues.

Cc: stable@vger.kernel.org # v6.9+
Fixes: 199cc14cb4f1 ("LoongArch: Add kernel livepatching support")
Reported-by: Xi Zhang <zhangxi@kylinos.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/kernel/stacktrace.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/loongarch/kernel/stacktrace.c b/arch/loongarch/kernel/stacktrace.c
index 9a038d1070d7..0454cce3b667 100644
--- a/arch/loongarch/kernel/stacktrace.c
+++ b/arch/loongarch/kernel/stacktrace.c
@@ -30,10 +30,15 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
 		}
 		regs->regs[1] = 0;
 		regs->regs[22] = 0;
+		regs->csr_prmd = task->thread.csr_prmd;
 	}
 
 	for (unwind_start(&state, task, regs);
 	     !unwind_done(&state); unwind_next_frame(&state)) {
+		/* Success path for user tasks */
+		if (user_mode(regs))
+			return;
+
 		addr = unwind_get_return_address(&state);
 		if (!addr || !consume_entry(cookie, addr))
 			break;
@@ -57,9 +62,14 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
 	}
 	regs->regs[1] = 0;
 	regs->regs[22] = 0;
+	regs->csr_prmd = task->thread.csr_prmd;
 
 	for (unwind_start(&state, task, regs);
 	     !unwind_done(&state) && !unwind_error(&state); unwind_next_frame(&state)) {
+		/* Success path for user tasks */
+		if (user_mode(regs))
+			return 0;
+
 		addr = unwind_get_return_address(&state);
 
 		/*
-- 
2.42.0

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Jinyang He]

On 2025-09-09 19:31, Tiezhu Yang wrote:

> When testing the kernel live patching with "modprobe livepatch-sample",
> there is a timeout over 15 seconds from "starting patching transition"
> to "patching complete", dmesg shows "unreliable stack" for user tasks
> in debug mode. When executing "rmmod livepatch-sample", there exists
> the similar issue.
>
> Like x86, arch_stack_walk_reliable() should return 0 for user tasks.
> It is necessary to set regs->csr_prmd as task->thread.csr_prmd first,
> then use user_mode() to check whether the task is in userspace.
>
> Here are the call chains:
>
>    klp_enable_patch()
>      klp_try_complete_transition()
>        klp_try_switch_task()
>          klp_check_and_switch_task()
>            klp_check_stack()
>              stack_trace_save_tsk_reliable()
>                arch_stack_walk_reliable()
>
> With this patch, it takes a short time for patching and unpatching.
>
> Before:
>
>    # modprobe livepatch-sample
>    # dmesg -T | tail -3
>    [Sat Sep  6 11:00:20 2025] livepatch: 'livepatch_sample': starting patching transition
>    [Sat Sep  6 11:00:35 2025] livepatch: signaling remaining tasks
>    [Sat Sep  6 11:00:36 2025] livepatch: 'livepatch_sample': patching complete
>
>    # echo 0 > /sys/kernel/livepatch/livepatch_sample/enabled
>    # rmmod livepatch_sample
>    rmmod: ERROR: Module livepatch_sample is in use
>    # rmmod livepatch_sample
>    # dmesg -T | tail -3
>    [Sat Sep  6 11:06:05 2025] livepatch: 'livepatch_sample': starting unpatching transition
>    [Sat Sep  6 11:06:20 2025] livepatch: signaling remaining tasks
>    [Sat Sep  6 11:06:21 2025] livepatch: 'livepatch_sample': unpatching complete
>
> After:
>
>    # modprobe livepatch-sample
>    # dmesg -T | tail -2
>    [Sat Sep  6 11:19:00 2025] livepatch: 'livepatch_sample': starting patching transition
>    [Sat Sep  6 11:19:01 2025] livepatch: 'livepatch_sample': patching complete
>
>    # echo 0 > /sys/kernel/livepatch/livepatch_sample/enabled
>    # rmmod livepatch_sample
>    # dmesg -T | tail -2
>    [Sat Sep  6 11:21:10 2025] livepatch: 'livepatch_sample': starting unpatching transition
>    [Sat Sep  6 11:21:11 2025] livepatch: 'livepatch_sample': unpatching complete
>
> While at it, do the similar thing for arch_stack_walk() to avoid
> potential issues.
>
> Cc: stable@vger.kernel.org # v6.9+
> Fixes: 199cc14cb4f1 ("LoongArch: Add kernel livepatching support")
> Reported-by: Xi Zhang <zhangxi@kylinos.cn>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
>   arch/loongarch/kernel/stacktrace.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
>
> diff --git a/arch/loongarch/kernel/stacktrace.c b/arch/loongarch/kernel/stacktrace.c
> index 9a038d1070d7..0454cce3b667 100644
> --- a/arch/loongarch/kernel/stacktrace.c
> +++ b/arch/loongarch/kernel/stacktrace.c
> @@ -30,10 +30,15 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
>   		}
>   		regs->regs[1] = 0;
>   		regs->regs[22] = 0;
> +		regs->csr_prmd = task->thread.csr_prmd;
>   	}
>   
>   	for (unwind_start(&state, task, regs);
>   	     !unwind_done(&state); unwind_next_frame(&state)) {
> +		/* Success path for user tasks */
> +		if (user_mode(regs))
> +			return;
> +
>   		addr = unwind_get_return_address(&state);
>   		if (!addr || !consume_entry(cookie, addr))
>   			break;
> @@ -57,9 +62,14 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
>   	}
>   	regs->regs[1] = 0;
>   	regs->regs[22] = 0;
> +	regs->csr_prmd = task->thread.csr_prmd;
>   
>   	for (unwind_start(&state, task, regs);
>   	     !unwind_done(&state) && !unwind_error(&state); unwind_next_frame(&state)) {
> +		/* Success path for user tasks */
> +		if (user_mode(regs))
> +			return 0;
> +
>   		addr = unwind_get_return_address(&state);
>   
>   		/*
Hi, Tiezhu,

We update stack info by get_stack_info when meet ORC_TYPE_REGS in
unwind_next_frame. And in arch_stack_walk(_reliable), we always
do unwind_done before unwind_next_frame. So is there anything
error in get_stack_info which causing regs is user_mode while
stack is not STACK_TYPE_UNKNOWN?

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Tiezhu Yang]

On 2025/9/10 上午9:11, Jinyang He wrote:
> On 2025-09-09 19:31, Tiezhu Yang wrote:
> 
>> When testing the kernel live patching with "modprobe livepatch-sample",
>> there is a timeout over 15 seconds from "starting patching transition"
>> to "patching complete", dmesg shows "unreliable stack" for user tasks
>> in debug mode. When executing "rmmod livepatch-sample", there exists
>> the similar issue.

...

>> @@ -57,9 +62,14 @@ int arch_stack_walk_reliable(stack_trace_consume_fn 
>> consume_entry,
>>       }
>>       regs->regs[1] = 0;
>>       regs->regs[22] = 0;
>> +    regs->csr_prmd = task->thread.csr_prmd;
>>       for (unwind_start(&state, task, regs);
>>            !unwind_done(&state) && !unwind_error(&state); 
>> unwind_next_frame(&state)) {
>> +        /* Success path for user tasks */
>> +        if (user_mode(regs))
>> +            return 0;
>> +
>>           addr = unwind_get_return_address(&state);
>>           /*
> Hi, Tiezhu,
> 
> We update stack info by get_stack_info when meet ORC_TYPE_REGS in
> unwind_next_frame. And in arch_stack_walk(_reliable), we always
> do unwind_done before unwind_next_frame. So is there anything
> error in get_stack_info which causing regs is user_mode while
> stack is not STACK_TYPE_UNKNOWN?

When testing the kernel live patching, the error code path in
unwind_next_frame() is:

   switch (orc->fp_reg) {
           case ORC_REG_PREV_SP:
                   p = (unsigned long *)(state->sp + orc->fp_offset);
                   if (!stack_access_ok(state, (unsigned long)p, 
sizeof(unsigned long)))
                           goto err;

for this case, get_stack_info() does not return 0 due to in_task_stack()
is not true, then goto error, state->stack_info.type = STACK_TYPE_UNKNOWN
and state->error = true. In arch_stack_walk_reliable(), the loop will be
break and it returns -EINVAL, thus causing unreliable stack.

Maybe it can check whether the task is in userspace and set
state->stack_info.type = STACK_TYPE_UNKNOWN in get_stack_info(),
but I think no need to do that because it has similar effect with
this patch.

Thanks,
Tiezhu

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Jinyang He]

On 2025-09-11 19:49, Tiezhu Yang wrote:

> On 2025/9/10 上午9:11, Jinyang He wrote:
>> On 2025-09-09 19:31, Tiezhu Yang wrote:
>>
>>> When testing the kernel live patching with "modprobe livepatch-sample",
>>> there is a timeout over 15 seconds from "starting patching transition"
>>> to "patching complete", dmesg shows "unreliable stack" for user tasks
>>> in debug mode. When executing "rmmod livepatch-sample", there exists
>>> the similar issue.
>
> ...
>
>>> @@ -57,9 +62,14 @@ int 
>>> arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
>>>       }
>>>       regs->regs[1] = 0;
>>>       regs->regs[22] = 0;
>>> +    regs->csr_prmd = task->thread.csr_prmd;
>>>       for (unwind_start(&state, task, regs);
>>>            !unwind_done(&state) && !unwind_error(&state); 
>>> unwind_next_frame(&state)) {
>>> +        /* Success path for user tasks */
>>> +        if (user_mode(regs))
>>> +            return 0;
>>> +
>>>           addr = unwind_get_return_address(&state);
>>>           /*
>> Hi, Tiezhu,
>>
>> We update stack info by get_stack_info when meet ORC_TYPE_REGS in
>> unwind_next_frame. And in arch_stack_walk(_reliable), we always
>> do unwind_done before unwind_next_frame. So is there anything
>> error in get_stack_info which causing regs is user_mode while
>> stack is not STACK_TYPE_UNKNOWN?
>
> When testing the kernel live patching, the error code path in
> unwind_next_frame() is:
>
>   switch (orc->fp_reg) {
>           case ORC_REG_PREV_SP:
>                   p = (unsigned long *)(state->sp + orc->fp_offset);
>                   if (!stack_access_ok(state, (unsigned long)p, 
> sizeof(unsigned long)))
>                           goto err;
>
> for this case, get_stack_info() does not return 0 due to in_task_stack()
> is not true, then goto error, state->stack_info.type = STACK_TYPE_UNKNOWN
> and state->error = true. In arch_stack_walk_reliable(), the loop will be
> break and it returns -EINVAL, thus causing unreliable stack.
The stop position of a complete stack backtrace on LoongArch should be
the top of the task stack or until the address is_entry_func.
Otherwise, it is not a complete stack backtrace, and thus I think it
is an "unreliable stack".
I'm curious about what the ORC info at this PC.

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Tiezhu Yang]

On 2025/9/12 上午9:55, Jinyang He wrote:
> On 2025-09-11 19:49, Tiezhu Yang wrote:
> 
>> On 2025/9/10 上午9:11, Jinyang He wrote:
>>> On 2025-09-09 19:31, Tiezhu Yang wrote:
>>>
>>>> When testing the kernel live patching with "modprobe livepatch-sample",
>>>> there is a timeout over 15 seconds from "starting patching transition"
>>>> to "patching complete", dmesg shows "unreliable stack" for user tasks
>>>> in debug mode. When executing "rmmod livepatch-sample", there exists
>>>> the similar issue.

...

>> for this case, get_stack_info() does not return 0 due to in_task_stack()
>> is not true, then goto error, state->stack_info.type = STACK_TYPE_UNKNOWN
>> and state->error = true. In arch_stack_walk_reliable(), the loop will be
>> break and it returns -EINVAL, thus causing unreliable stack.
> The stop position of a complete stack backtrace on LoongArch should be
> the top of the task stack or until the address is_entry_func.
> Otherwise, it is not a complete stack backtrace, and thus I think it
> is an "unreliable stack".
> I'm curious about what the ORC info at this PC.

The unwind process has problem, I have found the root cause and am
working to fix the "unreliable stack" issue, it should and can find
the last frame, and then the user_mode() check is not necessary.

Thanks,
Tiezhu

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Miroslav Benes]

Hi,

On Tue, 9 Sep 2025, Tiezhu Yang wrote:

> When testing the kernel live patching with "modprobe livepatch-sample",
> there is a timeout over 15 seconds from "starting patching transition"
> to "patching complete", dmesg shows "unreliable stack" for user tasks
> in debug mode. When executing "rmmod livepatch-sample", there exists
> the similar issue.
> 
> Like x86, arch_stack_walk_reliable() should return 0 for user tasks.
> It is necessary to set regs->csr_prmd as task->thread.csr_prmd first,
> then use user_mode() to check whether the task is in userspace.

it is a nice optimization for sure, but "unreliable stack" messages point 
to a fact that the unwinding of these tasks is probably suboptimal and 
could be improved, no?

It would also be nice to include these messages (not for all tasks) to the 
commit message.

Regards
Miroslav

Re: [PATCH v1 2/2] LoongArch: Return 0 for user tasks in arch_stack_walk_reliable()

[Tiezhu Yang]

On 2025/9/11 下午9:44, Miroslav Benes wrote:
> Hi,
> 
> On Tue, 9 Sep 2025, Tiezhu Yang wrote:
> 
>> When testing the kernel live patching with "modprobe livepatch-sample",
>> there is a timeout over 15 seconds from "starting patching transition"
>> to "patching complete", dmesg shows "unreliable stack" for user tasks
>> in debug mode. When executing "rmmod livepatch-sample", there exists
>> the similar issue.
>>
>> Like x86, arch_stack_walk_reliable() should return 0 for user tasks.
>> It is necessary to set regs->csr_prmd as task->thread.csr_prmd first,
>> then use user_mode() to check whether the task is in userspace.
> 
> it is a nice optimization for sure, but "unreliable stack" messages point
> to a fact that the unwinding of these tasks is probably suboptimal and
> could be improved, no?

Yes, makes sense, I will fix "unreliable stack" in the next version.

> It would also be nice to include these messages (not for all tasks) to the
> commit message.

OK, will do it.

Thanks,
Tiezhu