[RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Yafang Shao]

Livepatching allows for rapid experimentation with new kernel features
without interrupting production workloads. However, static livepatches lack
the flexibility required to tune features based on task-specific attributes,
such as cgroup membership, which is critical in multi-tenant k8s
environments. Furthermore, hardcoding logic into a livepatch prevents
dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use
case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value
   of that hook based on the current task's context.

A significant challenge arises when atomic-replace is enabled. In this
mode, deploying a new livepatch changes the target function's address,
forcing a re-attachment of the BPF program. This re-attachment latency is
unacceptable in critical paths, such as those handling networking policies.

To solve this, we introduce a hybrid livepatch mode that allows specific
patches to remain non-replaceable, ensuring the function address remains
stable and the BPF program stays attached.

Furthermore, this mechanism provides a lower-maintenance alternative to
out-of-tree BPF hooks. Given the complexities of upstreaming custom BPF
hooks (e.g., [0], [1]), this hybrid mode allows for the maintenance of
stable, minimal hook points via livepatching with significantly reduced
maintenance burden.

Link: https://lwn.net/Articles/1054030/ [0]
Link: https://lwn.net/Articles/1043548/ [1]

Yafang Shao (4):
  trace: Simplify kprobe overridable function check
  trace: Allow kprobes to override livepatched functions
  livepatch: Add "replaceable" attribute to klp_patch
  livepatch: Implement livepatch hybrid mode

 include/linux/livepatch.h   |  2 ++
 kernel/livepatch/core.c     | 50 +++++++++++++++++++++++++++++++
 kernel/trace/Kconfig        | 14 +++++++++
 kernel/trace/bpf_trace.c    | 14 ++++++---
 kernel/trace/trace_kprobe.c | 49 ++++++++++++------------------
 kernel/trace/trace_probe.h  | 59 +++++++++++++++++++++++++++----------
 6 files changed, 139 insertions(+), 49 deletions(-)

-- 
2.47.3

[RFC PATCH 1/4] trace: Simplify kprobe overridable function check

[Yafang Shao]

Simplify the logic for checking overridable kprobe functions by removing
redundant code.

No functional change.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 kernel/trace/bpf_trace.c    | 13 ++++++---
 kernel/trace/trace_kprobe.c | 40 +++++----------------------
 kernel/trace/trace_probe.h  | 54 ++++++++++++++++++++++++++-----------
 3 files changed, 54 insertions(+), 53 deletions(-)

diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 0b040a417442..c901ace836cb 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 	 * Kprobe override only works if they are on the function entry,
 	 * and only if they are on the opt-in list.
 	 */
-	if (prog->kprobe_override &&
-	    (!trace_kprobe_on_func_entry(event->tp_event) ||
-	     !trace_kprobe_error_injectable(event->tp_event)))
-		return -EINVAL;
+	if (prog->kprobe_override) {
+		struct trace_kprobe *tp = trace_kprobe_primary_from_call(event->tp_event);
+
+		if (!tp)
+			return -EINVAL;
+		if (!trace_kprobe_on_func_entry(tp) ||
+		    !trace_kprobe_error_injectable(tp))
+			return -EINVAL;
+	}
 
 	mutex_lock(&bpf_event_mutex);
 
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index a5dbb72528e0..768702674a5c 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = {
 	.match = trace_kprobe_match,
 };
 
-/*
- * Kprobe event core functions
- */
-struct trace_kprobe {
-	struct dyn_event	devent;
-	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
-	unsigned long __percpu *nhit;
-	const char		*symbol;	/* symbol name */
-	struct trace_probe	tp;
-};
-
 static bool is_trace_kprobe(struct dyn_event *ev)
 {
 	return ev->ops == &trace_kprobe_ops;
@@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	return addr;
 }
 
-static nokprobe_inline struct trace_kprobe *
-trace_kprobe_primary_from_call(struct trace_event_call *call)
-{
-	struct trace_probe *tp;
-
-	tp = trace_probe_primary_from_call(call);
-	if (WARN_ON_ONCE(!tp))
-		return NULL;
-
-	return container_of(tp, struct trace_kprobe, tp);
-}
-
-bool trace_kprobe_on_func_entry(struct trace_event_call *call)
+bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? (kprobe_on_func_entry(tk->rp.kp.addr,
-			tk->rp.kp.addr ? NULL : tk->rp.kp.symbol_name,
-			tk->rp.kp.addr ? 0 : tk->rp.kp.offset) == 0) : false;
+	return !kprobe_on_func_entry(tp->rp.kp.addr,
+			tp->rp.kp.addr ? NULL : tp->rp.kp.symbol_name,
+			tp->rp.kp.addr ? 0 : tp->rp.kp.offset);
 }
 
-bool trace_kprobe_error_injectable(struct trace_event_call *call)
+bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
-	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
-
-	return tk ? within_error_injection_list(trace_kprobe_address(tk)) :
-	       false;
+	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
 static int register_kprobe_event(struct trace_kprobe *tk);
diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
index 9fc56c937130..958eb78a9068 100644
--- a/kernel/trace/trace_probe.h
+++ b/kernel/trace/trace_probe.h
@@ -30,6 +30,7 @@
 
 #include "trace.h"
 #include "trace_output.h"
+#include "trace_dynevent.h"
 
 #define MAX_TRACE_ARGS		128
 #define MAX_ARGSTR_LEN		63
@@ -210,21 +211,6 @@ DECLARE_BASIC_PRINT_TYPE_FUNC(symbol);
 #define ASSIGN_FETCH_TYPE_END {}
 #define MAX_ARRAY_LEN	64
 
-#ifdef CONFIG_KPROBE_EVENTS
-bool trace_kprobe_on_func_entry(struct trace_event_call *call);
-bool trace_kprobe_error_injectable(struct trace_event_call *call);
-#else
-static inline bool trace_kprobe_on_func_entry(struct trace_event_call *call)
-{
-	return false;
-}
-
-static inline bool trace_kprobe_error_injectable(struct trace_event_call *call)
-{
-	return false;
-}
-#endif /* CONFIG_KPROBE_EVENTS */
-
 struct probe_arg {
 	struct fetch_insn	*code;
 	bool			dynamic;/* Dynamic array (string) is used */
@@ -271,6 +257,32 @@ struct event_file_link {
 	struct list_head		list;
 };
 
+/*
+ * Kprobe event core functions
+ */
+struct trace_kprobe {
+	struct dyn_event	devent;
+	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
+	unsigned long __percpu	*nhit;
+	const char		*symbol;	/* symbol name */
+	struct trace_probe	tp;
+};
+
+#ifdef CONFIG_KPROBE_EVENTS
+bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
+bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
+#else
+static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
+{
+	return false;
+}
+
+static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
+{
+	return false;
+}
+#endif /* CONFIG_KPROBE_EVENTS */
+
 static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
 {
 	return smp_load_acquire(&tp->event->flags);
@@ -329,6 +341,18 @@ trace_probe_primary_from_call(struct trace_event_call *call)
 	return list_first_entry_or_null(&tpe->probes, struct trace_probe, list);
 }
 
+static nokprobe_inline struct trace_kprobe *
+trace_kprobe_primary_from_call(struct trace_event_call *call)
+{
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(call);
+	if (WARN_ON_ONCE(!tp))
+		return NULL;
+
+	return container_of(tp, struct trace_kprobe, tp);
+}
+
 static inline struct list_head *trace_probe_probe_list(struct trace_probe *tp)
 {
 	return &tp->event->probes;
-- 
2.47.3

Re: [RFC PATCH 1/4] trace: Simplify kprobe overridable function check

[Masami Hiramatsu]

On Thu,  2 Apr 2026 17:26:04 +0800
Yafang Shao <laoar.shao@gmail.com> wrote:

> Simplify the logic for checking overridable kprobe functions by removing
> redundant code.
> 
> No functional change.

NACK.

trace_kprobe must be hidden inside the trace_kprobe.c. It is not
designed to be exposed. 

Thank you,

> 
> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> ---
>  kernel/trace/bpf_trace.c    | 13 ++++++---
>  kernel/trace/trace_kprobe.c | 40 +++++----------------------
>  kernel/trace/trace_probe.h  | 54 ++++++++++++++++++++++++++-----------
>  3 files changed, 54 insertions(+), 53 deletions(-)
> 
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index 0b040a417442..c901ace836cb 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -1929,10 +1929,15 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
>  	 * Kprobe override only works if they are on the function entry,
>  	 * and only if they are on the opt-in list.
>  	 */
> -	if (prog->kprobe_override &&
> -	    (!trace_kprobe_on_func_entry(event->tp_event) ||
> -	     !trace_kprobe_error_injectable(event->tp_event)))
> -		return -EINVAL;
> +	if (prog->kprobe_override) {
> +		struct trace_kprobe *tp = trace_kprobe_primary_from_call(event->tp_event);
> +
> +		if (!tp)
> +			return -EINVAL;
> +		if (!trace_kprobe_on_func_entry(tp) ||
> +		    !trace_kprobe_error_injectable(tp))
> +			return -EINVAL;
> +	}
>  
>  	mutex_lock(&bpf_event_mutex);
>  
> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
> index a5dbb72528e0..768702674a5c 100644
> --- a/kernel/trace/trace_kprobe.c
> +++ b/kernel/trace/trace_kprobe.c
> @@ -53,17 +53,6 @@ static struct dyn_event_operations trace_kprobe_ops = {
>  	.match = trace_kprobe_match,
>  };
>  
> -/*
> - * Kprobe event core functions
> - */
> -struct trace_kprobe {
> -	struct dyn_event	devent;
> -	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
> -	unsigned long __percpu *nhit;
> -	const char		*symbol;	/* symbol name */
> -	struct trace_probe	tp;
> -};
> -
>  static bool is_trace_kprobe(struct dyn_event *ev)
>  {
>  	return ev->ops == &trace_kprobe_ops;
> @@ -212,33 +201,16 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
>  	return addr;
>  }
>  
> -static nokprobe_inline struct trace_kprobe *
> -trace_kprobe_primary_from_call(struct trace_event_call *call)
> -{
> -	struct trace_probe *tp;
> -
> -	tp = trace_probe_primary_from_call(call);
> -	if (WARN_ON_ONCE(!tp))
> -		return NULL;
> -
> -	return container_of(tp, struct trace_kprobe, tp);
> -}
> -
> -bool trace_kprobe_on_func_entry(struct trace_event_call *call)
> +bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
>  {
> -	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
> -
> -	return tk ? (kprobe_on_func_entry(tk->rp.kp.addr,
> -			tk->rp.kp.addr ? NULL : tk->rp.kp.symbol_name,
> -			tk->rp.kp.addr ? 0 : tk->rp.kp.offset) == 0) : false;
> +	return !kprobe_on_func_entry(tp->rp.kp.addr,
> +			tp->rp.kp.addr ? NULL : tp->rp.kp.symbol_name,
> +			tp->rp.kp.addr ? 0 : tp->rp.kp.offset);
>  }
>  
> -bool trace_kprobe_error_injectable(struct trace_event_call *call)
> +bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
>  {
> -	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
> -
> -	return tk ? within_error_injection_list(trace_kprobe_address(tk)) :
> -	       false;
> +	return within_error_injection_list(trace_kprobe_address(tp));
>  }
>  
>  static int register_kprobe_event(struct trace_kprobe *tk);
> diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
> index 9fc56c937130..958eb78a9068 100644
> --- a/kernel/trace/trace_probe.h
> +++ b/kernel/trace/trace_probe.h
> @@ -30,6 +30,7 @@
>  
>  #include "trace.h"
>  #include "trace_output.h"
> +#include "trace_dynevent.h"
>  
>  #define MAX_TRACE_ARGS		128
>  #define MAX_ARGSTR_LEN		63
> @@ -210,21 +211,6 @@ DECLARE_BASIC_PRINT_TYPE_FUNC(symbol);
>  #define ASSIGN_FETCH_TYPE_END {}
>  #define MAX_ARRAY_LEN	64
>  
> -#ifdef CONFIG_KPROBE_EVENTS
> -bool trace_kprobe_on_func_entry(struct trace_event_call *call);
> -bool trace_kprobe_error_injectable(struct trace_event_call *call);
> -#else
> -static inline bool trace_kprobe_on_func_entry(struct trace_event_call *call)
> -{
> -	return false;
> -}
> -
> -static inline bool trace_kprobe_error_injectable(struct trace_event_call *call)
> -{
> -	return false;
> -}
> -#endif /* CONFIG_KPROBE_EVENTS */
> -
>  struct probe_arg {
>  	struct fetch_insn	*code;
>  	bool			dynamic;/* Dynamic array (string) is used */
> @@ -271,6 +257,32 @@ struct event_file_link {
>  	struct list_head		list;
>  };
>  
> +/*
> + * Kprobe event core functions
> + */
> +struct trace_kprobe {
> +	struct dyn_event	devent;
> +	struct kretprobe	rp;	/* Use rp.kp for kprobe use */
> +	unsigned long __percpu	*nhit;
> +	const char		*symbol;	/* symbol name */
> +	struct trace_probe	tp;
> +};
> +
> +#ifdef CONFIG_KPROBE_EVENTS
> +bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
> +bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
> +#else
> +static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
> +{
> +	return false;
> +}
> +
> +static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
> +{
> +	return false;
> +}
> +#endif /* CONFIG_KPROBE_EVENTS */
> +
>  static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
>  {
>  	return smp_load_acquire(&tp->event->flags);
> @@ -329,6 +341,18 @@ trace_probe_primary_from_call(struct trace_event_call *call)
>  	return list_first_entry_or_null(&tpe->probes, struct trace_probe, list);
>  }
>  
> +static nokprobe_inline struct trace_kprobe *
> +trace_kprobe_primary_from_call(struct trace_event_call *call)
> +{
> +	struct trace_probe *tp;
> +
> +	tp = trace_probe_primary_from_call(call);
> +	if (WARN_ON_ONCE(!tp))
> +		return NULL;
> +
> +	return container_of(tp, struct trace_kprobe, tp);
> +}
> +
>  static inline struct list_head *trace_probe_probe_list(struct trace_probe *tp)
>  {
>  	return &tp->event->probes;
> -- 
> 2.47.3
> 


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

[RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Yafang Shao]

Introduce the ability for kprobes to override the return values of
functions that have been livepatched. This functionality is guarded by the
CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 kernel/trace/Kconfig        | 14 ++++++++++++++
 kernel/trace/bpf_trace.c    |  3 ++-
 kernel/trace/trace_kprobe.c | 17 +++++++++++++++++
 kernel/trace/trace_probe.h  |  5 +++++
 4 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 49de13cae428..db712c8cb745 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG
 
           If unsure, say N.
 
+config KPROBE_OVERRIDE_KLP_FUNC
+	bool "Allow kprobes to override livepatched functions"
+	depends on KPROBES && LIVEPATCH
+	help
+	  This option allows BPF programs to use kprobes to override functions
+	  that have already been patched by Livepatch (KLP).
+
+	  Enabling this provides a mechanism to dynamically control execution
+	  flow without requiring a reboot or a new livepatch module. It
+	  effectively combines the persistence of livepatching with the
+	  programmability of BPF.
+
+	  If unsure, say N.
+
 source "kernel/trace/rv/Kconfig"
 
 endif # FTRACE
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index c901ace836cb..08ae2b1a912c 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
 		if (!tp)
 			return -EINVAL;
 		if (!trace_kprobe_on_func_entry(tp) ||
-		    !trace_kprobe_error_injectable(tp))
+		    (!trace_kprobe_error_injectable(tp) &&
+		     !trace_kprobe_klp_func_overridable(tp)))
 			return -EINVAL;
 	}
 
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 768702674a5c..6f05451fbc76 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 	return within_error_injection_list(trace_kprobe_address(tp));
 }
 
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	bool overridable = false;
+#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC
+	struct module *mod;
+	unsigned long addr;
+
+	addr = trace_kprobe_address(tp);
+	rcu_read_lock();
+	mod = __module_address(addr);
+	if (mod && mod->klp)
+		overridable = true;
+	rcu_read_unlock();
+#endif
+	return overridable;
+}
+
 static int register_kprobe_event(struct trace_kprobe *tk);
 static int unregister_kprobe_event(struct trace_kprobe *tk);
 
diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
index 958eb78a9068..84bd2617db7c 100644
--- a/kernel/trace/trace_probe.h
+++ b/kernel/trace/trace_probe.h
@@ -271,6 +271,7 @@ struct trace_kprobe {
 #ifdef CONFIG_KPROBE_EVENTS
 bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
 bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
+bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp);
 #else
 static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
 {
@@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
 {
 	return false;
 }
+static inline bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
+{
+	return false;
+}
 #endif /* CONFIG_KPROBE_EVENTS */
 
 static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
-- 
2.47.3

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Menglong Dong]

On 2026/4/2 17:26, Yafang Shao wrote:
> Introduce the ability for kprobes to override the return values of
> functions that have been livepatched. This functionality is guarded by the
> CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.

Hi, Yafang. This is a interesting idea.

For now, the bpf_override_return() can only be used on the kernel
functions that allow error injection to prevent the BPF program from
crash the kernel. If we use it on the kernel functions that patched
by the KLP, we can crash the kernel easily by return a invalid value
with bpf_override_return(), right? (Of course, we can crash the kernel
easily with KLP too ;)

I haven't figure out the use case yet. Can KLP be used together with
the BPF program that use bpf_override_return()? The KLP will modify
the RIP on the stack, and the bpf_override_return() will modify it too.
AFAIK, there can't be two ftrace_ops that both have the
FTRACE_OPS_FL_IPMODIFY flag. Did I miss something?

It will be helpful for me to understand the use case if a selftests is
offered :)

BTW, if we allow the usage of bpf_override_return() on the KLP patched
function, we should allow the usage of BPF_MODIFY_RETURN on this
case too, right?

Thanks!
Menglong Dong

> 
> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> ---
>  kernel/trace/Kconfig        | 14 ++++++++++++++
>  kernel/trace/bpf_trace.c    |  3 ++-
>  kernel/trace/trace_kprobe.c | 17 +++++++++++++++++
>  kernel/trace/trace_probe.h  |  5 +++++
>  4 files changed, 38 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
> index 49de13cae428..db712c8cb745 100644
> --- a/kernel/trace/Kconfig
> +++ b/kernel/trace/Kconfig
> @@ -1279,6 +1279,20 @@ config HIST_TRIGGERS_DEBUG
>  
>            If unsure, say N.
>  
> +config KPROBE_OVERRIDE_KLP_FUNC
> +	bool "Allow kprobes to override livepatched functions"
> +	depends on KPROBES && LIVEPATCH
> +	help
> +	  This option allows BPF programs to use kprobes to override functions
> +	  that have already been patched by Livepatch (KLP).
> +
> +	  Enabling this provides a mechanism to dynamically control execution
> +	  flow without requiring a reboot or a new livepatch module. It
> +	  effectively combines the persistence of livepatching with the
> +	  programmability of BPF.
> +
> +	  If unsure, say N.
> +
>  source "kernel/trace/rv/Kconfig"
>  
>  endif # FTRACE
> diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
> index c901ace836cb..08ae2b1a912c 100644
> --- a/kernel/trace/bpf_trace.c
> +++ b/kernel/trace/bpf_trace.c
> @@ -1935,7 +1935,8 @@ int perf_event_attach_bpf_prog(struct perf_event *event,
>  		if (!tp)
>  			return -EINVAL;
>  		if (!trace_kprobe_on_func_entry(tp) ||
> -		    !trace_kprobe_error_injectable(tp))
> +		    (!trace_kprobe_error_injectable(tp) &&
> +		     !trace_kprobe_klp_func_overridable(tp)))
>  			return -EINVAL;
>  	}
>  
> diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
> index 768702674a5c..6f05451fbc76 100644
> --- a/kernel/trace/trace_kprobe.c
> +++ b/kernel/trace/trace_kprobe.c
> @@ -213,6 +213,23 @@ bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
>  	return within_error_injection_list(trace_kprobe_address(tp));
>  }
>  
> +bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
> +{
> +	bool overridable = false;
> +#ifdef CONFIG_KPROBE_OVERRIDE_KLP_FUNC
> +	struct module *mod;
> +	unsigned long addr;
> +
> +	addr = trace_kprobe_address(tp);
> +	rcu_read_lock();
> +	mod = __module_address(addr);
> +	if (mod && mod->klp)
> +		overridable = true;
> +	rcu_read_unlock();
> +#endif
> +	return overridable;
> +}
> +
>  static int register_kprobe_event(struct trace_kprobe *tk);
>  static int unregister_kprobe_event(struct trace_kprobe *tk);
>  
> diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
> index 958eb78a9068..84bd2617db7c 100644
> --- a/kernel/trace/trace_probe.h
> +++ b/kernel/trace/trace_probe.h
> @@ -271,6 +271,7 @@ struct trace_kprobe {
>  #ifdef CONFIG_KPROBE_EVENTS
>  bool trace_kprobe_on_func_entry(struct trace_kprobe *tp);
>  bool trace_kprobe_error_injectable(struct trace_kprobe *tp);
> +bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp);
>  #else
>  static inline bool trace_kprobe_on_func_entry(struct trace_kprobe *tp)
>  {
> @@ -281,6 +282,10 @@ static inline bool trace_kprobe_error_injectable(struct trace_kprobe *tp)
>  {
>  	return false;
>  }
> +static inline bool trace_kprobe_klp_func_overridable(struct trace_kprobe *tp)
> +{
> +	return false;
> +}
>  #endif /* CONFIG_KPROBE_EVENTS */
>  
>  static inline unsigned int trace_probe_load_flag(struct trace_probe *tp)
> 

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Yafang Shao]

On Thu, Apr 2, 2026 at 8:48 PM Menglong Dong <menglong.dong@linux.dev> wrote:
>
> On 2026/4/2 17:26, Yafang Shao wrote:
> > Introduce the ability for kprobes to override the return values of
> > functions that have been livepatched. This functionality is guarded by the
> > CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.
>
> Hi, Yafang. This is a interesting idea.
>
> For now, the bpf_override_return() can only be used on the kernel
> functions that allow error injection to prevent the BPF program from
> crash the kernel. If we use it on the kernel functions that patched
> by the KLP, we can crash the kernel easily by return a invalid value
> with bpf_override_return(), right? (Of course, we can crash the kernel
> easily with KLP too ;)

Right.
Livepatch already grants the power to modify the kernel at will;
allowing BPF to override a patched function simply adds a layer of
runtime programmability to an existing modification.

>
> I haven't figure out the use case yet. Can KLP be used together with
> the BPF program that use bpf_override_return()?

The two mechanisms do not target the same entry point: whileKLP
modifies the original kernel function, bpf_override_return() is
applied to the newly patched function provided by the KLP module.

> The KLP will modify
> the RIP on the stack, and the bpf_override_return() will modify it too.
> AFAIK, there can't be two ftrace_ops that both have the
> FTRACE_OPS_FL_IPMODIFY flag. Did I miss something?

Correct, but as noted, they target different functions

>
> It will be helpful for me to understand the use case if a selftests is
> offered :)

Here is a recent use case from our production environment.

- The livepatch

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index e378bbe5705f..047e937bfa6d 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -5175,12 +5175,22 @@ int bond_update_slave_arr(struct bonding
*bond, struct slave *skipslave)
        return ret;
 }

+/* noclone to avoid bond_get_slave_hook.constprop.0 */
+__attribute__((__noclone__, __noinline__))
+int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count)
+{
+       return -1;
+}

 static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
                                                 struct sk_buff *skb,
                                                 struct bond_up_slave *slaves)
 {
        struct slave *slave;
        unsigned int count;
+       int slave_idx;
        u32 hash;

        hash = bond_xmit_hash(bond, skb);
@@ -5188,6 +5198,13 @@ static struct slave
*bond_xmit_3ad_xor_slave_get(struct bonding *bond,
        if (unlikely(!count))
                return NULL;

+       /* Try BPF hook first - returns slave index directly */
+       slave_idx = bond_get_slave_hook(skb, hash, count);
+       /* If BPF hook returned valid slave index, use it */
+       if (slave_idx >= 0 && slave_idx < count) {
+               slave = slaves->arr[slave_idx];
+               return slave;
+       }
        slave = slaves->arr[hash % count];
        return slave;
 }

- The BPF program

SEC("kprobe/bond_get_slave_hook")
int BPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count)
{
        unsigned short net_hdr_off;
        unsigned char *head;
        struct iphdr iph;
        int *slave_idx;
        __u32 daddr;

        __u16 proto = BPF_CORE_READ(skb, protocol);
        if (proto != bpf_htons(0x0800))
                return 0;

        head = BPF_CORE_READ(skb, head);
        net_hdr_off = BPF_CORE_READ(skb, network_header);

        if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0)
                return 0;

        daddr = iph.daddr;
        slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr);
        if (slave_idx) {
                int idx = *slave_idx;

                if (idx >= 0 && idx < (int)count)
                        bpf_override_return(ctx, idx);
        }
        return 0;
}

>
> BTW, if we allow the usage of bpf_override_return() on the KLP patched
> function, we should allow the usage of BPF_MODIFY_RETURN on this
> case too, right?

It's a possibility, but I haven't tested that specifically yet.

-- 
Regards
Yafang

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Menglong Dong]

On 2026/4/2 21:20 Yafang Shao <laoar.shao@gmail.com> write:
> On Thu, Apr 2, 2026 at 8:48 PM Menglong Dong <menglong.dong@linux.dev> wrote:
> >
> > On 2026/4/2 17:26, Yafang Shao wrote:
> > > Introduce the ability for kprobes to override the return values of
> > > functions that have been livepatched. This functionality is guarded by the
> > > CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.
> >
> > Hi, Yafang. This is a interesting idea.
> >
[...]
> 
> +/* noclone to avoid bond_get_slave_hook.constprop.0 */
> +__attribute__((__noclone__, __noinline__))
> +int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count)
> +{
> +       return -1;
> +}

Hi, yafang.

I see what you mean now. So you want to allow BPF program override
the return of all the kernel functions in a KLP module.

I think the security problem is a big issue. Image that we have a KLP
in our environment. Any users can crash the kernel by hook a BPF
program on it with the calling of bpf_override_write().

What's more, this is a little weird for me. If we allow to use bpf_override_return()
for the kernel functions in a KLP, why not we allow it in a common kernel
module, as KLP is a kind of kernel module. Then, why not we allow to
use it for all the kernel functions?

Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in
your example? Then we can override its return directly. This is a more
reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that
anyone can modify the return of this function safely.

WDYT?

BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title
of this patch. Then, the BPF maintainers can notice this patch ;)

Thanks!
Menglong Dong

> 
>  static struct slave *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
>                                                  struct sk_buff *skb,
>                                                  struct bond_up_slave *slaves)
>  {
>         struct slave *slave;
>         unsigned int count;
> +       int slave_idx;
>         u32 hash;
> 
>         hash = bond_xmit_hash(bond, skb);
> @@ -5188,6 +5198,13 @@ static struct slave
> *bond_xmit_3ad_xor_slave_get(struct bonding *bond,
>         if (unlikely(!count))
>                 return NULL;
> 
> +       /* Try BPF hook first - returns slave index directly */
> +       slave_idx = bond_get_slave_hook(skb, hash, count);
> +       /* If BPF hook returned valid slave index, use it */
> +       if (slave_idx >= 0 && slave_idx < count) {
> +               slave = slaves->arr[slave_idx];
> +               return slave;
> +       }
>         slave = slaves->arr[hash % count];
>         return slave;
>  }
> 
> - The BPF program
> 
> SEC("kprobe/bond_get_slave_hook")
> int BPF_KPROBE(slave_selector, struct sk_buff *skb, u32 hash, u32 count)
> {
>         unsigned short net_hdr_off;
>         unsigned char *head;
>         struct iphdr iph;
>         int *slave_idx;
>         __u32 daddr;
> 
>         __u16 proto = BPF_CORE_READ(skb, protocol);
>         if (proto != bpf_htons(0x0800))
>                 return 0;
> 
>         head = BPF_CORE_READ(skb, head);
>         net_hdr_off = BPF_CORE_READ(skb, network_header);
> 
>         if (bpf_probe_read_kernel(&iph, sizeof(iph), head + net_hdr_off) != 0)
>                 return 0;
> 
>         daddr = iph.daddr;
>         slave_idx = bpf_map_lookup_elem(&ip_slave_map, &daddr);
>         if (slave_idx) {
>                 int idx = *slave_idx;
> 
>                 if (idx >= 0 && idx < (int)count)
>                         bpf_override_return(ctx, idx);
>         }
>         return 0;
> }
> 
> >
> > BTW, if we allow the usage of bpf_override_return() on the KLP patched
> > function, we should allow the usage of BPF_MODIFY_RETURN on this
> > case too, right?
> 
> It's a possibility, but I haven't tested that specifically yet.
> 
> -- 
> Regards
> Yafang

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Steven Rostedt]

On Fri, 03 Apr 2026 18:25:59 +0800
Menglong Dong <menglong.dong@linux.dev> wrote:

> I think the security problem is a big issue. Image that we have a KLP
> in our environment. Any users can crash the kernel by hook a BPF
> program on it with the calling of bpf_override_write().

Right, livepatching may allow for rapid experimentation but that is not its
purpose. It is for fixing production systems without having to reboot.
Using BPF to change the return of a function is a huge security issue.

> 
> What's more, this is a little weird for me. If we allow to use bpf_override_return()
> for the kernel functions in a KLP, why not we allow it in a common kernel
> module, as KLP is a kind of kernel module. Then, why not we allow to
> use it for all the kernel functions?

Right.

> 
> Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in
> your example? Then we can override its return directly. This is a more
> reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that
> anyone can modify the return of this function safely.

If this were to go in, I say it would require both a kernel config, with
a big warning about this being a security hole, and a kernel command line
option to enable it, so that people don't accidentally have it enabled in
their config.

The command line should be something like:

  allow_bpf_to_rootkit_functions

-- Steve

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Yafang Shao]

On Fri, Apr 3, 2026 at 7:29 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 03 Apr 2026 18:25:59 +0800
> Menglong Dong <menglong.dong@linux.dev> wrote:
>
> > I think the security problem is a big issue. Image that we have a KLP
> > in our environment. Any users can crash the kernel by hook a BPF
> > program on it with the calling of bpf_override_write().
>
> Right, livepatching may allow for rapid experimentation but that is not its
> purpose. It is for fixing production systems without having to reboot.
> Using BPF to change the return of a function is a huge security issue.
>
> >
> > What's more, this is a little weird for me. If we allow to use bpf_override_return()
> > for the kernel functions in a KLP, why not we allow it in a common kernel
> > module, as KLP is a kind of kernel module. Then, why not we allow to
> > use it for all the kernel functions?
>
> Right.
>
> >
> > Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in
> > your example? Then we can override its return directly. This is a more
> > reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that
> > anyone can modify the return of this function safely.
>
> If this were to go in, I say it would require both a kernel config, with
> a big warning about this being a security hole, and a kernel command line
> option to enable it, so that people don't accidentally have it enabled in
> their config.
>
> The command line should be something like:
>
>   allow_bpf_to_rootkit_functions

The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In
the next revision, I will rename this to
CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel
command-line parameter, allow_bpf_to_rootkit_functions, to control
it.

--
Regards

Yafang

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Alexei Starovoitov]

On Fri, Apr 3, 2026 at 6:31 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> >
> > If this were to go in, I say it would require both a kernel config, with
> > a big warning about this being a security hole, and a kernel command line
> > option to enable it, so that people don't accidentally have it enabled in
> > their config.
> >
> > The command line should be something like:
> >
> >   allow_bpf_to_rootkit_functions
>
> The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In
> the next revision, I will rename this to
> CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel
> command-line parameter, allow_bpf_to_rootkit_functions, to control
> it.

No. Even with extra config this is not ok.

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Yafang Shao]

On Fri, Apr 3, 2026 at 10:26 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Apr 3, 2026 at 6:31 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > >
> > > If this were to go in, I say it would require both a kernel config, with
> > > a big warning about this being a security hole, and a kernel command line
> > > option to enable it, so that people don't accidentally have it enabled in
> > > their config.
> > >
> > > The command line should be something like:
> > >
> > >   allow_bpf_to_rootkit_functions
> >
> > The feature is currently gated by CONFIG_KPROBE_OVERRIDE_KLP_FUNC. In
> > the next revision, I will rename this to
> > CONFIG_ALLOW_BPF_TO_ROOTKIT_FUNCS and introduce a corresponding kernel
> > command-line parameter, allow_bpf_to_rootkit_functions, to control
> > it.
>
> No. Even with extra config this is not ok.

I will send patch #3 and #4 as a standalone patchset to upstream the
hybrid livepatch first and then figure out how to move
allow_bpf_to_rootkit_functions forward.

--
Regards
Yafang

Re: [RFC PATCH 2/4] trace: Allow kprobes to override livepatched functions

[Yafang Shao]

On Fri, Apr 3, 2026 at 6:26 PM Menglong Dong <menglong.dong@linux.dev> wrote:
>
> On 2026/4/2 21:20 Yafang Shao <laoar.shao@gmail.com> write:
> > On Thu, Apr 2, 2026 at 8:48 PM Menglong Dong <menglong.dong@linux.dev> wrote:
> > >
> > > On 2026/4/2 17:26, Yafang Shao wrote:
> > > > Introduce the ability for kprobes to override the return values of
> > > > functions that have been livepatched. This functionality is guarded by the
> > > > CONFIG_KPROBE_OVERRIDE_KLP_FUNC configuration option.
> > >
> > > Hi, Yafang. This is a interesting idea.
> > >
> [...]
> >
> > +/* noclone to avoid bond_get_slave_hook.constprop.0 */
> > +__attribute__((__noclone__, __noinline__))
> > +int bond_get_slave_hook(struct sk_buff *skb, u32 hash, unsigned int count)
> > +{
> > +       return -1;
> > +}
>
> Hi, yafang.
>
> I see what you mean now. So you want to allow BPF program override
> the return of all the kernel functions in a KLP module.
>
> I think the security problem is a big issue. Image that we have a KLP
> in our environment. Any users can crash the kernel by hook a BPF
> program on it with the calling of bpf_override_write().

This feature is guarded by the CONFIG_KPROBE_OVERRIDE_KLP_FUNC
configuration option, which is disabled by default. Consequently, the
user must explicitly enable this option to utilize the feature.

>
> What's more, this is a little weird for me. If we allow to use bpf_override_return()
> for the kernel functions in a KLP, why not we allow it in a common kernel
> module, as KLP is a kind of kernel module. Then, why not we allow to
> use it for all the kernel functions?

By leveraging KLP, we can rapidly deploy new features without
interrupting production workloads. Accordingly, this feature is
specifically targeted at KLP-patched functions to maintain that
seamless delivery model.

>
> Can we mark the "bond_get_slave_hook" with ALLOW_ERROR_INJECTION() in
> your example? Then we can override its return directly. This is a more
> reasonable for me. With ALLOW_ERROR_INJECTION(), we are telling people that
> anyone can modify the return of this function safely.

It is unfortunate that ALLOW_ERROR_INJECTION() is incompatible with
KLP-patched functions, as this limits our ability to perform fault
injection on livepatched code

>
> WDYT?
>
> BTW, this is a BPF modification, so maybe we can use "bpf: xxx" for the title
> of this patch. Then, the BPF maintainers can notice this patch ;)

I agree with the suggestion. I will update the subject prefix to
"trace, bpf:" in the next version.

-- 
Regards
Yafang

[RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Yafang Shao]

Add a new replaceable attribute to allow the coexistence of both
atomic-replace and non-atomic-replace livepatches. If replaceable is set to
0, the livepatch will not be replaced by a subsequent atomic-replace
operation.

This is a preparatory patch for following changes.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 include/linux/livepatch.h |  2 ++
 kernel/livepatch/core.c   | 44 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/include/linux/livepatch.h b/include/linux/livepatch.h
index ba9e3988c07c..d88a6966e5f2 100644
--- a/include/linux/livepatch.h
+++ b/include/linux/livepatch.h
@@ -124,6 +124,7 @@ struct klp_state {
  * @objs:	object entries for kernel objects to be patched
  * @states:	system states that can get modified
  * @replace:	replace all actively used patches
+ * @replaceable:	whether this patch can be replaced or not
  * @list:	list node for global list of actively used patches
  * @kobj:	kobject for sysfs resources
  * @obj_list:	dynamic list of the object entries
@@ -138,6 +139,7 @@ struct klp_patch {
 	struct klp_object *objs;
 	struct klp_state *states;
 	bool replace;
+	bool replaceable;
 
 	/* internal */
 	struct list_head list;
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 28d15ba58a26..04f9e84f114f 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -351,6 +351,7 @@ int klp_apply_section_relocs(struct module *pmod, Elf_Shdr *sechdrs,
  * /sys/kernel/livepatch/<patch>/transition
  * /sys/kernel/livepatch/<patch>/force
  * /sys/kernel/livepatch/<patch>/replace
+ * /sys/kernel/livepatch/<patch>/replaceable
  * /sys/kernel/livepatch/<patch>/stack_order
  * /sys/kernel/livepatch/<patch>/<object>
  * /sys/kernel/livepatch/<patch>/<object>/patched
@@ -478,17 +479,60 @@ static ssize_t stack_order_show(struct kobject *kobj,
 	return sysfs_emit(buf, "%d\n", stack_order);
 }
 
+static ssize_t replaceable_store(struct kobject *kobj, struct kobj_attribute *attr,
+				const char *buf, size_t count)
+{
+	struct klp_patch *patch;
+	bool replaceable;
+	int ret;
+
+	ret = kstrtobool(buf, &replaceable);
+	if (ret)
+		return ret;
+
+	patch = container_of(kobj, struct klp_patch, kobj);
+
+	mutex_lock(&klp_mutex);
+
+	if (patch->replaceable == replaceable)
+		goto out;
+
+	if (patch == klp_transition_patch) {
+		ret = -EAGAIN;
+		goto out;
+	}
+
+	patch->replaceable = replaceable;
+
+out:
+	mutex_unlock(&klp_mutex);
+
+	if (ret)
+		return ret;
+	return count;
+}
+static ssize_t replaceable_show(struct kobject *kobj,
+			       struct kobj_attribute *attr, char *buf)
+{
+	struct klp_patch *patch;
+
+	patch = container_of(kobj, struct klp_patch, kobj);
+	return sysfs_emit(buf, "%d\n", patch->replaceable);
+}
+
 static struct kobj_attribute enabled_kobj_attr = __ATTR_RW(enabled);
 static struct kobj_attribute transition_kobj_attr = __ATTR_RO(transition);
 static struct kobj_attribute force_kobj_attr = __ATTR_WO(force);
 static struct kobj_attribute replace_kobj_attr = __ATTR_RO(replace);
 static struct kobj_attribute stack_order_kobj_attr = __ATTR_RO(stack_order);
+static struct kobj_attribute replaceable_kobj_attr = __ATTR_RW(replaceable);
 static struct attribute *klp_patch_attrs[] = {
 	&enabled_kobj_attr.attr,
 	&transition_kobj_attr.attr,
 	&force_kobj_attr.attr,
 	&replace_kobj_attr.attr,
 	&stack_order_kobj_attr.attr,
+	&replaceable_kobj_attr.attr,
 	NULL
 };
 ATTRIBUTE_GROUPS(klp_patch);
-- 
2.47.3

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Song Liu]

On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> Add a new replaceable attribute to allow the coexistence of both
> atomic-replace and non-atomic-replace livepatches. If replaceable is set to
> 0, the livepatch will not be replaced by a subsequent atomic-replace
> operation.
>
> This is a preparatory patch for following changes.

IIRC, the use case for this change is when multiple users load various
livepatch modules on the same system. I still don't believe this is the
right way to manage livepatches. That said, I won't really NACK this
if other folks think this is a useful option.

In case we really want a feature like this, shall we add a replaceable
flag to each function (klp_func)? This will give us fine granularity
control. For example, user A has a non-replaceable livepatch on
func_a; later user B wants to patch another version of func_a. Then
some logic can detect such a conflict and reject the load of user B's
livepatch. Does this make sense?

Thanks,
Song

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Dylan Hatch]

On Fri, Apr 3, 2026 at 9:19 AM Song Liu <song@kernel.org> wrote:
>
> On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > Add a new replaceable attribute to allow the coexistence of both
> > atomic-replace and non-atomic-replace livepatches. If replaceable is set to
> > 0, the livepatch will not be replaced by a subsequent atomic-replace
> > operation.
> >
> > This is a preparatory patch for following changes.
>
> IIRC, the use case for this change is when multiple users load various
> livepatch modules on the same system. I still don't believe this is the
> right way to manage livepatches. That said, I won't really NACK this
> if other folks think this is a useful option.

In our production fleet, we apply exactly one cumulative livepatch
module, and we use per-kernel build "livepatch release" branches to
track the contents of these cumulative livepatches. This model has
worked relatively well for us, but there are some painpoints.

We are often under pressure to selectively deploy a livepatch fix to
certain subpopulations of production. If the subpopulation is running
the same build of everything else, this would require us to introduce
another branching factor to the "livepatch release" branches --
something we do not support due to the added toil and complexity.

However, if we had the ability to build "off-band" livepatch modules
that were marked as non-replaceable, we could support these selective
patches without the additional branching factor. I will have to
circulate the idea internally, but to me this seems like a very useful
option to have in certain cases.

>
> In case we really want a feature like this, shall we add a replaceable
> flag to each function (klp_func)? This will give us fine granularity
> control. For example, user A has a non-replaceable livepatch on
> func_a; later user B wants to patch another version of func_a. Then
> some logic can detect such a conflict and reject the load of user B's
> livepatch. Does this make sense?

I agree with this. The ability to reject livepatches that try to
change functions that have already been patched seems like an
important safeguard.

Thanks,
Dylan

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Song Liu]

On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <dylanbhatch@google.com> wrote:
[...]
> > IIRC, the use case for this change is when multiple users load various
> > livepatch modules on the same system. I still don't believe this is the
> > right way to manage livepatches. That said, I won't really NACK this
> > if other folks think this is a useful option.
>
> In our production fleet, we apply exactly one cumulative livepatch
> module, and we use per-kernel build "livepatch release" branches to
> track the contents of these cumulative livepatches. This model has
> worked relatively well for us, but there are some painpoints.
>
> We are often under pressure to selectively deploy a livepatch fix to
> certain subpopulations of production. If the subpopulation is running
> the same build of everything else, this would require us to introduce
> another branching factor to the "livepatch release" branches --
> something we do not support due to the added toil and complexity.
>
> However, if we had the ability to build "off-band" livepatch modules
> that were marked as non-replaceable, we could support these selective
> patches without the additional branching factor. I will have to
> circulate the idea internally, but to me this seems like a very useful
> option to have in certain cases.

 IIUC, the plan is:

- The regular livepatches are cumulative, have the replace flag; and
  are replaceable.
- The occasional "off-band" livepatches do not have the replace flag,
  and are not replaceable.

With this setup, for systems with off-band livepatches loaded, we can
still release a cumulative livepatch to replace the previous cumulative
livepatch. Is this the expected use case?

I think there are a few issues with this:
1. The "off-band" livepatches cannot be replaced atomically. To upgrade
   "off-band' livepatches, we will have to unload the old version and load
   the new version later.
2. Any conflict with the off-band livepatches and regular livepatches will
   be difficult to manage. IOW, we kind removed the benefit of cumulative
   livepatches. For example, what shall we do if we really need two fixes
   to the same kernel functions: one from the original branch, the other
   from the off-band branch?

Thanks,
Song

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Yafang Shao]

On Sat, Apr 4, 2026 at 5:36 AM Song Liu <song@kernel.org> wrote:
>
> On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <dylanbhatch@google.com> wrote:
> [...]
> > > IIRC, the use case for this change is when multiple users load various
> > > livepatch modules on the same system. I still don't believe this is the
> > > right way to manage livepatches. That said, I won't really NACK this
> > > if other folks think this is a useful option.
> >
> > In our production fleet, we apply exactly one cumulative livepatch
> > module, and we use per-kernel build "livepatch release" branches to
> > track the contents of these cumulative livepatches. This model has
> > worked relatively well for us, but there are some painpoints.
> >
> > We are often under pressure to selectively deploy a livepatch fix to
> > certain subpopulations of production. If the subpopulation is running
> > the same build of everything else, this would require us to introduce
> > another branching factor to the "livepatch release" branches --
> > something we do not support due to the added toil and complexity.
> >
> > However, if we had the ability to build "off-band" livepatch modules
> > that were marked as non-replaceable, we could support these selective
> > patches without the additional branching factor. I will have to
> > circulate the idea internally, but to me this seems like a very useful
> > option to have in certain cases.
>
>  IIUC, the plan is:
>
> - The regular livepatches are cumulative, have the replace flag; and
>   are replaceable.
> - The occasional "off-band" livepatches do not have the replace flag,
>   and are not replaceable.
>
> With this setup, for systems with off-band livepatches loaded, we can
> still release a cumulative livepatch to replace the previous cumulative
> livepatch. Is this the expected use case?

That matches our expected use case.

>
> I think there are a few issues with this:
> 1. The "off-band" livepatches cannot be replaced atomically. To upgrade
>    "off-band' livepatches, we will have to unload the old version and load
>    the new version later.

Right. That is how the non-atomic-replace patch works.

> 2. Any conflict with the off-band livepatches and regular livepatches will
>    be difficult to manage.

We need to manage this conflict with a complex user script. That said,
everything can be controlled from userspace.

> IOW, we kind removed the benefit of cumulative
>    livepatches. For example, what shall we do if we really need two fixes
>    to the same kernel functions: one from the original branch, the other
>    from the off-band branch?

We run tens of livepatches on our production servers and have never
run into this issue. It's an extremely rare case — and if it does
happen, a user script should be able to handle it just fine.

-- 
Regards
Yafang

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Song Liu]

On Mon, Apr 6, 2026 at 4:08 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> On Sat, Apr 4, 2026 at 5:36 AM Song Liu <song@kernel.org> wrote:
> >
> > On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <dylanbhatch@google.com> wrote:
> > [...]
> > > > IIRC, the use case for this change is when multiple users load various
> > > > livepatch modules on the same system. I still don't believe this is the
> > > > right way to manage livepatches. That said, I won't really NACK this
> > > > if other folks think this is a useful option.
> > >
> > > In our production fleet, we apply exactly one cumulative livepatch
> > > module, and we use per-kernel build "livepatch release" branches to
> > > track the contents of these cumulative livepatches. This model has
> > > worked relatively well for us, but there are some painpoints.
> > >
> > > We are often under pressure to selectively deploy a livepatch fix to
> > > certain subpopulations of production. If the subpopulation is running
> > > the same build of everything else, this would require us to introduce
> > > another branching factor to the "livepatch release" branches --
> > > something we do not support due to the added toil and complexity.
> > >
> > > However, if we had the ability to build "off-band" livepatch modules
> > > that were marked as non-replaceable, we could support these selective
> > > patches without the additional branching factor. I will have to
> > > circulate the idea internally, but to me this seems like a very useful
> > > option to have in certain cases.
> >
> >  IIUC, the plan is:
> >
> > - The regular livepatches are cumulative, have the replace flag; and
> >   are replaceable.
> > - The occasional "off-band" livepatches do not have the replace flag,
> >   and are not replaceable.
> >
> > With this setup, for systems with off-band livepatches loaded, we can
> > still release a cumulative livepatch to replace the previous cumulative
> > livepatch. Is this the expected use case?
>
> That matches our expected use case.

If we really want to serve use cases like this, I think we can introduce
some replace tag concept: Each livepatch will have a tag, u32 number.
Newly loaded livepatch will only replace existing livepatch with the
same tag. We can even reuse the existing "bool replace" in klp_patch,
and make it u32: replace=0 means no replace; replace > 0 are the
replace tag.

For current users of cumulative patches, all the livepatch will have the
same tag, say 1. For your use case, you can assign each user a
unique tag. Then all these users can do atomic upgrades of their
own livepatches.

We may also need to check whether two livepatches of different tags
touch the same kernel function. When that happens, the later
livepatch should fail to load.

Does this make sense?

Thanks,
Song

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Joe Lawrence]

On Mon, Apr 06, 2026 at 11:11:27AM -0700, Song Liu wrote:
> On Mon, Apr 6, 2026 at 4:08 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > On Sat, Apr 4, 2026 at 5:36 AM Song Liu <song@kernel.org> wrote:
> > >
> > > On Fri, Apr 3, 2026 at 1:55 PM Dylan Hatch <dylanbhatch@google.com> wrote:
> > > [...]
> > > > > IIRC, the use case for this change is when multiple users load various
> > > > > livepatch modules on the same system. I still don't believe this is the
> > > > > right way to manage livepatches. That said, I won't really NACK this
> > > > > if other folks think this is a useful option.
> > > >
> > > > In our production fleet, we apply exactly one cumulative livepatch
> > > > module, and we use per-kernel build "livepatch release" branches to
> > > > track the contents of these cumulative livepatches. This model has
> > > > worked relatively well for us, but there are some painpoints.
> > > >
> > > > We are often under pressure to selectively deploy a livepatch fix to
> > > > certain subpopulations of production. If the subpopulation is running
> > > > the same build of everything else, this would require us to introduce
> > > > another branching factor to the "livepatch release" branches --
> > > > something we do not support due to the added toil and complexity.
> > > >
> > > > However, if we had the ability to build "off-band" livepatch modules
> > > > that were marked as non-replaceable, we could support these selective
> > > > patches without the additional branching factor. I will have to
> > > > circulate the idea internally, but to me this seems like a very useful
> > > > option to have in certain cases.
> > >
> > >  IIUC, the plan is:
> > >
> > > - The regular livepatches are cumulative, have the replace flag; and
> > >   are replaceable.
> > > - The occasional "off-band" livepatches do not have the replace flag,
> > >   and are not replaceable.
> > >
> > > With this setup, for systems with off-band livepatches loaded, we can
> > > still release a cumulative livepatch to replace the previous cumulative
> > > livepatch. Is this the expected use case?
> >
> > That matches our expected use case.
> 
> If we really want to serve use cases like this, I think we can introduce
> some replace tag concept: Each livepatch will have a tag, u32 number.
> Newly loaded livepatch will only replace existing livepatch with the
> same tag. We can even reuse the existing "bool replace" in klp_patch,
> and make it u32: replace=0 means no replace; replace > 0 are the
> replace tag.
> 
> For current users of cumulative patches, all the livepatch will have the
> same tag, say 1. For your use case, you can assign each user a
> unique tag. Then all these users can do atomic upgrades of their
> own livepatches.
> 
> We may also need to check whether two livepatches of different tags
> touch the same kernel function. When that happens, the later
> livepatch should fail to load.
> 
> Does this make sense?
> 

I haven't been following the thread carefully, but could the Livepatch
system state API (see Documentation/livepatch/system-state.rst) be
leveraged somehow instead of adding further replace semantics?

--
Joe

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Song Liu]

On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <joe.lawrence@redhat.com> wrote:
[...]
> > > > - The regular livepatches are cumulative, have the replace flag; and
> > > >   are replaceable.
> > > > - The occasional "off-band" livepatches do not have the replace flag,
> > > >   and are not replaceable.
> > > >
> > > > With this setup, for systems with off-band livepatches loaded, we can
> > > > still release a cumulative livepatch to replace the previous cumulative
> > > > livepatch. Is this the expected use case?
> > >
> > > That matches our expected use case.
> >
> > If we really want to serve use cases like this, I think we can introduce
> > some replace tag concept: Each livepatch will have a tag, u32 number.
> > Newly loaded livepatch will only replace existing livepatch with the
> > same tag. We can even reuse the existing "bool replace" in klp_patch,
> > and make it u32: replace=0 means no replace; replace > 0 are the
> > replace tag.
> >
> > For current users of cumulative patches, all the livepatch will have the
> > same tag, say 1. For your use case, you can assign each user a
> > unique tag. Then all these users can do atomic upgrades of their
> > own livepatches.
> >
> > We may also need to check whether two livepatches of different tags
> > touch the same kernel function. When that happens, the later
> > livepatch should fail to load.
> >
> > Does this make sense?
> >
>
> I haven't been following the thread carefully, but could the Livepatch
> system state API (see Documentation/livepatch/system-state.rst) be
> leveraged somehow instead of adding further replace semantics?

AFAICT, system state will not help Yafang's use case.

Thanks,
Song

Re: [RFC PATCH 3/4] livepatch: Add "replaceable" attribute to klp_patch

[Yafang Shao]

On Tue, Apr 7, 2026 at 10:54 AM Song Liu <song@kernel.org> wrote:
>
> On Mon, Apr 6, 2026 at 2:12 PM Joe Lawrence <joe.lawrence@redhat.com> wrote:
> [...]
> > > > > - The regular livepatches are cumulative, have the replace flag; and
> > > > >   are replaceable.
> > > > > - The occasional "off-band" livepatches do not have the replace flag,
> > > > >   and are not replaceable.
> > > > >
> > > > > With this setup, for systems with off-band livepatches loaded, we can
> > > > > still release a cumulative livepatch to replace the previous cumulative
> > > > > livepatch. Is this the expected use case?
> > > >
> > > > That matches our expected use case.
> > >
> > > If we really want to serve use cases like this, I think we can introduce
> > > some replace tag concept: Each livepatch will have a tag, u32 number.
> > > Newly loaded livepatch will only replace existing livepatch with the
> > > same tag. We can even reuse the existing "bool replace" in klp_patch,
> > > and make it u32: replace=0 means no replace; replace > 0 are the
> > > replace tag.
> > >
> > > For current users of cumulative patches, all the livepatch will have the
> > > same tag, say 1. For your use case, you can assign each user a
> > > unique tag. Then all these users can do atomic upgrades of their
> > > own livepatches.
> > >
> > > We may also need to check whether two livepatches of different tags
> > > touch the same kernel function. When that happens, the later
> > > livepatch should fail to load.

That sounds like a viable solution. I'll look into it and see how we
can implement it.

> > >
> > > Does this make sense?
> > >
> >
> > I haven't been following the thread carefully, but could the Livepatch
> > system state API (see Documentation/livepatch/system-state.rst) be
> > leveraged somehow instead of adding further replace semantics?
>
> AFAICT, system state will not help Yafang's use case.

Right.

-- 
Regards
Yafang

[RFC PATCH 4/4] livepatch: Implement livepatch hybrid mode

[Yafang Shao]

Livepatching allows for rapid experimentation with new kernel features
without interrupting production workloads. However, static livepatches lack
the flexibility required to tune features based on task-specific attributes,
such as cgroup membership, which is critical in multi-tenant k8s
environments. Furthermore, hardcoding logic into a livepatch prevents
dynamic adjustments based on the runtime environment.

To address this, we propose a hybrid approach using BPF. Our production use
case involves:

1. Deploying a Livepatch function to serve as a stable BPF hook.

2. Utilizing bpf_override_return() to dynamically modify the return value
   of that hook based on the current task's context.

A significant challenge arises when atomic-replace is enabled. In this
mode, deploying a new livepatch changes the target function's address,
forcing a re-attachment of the BPF program. This re-attachment latency is
unacceptable in critical paths, such as those handling networking policies.

To solve this, we introduce a hybrid livepatch mode that allows specific
patches to remain non-replaceable, ensuring the function address remains
stable and the BPF program stays attached.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 kernel/livepatch/core.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 04f9e84f114f..5a44154131c8 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -665,6 +665,8 @@ static int klp_add_nops(struct klp_patch *patch)
 		klp_for_each_object(old_patch, old_obj) {
 			int err;
 
+			if (!old_patch->replaceable)
+				continue;
 			err = klp_add_object_nops(patch, old_obj);
 			if (err)
 				return err;
@@ -837,6 +839,8 @@ void klp_free_replaced_patches_async(struct klp_patch *new_patch)
 	klp_for_each_patch_safe(old_patch, tmp_patch) {
 		if (old_patch == new_patch)
 			return;
+		if (!old_patch->replaceable)
+			continue;
 		klp_free_patch_async(old_patch);
 	}
 }
@@ -1239,6 +1243,8 @@ void klp_unpatch_replaced_patches(struct klp_patch *new_patch)
 		if (old_patch == new_patch)
 			return;
 
+		if (!old_patch->replaceable)
+			continue;
 		old_patch->enabled = false;
 		klp_unpatch_objects(old_patch);
 	}
-- 
2.47.3

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Song Liu]

Hi Yafang,

On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> Livepatching allows for rapid experimentation with new kernel features
> without interrupting production workloads. However, static livepatches lack
> the flexibility required to tune features based on task-specific attributes,
> such as cgroup membership, which is critical in multi-tenant k8s
> environments. Furthermore, hardcoding logic into a livepatch prevents
> dynamic adjustments based on the runtime environment.
>
> To address this, we propose a hybrid approach using BPF. Our production use
> case involves:
>
> 1. Deploying a Livepatch function to serve as a stable BPF hook.
>
> 2. Utilizing bpf_override_return() to dynamically modify the return value
>    of that hook based on the current task's context.

Could you please provide a specific use case that can benefit from this?
AFAICT, livepatch is more flexible but risky (may cause crash); while
BPF is safe, but less flexible. The combination you are proposing seems
to get the worse of the two sides. Maybe it can indeed get the benefit of
both sides in some cases, but I cannot think of such examples.

Thanks,
Song

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Yafang Shao]

On Sat, Apr 4, 2026 at 12:07 AM Song Liu <song@kernel.org> wrote:
>
> Hi Yafang,
>
> On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > Livepatching allows for rapid experimentation with new kernel features
> > without interrupting production workloads. However, static livepatches lack
> > the flexibility required to tune features based on task-specific attributes,
> > such as cgroup membership, which is critical in multi-tenant k8s
> > environments. Furthermore, hardcoding logic into a livepatch prevents
> > dynamic adjustments based on the runtime environment.
> >
> > To address this, we propose a hybrid approach using BPF. Our production use
> > case involves:
> >
> > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> >
> > 2. Utilizing bpf_override_return() to dynamically modify the return value
> >    of that hook based on the current task's context.
>
> Could you please provide a specific use case that can benefit from this?
> AFAICT, livepatch is more flexible but risky (may cause crash); while
> BPF is safe, but less flexible. The combination you are proposing seems
> to get the worse of the two sides. Maybe it can indeed get the benefit of
> both sides in some cases, but I cannot think of such examples.
>

Here is an example we recently deployed on our production servers:

  https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/

In one of our specific clusters, we needed to send BGP traffic out
through specific NICs based on the destination IP. To achieve this
without interrupting service, we live-patched
bond_xmit_3ad_xor_slave_get(), added a new hook called
bond_get_slave_hook(), and then ran a BPF program attached to that
hook to select the outgoing NIC from the SKB. This allowed us to
rapidly deploy the feature with zero downtime.

-- 
Regards
Yafang

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Song Liu]

On Mon, Apr 6, 2026 at 3:55 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> On Sat, Apr 4, 2026 at 12:07 AM Song Liu <song@kernel.org> wrote:
> >
> > Hi Yafang,
> >
> > On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > Livepatching allows for rapid experimentation with new kernel features
> > > without interrupting production workloads. However, static livepatches lack
> > > the flexibility required to tune features based on task-specific attributes,
> > > such as cgroup membership, which is critical in multi-tenant k8s
> > > environments. Furthermore, hardcoding logic into a livepatch prevents
> > > dynamic adjustments based on the runtime environment.
> > >
> > > To address this, we propose a hybrid approach using BPF. Our production use
> > > case involves:
> > >
> > > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> > >
> > > 2. Utilizing bpf_override_return() to dynamically modify the return value
> > >    of that hook based on the current task's context.
> >
> > Could you please provide a specific use case that can benefit from this?
> > AFAICT, livepatch is more flexible but risky (may cause crash); while
> > BPF is safe, but less flexible. The combination you are proposing seems
> > to get the worse of the two sides. Maybe it can indeed get the benefit of
> > both sides in some cases, but I cannot think of such examples.
> >
>
> Here is an example we recently deployed on our production servers:
>
>   https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
>
> In one of our specific clusters, we needed to send BGP traffic out
> through specific NICs based on the destination IP. To achieve this
> without interrupting service, we live-patched
> bond_xmit_3ad_xor_slave_get(), added a new hook called
> bond_get_slave_hook(), and then ran a BPF program attached to that
> hook to select the outgoing NIC from the SKB. This allowed us to
> rapidly deploy the feature with zero downtime.

I guess the idea here is: keep the risk part simple, and implement
it in module/livepatch, then use BPF for the flexible and programmable
part safe.

Can we use struct_ops instead of bpf_override_return for this case?
This should make the solution more flexible.

Thanks,
Song

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Yafang Shao]

On Tue, Apr 7, 2026 at 2:26 AM Song Liu <song@kernel.org> wrote:
>
> On Mon, Apr 6, 2026 at 3:55 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > On Sat, Apr 4, 2026 at 12:07 AM Song Liu <song@kernel.org> wrote:
> > >
> > > Hi Yafang,
> > >
> > > On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > >
> > > > Livepatching allows for rapid experimentation with new kernel features
> > > > without interrupting production workloads. However, static livepatches lack
> > > > the flexibility required to tune features based on task-specific attributes,
> > > > such as cgroup membership, which is critical in multi-tenant k8s
> > > > environments. Furthermore, hardcoding logic into a livepatch prevents
> > > > dynamic adjustments based on the runtime environment.
> > > >
> > > > To address this, we propose a hybrid approach using BPF. Our production use
> > > > case involves:
> > > >
> > > > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> > > >
> > > > 2. Utilizing bpf_override_return() to dynamically modify the return value
> > > >    of that hook based on the current task's context.
> > >
> > > Could you please provide a specific use case that can benefit from this?
> > > AFAICT, livepatch is more flexible but risky (may cause crash); while
> > > BPF is safe, but less flexible. The combination you are proposing seems
> > > to get the worse of the two sides. Maybe it can indeed get the benefit of
> > > both sides in some cases, but I cannot think of such examples.
> > >
> >
> > Here is an example we recently deployed on our production servers:
> >
> >   https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
> >
> > In one of our specific clusters, we needed to send BGP traffic out
> > through specific NICs based on the destination IP. To achieve this
> > without interrupting service, we live-patched
> > bond_xmit_3ad_xor_slave_get(), added a new hook called
> > bond_get_slave_hook(), and then ran a BPF program attached to that
> > hook to select the outgoing NIC from the SKB. This allowed us to
> > rapidly deploy the feature with zero downtime.
>
> I guess the idea here is: keep the risk part simple, and implement
> it in module/livepatch, then use BPF for the flexible and programmable
> part safe.

Right

>
> Can we use struct_ops instead of bpf_override_return for this case?
> This should make the solution more flexible.

Upstreaming struct_ops based BPF hooks is a challenging process, as
seen in these examples:

  https://lwn.net/Articles/1054030/
  https://lwn.net/Articles/1043548/

Even when successful, upstreaming can take a significant amount of
time—often longer than our production requirements allow. To bridge
this gap, we developed this livepatch+BPF solution. This allows us to
rapidly deploy new features without maintaining custom hooks in our
local kernel. Because these livepatch-based hooks are lightweight,
they minimize maintenance overhead and simplify kernel upgrades (e.g.,
from 6.1 to 6.18).

That said, we would still prefer to have our hooks accepted upstream
to eliminate the need for self-maintenance entirely.

-- 
Regards
Yafang

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Song Liu]

On Mon, Apr 6, 2026 at 7:22 PM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> On Tue, Apr 7, 2026 at 2:26 AM Song Liu <song@kernel.org> wrote:
> >
> > On Mon, Apr 6, 2026 at 3:55 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > On Sat, Apr 4, 2026 at 12:07 AM Song Liu <song@kernel.org> wrote:
> > > >
> > > > Hi Yafang,
> > > >
> > > > On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > > >
> > > > > Livepatching allows for rapid experimentation with new kernel features
> > > > > without interrupting production workloads. However, static livepatches lack
> > > > > the flexibility required to tune features based on task-specific attributes,
> > > > > such as cgroup membership, which is critical in multi-tenant k8s
> > > > > environments. Furthermore, hardcoding logic into a livepatch prevents
> > > > > dynamic adjustments based on the runtime environment.
> > > > >
> > > > > To address this, we propose a hybrid approach using BPF. Our production use
> > > > > case involves:
> > > > >
> > > > > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> > > > >
> > > > > 2. Utilizing bpf_override_return() to dynamically modify the return value
> > > > >    of that hook based on the current task's context.
> > > >
> > > > Could you please provide a specific use case that can benefit from this?
> > > > AFAICT, livepatch is more flexible but risky (may cause crash); while
> > > > BPF is safe, but less flexible. The combination you are proposing seems
> > > > to get the worse of the two sides. Maybe it can indeed get the benefit of
> > > > both sides in some cases, but I cannot think of such examples.
> > > >
> > >
> > > Here is an example we recently deployed on our production servers:
> > >
> > >   https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
> > >
> > > In one of our specific clusters, we needed to send BGP traffic out
> > > through specific NICs based on the destination IP. To achieve this
> > > without interrupting service, we live-patched
> > > bond_xmit_3ad_xor_slave_get(), added a new hook called
> > > bond_get_slave_hook(), and then ran a BPF program attached to that
> > > hook to select the outgoing NIC from the SKB. This allowed us to
> > > rapidly deploy the feature with zero downtime.
> >
> > I guess the idea here is: keep the risk part simple, and implement
> > it in module/livepatch, then use BPF for the flexible and programmable
> > part safe.
>
> Right
>
> >
> > Can we use struct_ops instead of bpf_override_return for this case?
> > This should make the solution more flexible.
>
> Upstreaming struct_ops based BPF hooks is a challenging process, as
> seen in these examples:
>
>   https://lwn.net/Articles/1054030/
>   https://lwn.net/Articles/1043548/
>
> Even when successful, upstreaming can take a significant amount of
> time—often longer than our production requirements allow. To bridge
> this gap, we developed this livepatch+BPF solution. This allows us to
> rapidly deploy new features without maintaining custom hooks in our
> local kernel. Because these livepatch-based hooks are lightweight,
> they minimize maintenance overhead and simplify kernel upgrades (e.g.,
> from 6.1 to 6.18).

I didn't mean upstream struct_ops.

We can define the struct_ops in an OOT kernel module. Then we
can attach BPF programs to the struct_ops. We may need
livepatch to connect the new struct_ops to original kernel logic.

I think kernel side of this solution is mostly available, but we may
need some work on the toolchain side.

Does this make sense?

Thanks,
Song

> That said, we would still prefer to have our hooks accepted upstream
> to eliminate the need for self-maintenance entirely.

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Yafang Shao]

On Tue, Apr 7, 2026 at 10:47 AM Song Liu <song@kernel.org> wrote:
>
> On Mon, Apr 6, 2026 at 7:22 PM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > On Tue, Apr 7, 2026 at 2:26 AM Song Liu <song@kernel.org> wrote:
> > >
> > > On Mon, Apr 6, 2026 at 3:55 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > >
> > > > On Sat, Apr 4, 2026 at 12:07 AM Song Liu <song@kernel.org> wrote:
> > > > >
> > > > > Hi Yafang,
> > > > >
> > > > > On Thu, Apr 2, 2026 at 2:26 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > > > >
> > > > > > Livepatching allows for rapid experimentation with new kernel features
> > > > > > without interrupting production workloads. However, static livepatches lack
> > > > > > the flexibility required to tune features based on task-specific attributes,
> > > > > > such as cgroup membership, which is critical in multi-tenant k8s
> > > > > > environments. Furthermore, hardcoding logic into a livepatch prevents
> > > > > > dynamic adjustments based on the runtime environment.
> > > > > >
> > > > > > To address this, we propose a hybrid approach using BPF. Our production use
> > > > > > case involves:
> > > > > >
> > > > > > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> > > > > >
> > > > > > 2. Utilizing bpf_override_return() to dynamically modify the return value
> > > > > >    of that hook based on the current task's context.
> > > > >
> > > > > Could you please provide a specific use case that can benefit from this?
> > > > > AFAICT, livepatch is more flexible but risky (may cause crash); while
> > > > > BPF is safe, but less flexible. The combination you are proposing seems
> > > > > to get the worse of the two sides. Maybe it can indeed get the benefit of
> > > > > both sides in some cases, but I cannot think of such examples.
> > > > >
> > > >
> > > > Here is an example we recently deployed on our production servers:
> > > >
> > > >   https://lore.kernel.org/bpf/CALOAHbDnNba_w_nWH3-S9GAXw0+VKuLTh1gy5hy9Yqgeo4C0iA@mail.gmail.com/
> > > >
> > > > In one of our specific clusters, we needed to send BGP traffic out
> > > > through specific NICs based on the destination IP. To achieve this
> > > > without interrupting service, we live-patched
> > > > bond_xmit_3ad_xor_slave_get(), added a new hook called
> > > > bond_get_slave_hook(), and then ran a BPF program attached to that
> > > > hook to select the outgoing NIC from the SKB. This allowed us to
> > > > rapidly deploy the feature with zero downtime.
> > >
> > > I guess the idea here is: keep the risk part simple, and implement
> > > it in module/livepatch, then use BPF for the flexible and programmable
> > > part safe.
> >
> > Right
> >
> > >
> > > Can we use struct_ops instead of bpf_override_return for this case?
> > > This should make the solution more flexible.
> >
> > Upstreaming struct_ops based BPF hooks is a challenging process, as
> > seen in these examples:
> >
> >   https://lwn.net/Articles/1054030/
> >   https://lwn.net/Articles/1043548/
> >
> > Even when successful, upstreaming can take a significant amount of
> > time—often longer than our production requirements allow. To bridge
> > this gap, we developed this livepatch+BPF solution. This allows us to
> > rapidly deploy new features without maintaining custom hooks in our
> > local kernel. Because these livepatch-based hooks are lightweight,
> > they minimize maintenance overhead and simplify kernel upgrades (e.g.,
> > from 6.1 to 6.18).
>
> I didn't mean upstream struct_ops.
>
> We can define the struct_ops in an OOT kernel module. Then we
> can attach BPF programs to the struct_ops. We may need
> livepatch to connect the new struct_ops to original kernel logic.
>
> I think kernel side of this solution is mostly available, but we may
> need some work on the toolchain side.
>
> Does this make sense?

Are there actual benefits to using struct_ops instead of
bpf_override_return? So far, I’ve only found it adds complexity
without much gain.
Can we add something like ALLOW_LIVEPATCH_ERROR_INJECTION() to allow
error injection on functions defined inside a livepatch?

-- 
Regards
Yafang

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Christoph Hellwig]

On Thu, Apr 02, 2026 at 05:26:03PM +0800, Yafang Shao wrote:
> Livepatching allows for rapid experimentation with new kernel features
> without interrupting production workloads.

Myabe it allows, or based on the rest of the mail not quite.  But that
is certainly not the intent at all, the intent is to fix critical
bugs without downtime.

> However, static livepatches lack
> the flexibility required to tune features based on task-specific attributes,
> such as cgroup membership, which is critical in multi-tenant k8s
> environments. Furthermore, hardcoding logic into a livepatch prevents
> dynamic adjustments based on the runtime environment.
> 
> To address this, we propose a hybrid approach using BPF. Our production use
> case involves:
> 
> 1. Deploying a Livepatch function to serve as a stable BPF hook.
> 
> 2. Utilizing bpf_override_return() to dynamically modify the return value
>    of that hook based on the current task's context.

Whol f**. now.  Is this a delayed April 1st post?

Re: [RFC PATCH 0/4] trace, livepatch: Allow kprobe return overriding for livepatched functions

[Yafang Shao]

On Mon, Apr 6, 2026 at 1:36 PM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Thu, Apr 02, 2026 at 05:26:03PM +0800, Yafang Shao wrote:
> > Livepatching allows for rapid experimentation with new kernel features
> > without interrupting production workloads.
>
> Myabe it allows, or based on the rest of the mail not quite.  But that
> is certainly not the intent at all, the intent is to fix critical
> bugs without downtime.
>
> > However, static livepatches lack
> > the flexibility required to tune features based on task-specific attributes,
> > such as cgroup membership, which is critical in multi-tenant k8s
> > environments. Furthermore, hardcoding logic into a livepatch prevents
> > dynamic adjustments based on the runtime environment.
> >
> > To address this, we propose a hybrid approach using BPF. Our production use
> > case involves:
> >
> > 1. Deploying a Livepatch function to serve as a stable BPF hook.
> >
> > 2. Utilizing bpf_override_return() to dynamically modify the return value
> >    of that hook based on the current task's context.
>
> Whol f**. now.  Is this a delayed April 1st post?

You're already in my spam list. Don't expect any further replies. Feel
free to keep your verbose rubbish to yourself.

-- 
Regards
Yafang