* [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines
@ 2025-09-23 21:51 Jiri Olsa
2025-09-23 21:51 ` [PATCH 1/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Jiri Olsa
` (8 more replies)
0 siblings, 9 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
hi,
while poking the multi-tracing interface I ended up with just one ftrace_ops
object to attach all trampolines.
This change allows to use less direct API calls during the attachment changes
in the future code, so in effect speeding up the attachment.
In current code we get a speed up from using just a single ftrace_ops object.
I got following speed up when measuring simple attach/detach 300 times [1].
- with current code:
perf stat -e cycles:k,cycles:u,instructions:u,instructions:k -- ./test_progs -t krava -w0
#158 krava:OK
Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
Performance counter stats for './test_progs -t krava -w0':
12,003,420,519 cycles:k
63,239,794 cycles:u
102,155,625 instructions:u # 1.62 insn per cycle
11,614,183,764 instructions:k # 0.97 insn per cycle
35.448142362 seconds time elapsed
0.011032000 seconds user
2.478243000 seconds sys
- with the fix:
perf stat -e cycles:k,cycles:u,instructions:u,instructions:k -- ./test_progs -t krava -w0
#158 krava:OK
Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED
Performance counter stats for './test_progs -t krava -w0':
14,327,218,656 cycles:k
46,285,275 cycles:u
102,125,592 instructions:u # 2.21 insn per cycle
14,620,692,457 instructions:k # 1.02 insn per cycle
2.860883990 seconds time elapsed
0.009884000 seconds user
2.777032000 seconds sys
The speedup seems to be related to the fact that with single ftrace_ops object
we don't call ftrace_shutdown anymore (we use ftrace_update_ops instead) and
we skip the 300 synchronize rcu calls (each ~100ms) at the end of that function.
v1 changes:
- make the change x86 specific, after discussing with Mark options for
arm64 [Mark]
thanks,
jirka
[1] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=test&id=1b7fc74c36a93e90816f58c37a84522f0949095a
---
Jiri Olsa (9):
ftrace: Make alloc_and_copy_ftrace_hash direct friendly
ftrace: Add register_ftrace_direct_hash function
ftrace: Add unregister_ftrace_direct_hash function
ftrace: Add modify_ftrace_direct_hash function
ftrace: Export some of hash related functions
ftrace: Use direct hash interface in direct functions
bpf: Add trampoline ip hash table
ftrace: Factor ftrace_ops ops_func interface
bpf, x86: Use single ftrace_ops for direct calls
arch/x86/Kconfig | 1 +
include/linux/bpf.h | 7 +-
include/linux/ftrace.h | 48 ++++++++++---
kernel/bpf/trampoline.c | 128 +++++++++++++++++++++++++--------
kernel/trace/Kconfig | 3 +
kernel/trace/ftrace.c | 477 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------
kernel/trace/trace.h | 8 ---
kernel/trace/trace_selftest.c | 5 +-
8 files changed, 447 insertions(+), 230 deletions(-)
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function Jiri Olsa
` (7 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Make alloc_and_copy_ftrace_hash to copy also direct address
for each hash entry.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
kernel/trace/ftrace.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index a69067367c29..a45556257963 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1186,7 +1186,7 @@ static void __add_hash_entry(struct ftrace_hash *hash,
}
static struct ftrace_func_entry *
-add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
+add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned long direct)
{
struct ftrace_func_entry *entry;
@@ -1195,11 +1195,18 @@ add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
return NULL;
entry->ip = ip;
+ entry->direct = direct;
__add_hash_entry(hash, entry);
return entry;
}
+static struct ftrace_func_entry *
+add_hash_entry(struct ftrace_hash *hash, unsigned long ip)
+{
+ return add_hash_entry_direct(hash, ip, 0);
+}
+
static void
free_hash_entry(struct ftrace_hash *hash,
struct ftrace_func_entry *entry)
@@ -1372,7 +1379,7 @@ alloc_and_copy_ftrace_hash(int size_bits, struct ftrace_hash *hash)
size = 1 << hash->size_bits;
for (i = 0; i < size; i++) {
hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
- if (add_hash_entry(new_hash, entry->ip) == NULL)
+ if (add_hash_entry_direct(new_hash, entry->ip, entry->direct) == NULL)
goto free_hash;
}
}
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
2025-09-23 21:51 ` [PATCH 1/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-24 9:04 ` Steven Rostedt
2025-09-23 21:51 ` [PATCH 3/9] ftrace: Add unregister_ftrace_direct_hash function Jiri Olsa
` (6 subsequent siblings)
8 siblings, 1 reply; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Adding register_ftrace_direct_hash function that registers
all entries (ip -> direct) provided in hash argument.
The difference to current register_ftrace_direct is
- hash argument that allows to register multiple ip -> direct
entries at once
- we can call register_ftrace_direct_hash multiple times on the
same ftrace_ops object, becase after first registration with
register_ftrace_function_nolock, it uses ftrace_update_ops to
update the ftrace_ops object
This change will allow us to have simple ftrace_ops for all bpf
direct interface users in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 7 +++
kernel/trace/ftrace.c | 110 +++++++++++++++++++++++++++++++++++++++++
2 files changed, 117 insertions(+)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 7ded7df6e9b5..2705c292341a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -526,6 +526,8 @@ int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
+int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+
void ftrace_stub_direct_tramp(void);
#else
@@ -552,6 +554,11 @@ static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned l
return -ENODEV;
}
+int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+ return -ENODEV;
+}
+
/*
* This must be implemented by the architecture.
* It is the way the ftrace direct_ops helper, when called
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index a45556257963..06528af2281f 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6219,6 +6219,116 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
return err;
}
EXPORT_SYMBOL_GPL(modify_ftrace_direct);
+
+static unsigned long hash_count(struct ftrace_hash *hash)
+{
+ return hash ? hash->count : 0;
+}
+
+/**
+ * hash_add - adds two struct ftrace_hash and returns the result
+ * @a: struct ftrace_hash object
+ * @b: struct ftrace_hash object
+ *
+ * Returns struct ftrace_hash object on success, NULL on error.
+ */
+static struct ftrace_hash *hash_add(struct ftrace_hash *a, struct ftrace_hash *b)
+{
+ struct ftrace_func_entry *entry;
+ struct ftrace_hash *add;
+ int size, i;
+
+ size = hash_count(a) + hash_count(b);
+ if (size > 32)
+ size = 32;
+
+ add = alloc_and_copy_ftrace_hash(fls(size), a);
+ if (!add)
+ goto error;
+
+ size = 1 << b->size_bits;
+ for (i = 0; i < size; i++) {
+ hlist_for_each_entry(entry, &b->buckets[i], hlist) {
+ if (add_hash_entry_direct(add, entry->ip, entry->direct) == NULL)
+ goto error;
+ }
+ }
+ return add;
+
+ error:
+ free_ftrace_hash(add);
+ return NULL;
+}
+
+int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+ struct ftrace_hash *filter_hash = NULL, *new_hash = NULL, *free_hash = NULL;
+ struct ftrace_func_entry *entry;
+ int i, size, err = -EINVAL;
+ bool reg;
+
+ if (!hash_count(hash))
+ return 0;
+
+ mutex_lock(&direct_mutex);
+
+ /* Make sure requested entries are not already registered. */
+ size = 1 << hash->size_bits;
+ for (i = 0; i < size; i++) {
+ hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
+ if (__ftrace_lookup_ip(direct_functions, entry->ip))
+ goto out_unlock;
+ }
+ }
+
+ filter_hash = ops->func_hash ? ops->func_hash->filter_hash : NULL;
+
+ /* If there's nothing in filter_hash we need to register the ops. */
+ reg = hash_count(filter_hash) == 0;
+ if (reg) {
+ if (ops->func || ops->trampoline)
+ goto out_unlock;
+ if (ops->flags & FTRACE_OPS_FL_ENABLED)
+ goto out_unlock;
+ }
+
+ err = -ENOMEM;
+ filter_hash = hash_add(filter_hash, hash);
+ if (!filter_hash)
+ goto out_unlock;
+
+ new_hash = hash_add(direct_functions, hash);
+ if (!new_hash)
+ goto out_unlock;
+
+ free_hash = direct_functions;
+ rcu_assign_pointer(direct_functions, new_hash);
+
+ if (reg) {
+ ops->func = call_direct_funcs;
+ ops->flags = MULTI_FLAGS;
+ ops->trampoline = FTRACE_REGS_ADDR;
+ ops->local_hash.filter_hash = filter_hash;
+
+ err = register_ftrace_function_nolock(ops);
+ if (!err)
+ filter_hash = NULL;
+ } else {
+ err = ftrace_update_ops(ops, filter_hash, EMPTY_HASH);
+ }
+
+ out_unlock:
+ mutex_unlock(&direct_mutex);
+
+ if (free_hash && free_hash != EMPTY_HASH)
+ call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
+ if (filter_hash)
+ free_ftrace_hash(filter_hash);
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(register_ftrace_direct_hash);
+
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 3/9] ftrace: Add unregister_ftrace_direct_hash function
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
2025-09-23 21:51 ` [PATCH 1/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Jiri Olsa
2025-09-23 21:51 ` [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 4/9] ftrace: Add modify_ftrace_direct_hash function Jiri Olsa
` (5 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Adding unregister_ftrace_direct_hash function that unregisters
all entries (ip -> direct) provided in hash argument.
The difference to current unregister_ftrace_direct is
- hash argument that allows to unregister multiple ip -> direct
entries at once
- we can call unregister_ftrace_direct_hash multiple times on the
same ftrace_ops object, becase we do not need to unregister
all entries at once, we can do it gradualy with the help of
ftrace_update_ops function
This change will allow us to have simple ftrace_ops for all bpf
direct interface users in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 6 +++
kernel/trace/ftrace.c | 98 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 104 insertions(+)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 2705c292341a..55f5ead5d4ff 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -527,6 +527,7 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
void ftrace_stub_direct_tramp(void);
@@ -559,6 +560,11 @@ int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash
return -ENODEV;
}
+int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+ return -ENODEV;
+}
+
/*
* This must be implemented by the architecture.
* It is the way the ftrace direct_ops helper, when called
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 06528af2281f..ab5739f72933 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6329,6 +6329,104 @@ int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash
}
EXPORT_SYMBOL_GPL(register_ftrace_direct_hash);
+/**
+ * hash_sub - substracts @b from @a and returns the result
+ * @a: struct ftrace_hash object
+ * @b: struct ftrace_hash object
+ *
+ * Returns struct ftrace_hash object on success, NULL on error.
+ */
+static struct ftrace_hash *hash_sub(struct ftrace_hash *a, struct ftrace_hash *b)
+{
+ struct ftrace_func_entry *entry, *del;
+ struct ftrace_hash *sub;
+ int size, i;
+
+ sub = alloc_and_copy_ftrace_hash(a->size_bits, a);
+ if (!sub)
+ goto error;
+
+ size = 1 << b->size_bits;
+ for (i = 0; i < size; i++) {
+ hlist_for_each_entry(entry, &b->buckets[i], hlist) {
+ del = __ftrace_lookup_ip(sub, entry->ip);
+ if (WARN_ON_ONCE(!del))
+ goto error;
+ remove_hash_entry(sub, del);
+ kfree(del);
+ }
+ }
+ return sub;
+
+ error:
+ free_ftrace_hash(sub);
+ return NULL;
+}
+
+int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
+{
+ struct ftrace_hash *new_hash = NULL, *filter_hash = NULL, *free_hash = NULL;
+ struct ftrace_func_entry *del, *entry;
+ unsigned long size, i;
+ int err = -EINVAL;
+
+ if (!hash_count(hash))
+ return 0;
+ if (check_direct_multi(ops))
+ return -EINVAL;
+ if (!(ops->flags & FTRACE_OPS_FL_ENABLED))
+ return -EINVAL;
+ if (direct_functions == EMPTY_HASH)
+ return -EINVAL;
+
+ mutex_lock(&direct_mutex);
+
+ /* Make sure requested entries are already registered. */
+ size = 1 << hash->size_bits;
+ for (i = 0; i < size; i++) {
+ hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
+ del = __ftrace_lookup_ip(direct_functions, entry->ip);
+ if (!del || del->direct != entry->direct)
+ goto out_unlock;
+ }
+ }
+
+ err = -ENOMEM;
+ filter_hash = hash_sub(ops->func_hash->filter_hash, hash);
+ if (!filter_hash)
+ goto out_unlock;
+
+ new_hash = hash_sub(direct_functions, hash);
+ if (!new_hash)
+ goto out_unlock;
+
+ /* If there's nothing left, we need to unregister the ops. */
+ if (ftrace_hash_empty(filter_hash)) {
+ err = unregister_ftrace_function(ops);
+ /* cleanup for possible another register call */
+ ops->func = NULL;
+ ops->trampoline = 0;
+ ftrace_free_filter(ops);
+ ops->func_hash->filter_hash = NULL;
+ } else {
+ err = ftrace_update_ops(ops, filter_hash, EMPTY_HASH);
+ }
+
+ free_hash = direct_functions;
+ rcu_assign_pointer(direct_functions, new_hash);
+
+ out_unlock:
+ mutex_unlock(&direct_mutex);
+
+ if (free_hash && free_hash != EMPTY_HASH)
+ call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
+ if (filter_hash)
+ free_ftrace_hash(filter_hash);
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash);
+
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 4/9] ftrace: Add modify_ftrace_direct_hash function
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (2 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 3/9] ftrace: Add unregister_ftrace_direct_hash function Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 5/9] ftrace: Export some of hash related functions Jiri Olsa
` (4 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Adding modify_ftrace_direct_hash function that modifies
all entries (ip -> direct) provided in hash argument.
The difference to current unregister_ftrace_direct is
- hash argument that allows to modify multiple ip -> direct
entries at once
This change will allow us to have simple ftrace_ops for all bpf
direct interface users in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 6 +++++
kernel/trace/ftrace.c | 58 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 64 insertions(+)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 55f5ead5d4ff..1cf6d0bb9a81 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -528,6 +528,7 @@ int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
+int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock);
void ftrace_stub_direct_tramp(void);
@@ -565,6 +566,11 @@ int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *ha
return -ENODEV;
}
+int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
+{
+ return -ENODEV;
+}
+
/*
* This must be implemented by the architecture.
* It is the way the ftrace direct_ops helper, when called
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index ab5739f72933..4eb08817e4ee 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -6427,6 +6427,64 @@ int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *ha
}
EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash);
+int modify_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash, bool do_direct_lock)
+{
+ struct ftrace_func_entry *entry, *tmp;
+ static struct ftrace_ops tmp_ops = {
+ .func = ftrace_stub,
+ .flags = FTRACE_OPS_FL_STUB,
+ };
+ unsigned long size, i;
+ int err;
+
+ if (!hash_count(hash))
+ return 0;
+ if (check_direct_multi(ops))
+ return -EINVAL;
+ if (!(ops->flags & FTRACE_OPS_FL_ENABLED))
+ return -EINVAL;
+ if (direct_functions == EMPTY_HASH)
+ return -EINVAL;
+
+ if (do_direct_lock)
+ mutex_lock(&direct_mutex);
+
+ /* Enable the tmp_ops to have the same functions as the direct ops */
+ ftrace_ops_init(&tmp_ops);
+ tmp_ops.func_hash = ops->func_hash;
+
+ err = register_ftrace_function_nolock(&tmp_ops);
+ if (err)
+ goto unlock;
+
+ /*
+ * Now the ftrace_ops_list_func() is called to do the direct callers.
+ * We can safely change the direct functions attached to each entry.
+ */
+ mutex_lock(&ftrace_lock);
+
+ size = 1 << hash->size_bits;
+ for (i = 0; i < size; i++) {
+ hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
+ tmp = __ftrace_lookup_ip(direct_functions, entry->ip);
+ if (!tmp)
+ continue;
+ tmp->direct = entry->direct;
+ }
+ }
+
+ mutex_unlock(&ftrace_lock);
+
+ /* Removing the tmp_ops will add the updated direct callers to the functions */
+ unregister_ftrace_function(&tmp_ops);
+
+unlock:
+ if (do_direct_lock)
+ mutex_unlock(&direct_mutex);
+ return err;
+}
+EXPORT_SYMBOL_GPL(modify_ftrace_direct_hash);
+
#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
/**
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 5/9] ftrace: Export some of hash related functions
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (3 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 4/9] ftrace: Add modify_ftrace_direct_hash function Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 6/9] ftrace: Use direct hash interface in direct functions Jiri Olsa
` (3 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
We are going to use these functions in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 16 ++++++++++++++++
kernel/trace/ftrace.c | 7 +++----
kernel/trace/trace.h | 8 --------
3 files changed, 19 insertions(+), 12 deletions(-)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 1cf6d0bb9a81..a8e351df3a52 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -397,6 +397,22 @@ enum ftrace_ops_cmd {
typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cmd cmd);
#ifdef CONFIG_DYNAMIC_FTRACE
+
+#define FTRACE_HASH_DEFAULT_BITS 10
+
+struct ftrace_hash {
+ unsigned long size_bits;
+ struct hlist_head *buckets;
+ unsigned long count;
+ unsigned long flags;
+ struct rcu_head rcu;
+};
+
+struct ftrace_hash *alloc_ftrace_hash(int size_bits);
+void free_ftrace_hash(struct ftrace_hash *hash);
+struct ftrace_func_entry *add_hash_entry_direct(struct ftrace_hash *hash,
+ unsigned long ip, unsigned long direct);
+
/* The hash used to know what functions callbacks trace */
struct ftrace_ops_hash {
struct ftrace_hash __rcu *notrace_hash;
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 4eb08817e4ee..75bea44e4f8e 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -68,7 +68,6 @@
})
/* hash bits for specific function selection */
-#define FTRACE_HASH_DEFAULT_BITS 10
#define FTRACE_HASH_MAX_BITS 12
#ifdef CONFIG_DYNAMIC_FTRACE
@@ -1185,7 +1184,7 @@ static void __add_hash_entry(struct ftrace_hash *hash,
hash->count++;
}
-static struct ftrace_func_entry *
+struct ftrace_func_entry *
add_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigned long direct)
{
struct ftrace_func_entry *entry;
@@ -1265,7 +1264,7 @@ static void clear_ftrace_mod_list(struct list_head *head)
mutex_unlock(&ftrace_lock);
}
-static void free_ftrace_hash(struct ftrace_hash *hash)
+void free_ftrace_hash(struct ftrace_hash *hash)
{
if (!hash || hash == EMPTY_HASH)
return;
@@ -1305,7 +1304,7 @@ void ftrace_free_filter(struct ftrace_ops *ops)
}
EXPORT_SYMBOL_GPL(ftrace_free_filter);
-static struct ftrace_hash *alloc_ftrace_hash(int size_bits)
+struct ftrace_hash *alloc_ftrace_hash(int size_bits)
{
struct ftrace_hash *hash;
int size;
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 5f4bed5842f9..5070aafa590c 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -899,14 +899,6 @@ enum {
FTRACE_HASH_FL_MOD = (1 << 0),
};
-struct ftrace_hash {
- unsigned long size_bits;
- struct hlist_head *buckets;
- unsigned long count;
- unsigned long flags;
- struct rcu_head rcu;
-};
-
struct ftrace_func_entry *
ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip);
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 6/9] ftrace: Use direct hash interface in direct functions
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (4 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 5/9] ftrace: Export some of hash related functions Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 7/9] bpf: Add trampoline ip hash table Jiri Olsa
` (2 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Implement current *_ftrace_direct function with their *_hash
function counterparts.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 17 +--
kernel/bpf/trampoline.c | 10 +-
kernel/trace/ftrace.c | 232 +++++-----------------------------
kernel/trace/trace_selftest.c | 5 +-
4 files changed, 45 insertions(+), 219 deletions(-)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index a8e351df3a52..42da4dd08759 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -536,11 +536,10 @@ struct ftrace_func_entry {
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
unsigned long ftrace_find_rec_direct(unsigned long ip);
-int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
-int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
+int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr);
+int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr,
bool free_filters);
-int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr);
-int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr);
+int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr, bool lock_direct_mutex);
int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
@@ -554,20 +553,16 @@ static inline unsigned long ftrace_find_rec_direct(unsigned long ip)
{
return 0;
}
-static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
+static inline int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr)
{
return -ENODEV;
}
-static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
+static inline int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr,
bool free_filters)
{
return -ENODEV;
}
-static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
-{
- return -ENODEV;
-}
-static inline int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr)
+static inline int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr, bool lock_direct_mutex)
{
return -ENODEV;
}
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 5949095e51c3..97df6e482bf4 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -181,7 +181,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr)
int ret;
if (tr->func.ftrace_managed)
- ret = unregister_ftrace_direct(tr->fops, (long)old_addr, false);
+ ret = unregister_ftrace_direct(tr->fops, (unsigned long) ip, (long)old_addr, false);
else
ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, NULL);
@@ -195,10 +195,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad
int ret;
if (tr->func.ftrace_managed) {
- if (lock_direct_mutex)
- ret = modify_ftrace_direct(tr->fops, (long)new_addr);
- else
- ret = modify_ftrace_direct_nolock(tr->fops, (long)new_addr);
+ ret = modify_ftrace_direct(tr->fops, (unsigned long) ip, (long)new_addr, lock_direct_mutex);
} else {
ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, old_addr, new_addr);
}
@@ -220,8 +217,7 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr)
}
if (tr->func.ftrace_managed) {
- ftrace_set_filter_ip(tr->fops, (unsigned long)ip, 0, 1);
- ret = register_ftrace_direct(tr->fops, (long)new_addr);
+ ret = register_ftrace_direct(tr->fops, (unsigned long)ip, (long)new_addr);
} else {
ret = bpf_arch_text_poke(ip, BPF_MOD_CALL, NULL, new_addr);
}
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 75bea44e4f8e..6e9d5975477d 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -5935,28 +5935,24 @@ static int check_direct_multi(struct ftrace_ops *ops)
return 0;
}
-static void remove_direct_functions_hash(struct ftrace_hash *hash, unsigned long addr)
+static void register_ftrace_direct_cb(struct rcu_head *rhp)
{
- struct ftrace_func_entry *entry, *del;
- int size, i;
+ struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
- size = 1 << hash->size_bits;
- for (i = 0; i < size; i++) {
- hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
- del = __ftrace_lookup_ip(direct_functions, entry->ip);
- if (del && del->direct == addr) {
- remove_hash_entry(direct_functions, del);
- kfree(del);
- }
- }
- }
+ free_ftrace_hash(fhp);
}
-static void register_ftrace_direct_cb(struct rcu_head *rhp)
+static struct ftrace_hash *hash_from_ip(unsigned long ip, unsigned long addr)
{
- struct ftrace_hash *fhp = container_of(rhp, struct ftrace_hash, rcu);
+ struct ftrace_hash *hash;
- free_ftrace_hash(fhp);
+ ip = ftrace_location(ip);
+ if (!ip)
+ return NULL;
+ hash = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS);
+ if (!hash || !add_hash_entry_direct(hash, ip, addr))
+ return NULL;
+ return hash;
}
/**
@@ -5981,89 +5977,17 @@ static void register_ftrace_direct_cb(struct rcu_head *rhp)
* -ENODEV - @ip does not point to a ftrace nop location (or not supported)
* -ENOMEM - There was an allocation failure.
*/
-int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
+int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr)
{
- struct ftrace_hash *hash, *new_hash = NULL, *free_hash = NULL;
- struct ftrace_func_entry *entry, *new;
- int err = -EBUSY, size, i;
-
- if (ops->func || ops->trampoline)
- return -EINVAL;
- if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED))
- return -EINVAL;
- if (ops->flags & FTRACE_OPS_FL_ENABLED)
- return -EINVAL;
-
- hash = ops->func_hash->filter_hash;
- if (ftrace_hash_empty(hash))
- return -EINVAL;
-
- mutex_lock(&direct_mutex);
-
- /* Make sure requested entries are not already registered.. */
- size = 1 << hash->size_bits;
- for (i = 0; i < size; i++) {
- hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
- if (ftrace_find_rec_direct(entry->ip))
- goto out_unlock;
- }
- }
-
- err = -ENOMEM;
-
- /* Make a copy hash to place the new and the old entries in */
- size = hash->count + direct_functions->count;
- size = fls(size);
- if (size > FTRACE_HASH_MAX_BITS)
- size = FTRACE_HASH_MAX_BITS;
- new_hash = alloc_ftrace_hash(size);
- if (!new_hash)
- goto out_unlock;
-
- /* Now copy over the existing direct entries */
- size = 1 << direct_functions->size_bits;
- for (i = 0; i < size; i++) {
- hlist_for_each_entry(entry, &direct_functions->buckets[i], hlist) {
- new = add_hash_entry(new_hash, entry->ip);
- if (!new)
- goto out_unlock;
- new->direct = entry->direct;
- }
- }
-
- /* ... and add the new entries */
- size = 1 << hash->size_bits;
- for (i = 0; i < size; i++) {
- hlist_for_each_entry(entry, &hash->buckets[i], hlist) {
- new = add_hash_entry(new_hash, entry->ip);
- if (!new)
- goto out_unlock;
- /* Update both the copy and the hash entry */
- new->direct = addr;
- entry->direct = addr;
- }
- }
-
- free_hash = direct_functions;
- rcu_assign_pointer(direct_functions, new_hash);
- new_hash = NULL;
-
- ops->func = call_direct_funcs;
- ops->flags = MULTI_FLAGS;
- ops->trampoline = FTRACE_REGS_ADDR;
- ops->direct_call = addr;
-
- err = register_ftrace_function_nolock(ops);
-
- out_unlock:
- mutex_unlock(&direct_mutex);
-
- if (free_hash && free_hash != EMPTY_HASH)
- call_rcu_tasks(&free_hash->rcu, register_ftrace_direct_cb);
+ struct ftrace_hash *hash;
+ int err;
- if (new_hash)
- free_ftrace_hash(new_hash);
+ hash = hash_from_ip(ip, addr);
+ if (!hash)
+ return -ENOMEM;
+ err = register_ftrace_direct_hash(ops, hash);
+ free_ftrace_hash(hash);
return err;
}
EXPORT_SYMBOL_GPL(register_ftrace_direct);
@@ -6083,111 +6007,24 @@ EXPORT_SYMBOL_GPL(register_ftrace_direct);
* 0 on success
* -EINVAL - The @ops object was not properly registered.
*/
-int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long addr,
+int unregister_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr,
bool free_filters)
{
- struct ftrace_hash *hash = ops->func_hash->filter_hash;
+ struct ftrace_hash *hash;
int err;
- if (check_direct_multi(ops))
- return -EINVAL;
- if (!(ops->flags & FTRACE_OPS_FL_ENABLED))
- return -EINVAL;
-
- mutex_lock(&direct_mutex);
- err = unregister_ftrace_function(ops);
- remove_direct_functions_hash(hash, addr);
- mutex_unlock(&direct_mutex);
-
- /* cleanup for possible another register call */
- ops->func = NULL;
- ops->trampoline = 0;
+ hash = hash_from_ip(ip, addr);
+ if (!hash)
+ return -ENOMEM;
+ err = unregister_ftrace_direct_hash(ops, hash);
+ free_ftrace_hash(hash);
if (free_filters)
ftrace_free_filter(ops);
return err;
}
EXPORT_SYMBOL_GPL(unregister_ftrace_direct);
-static int
-__modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
-{
- struct ftrace_hash *hash;
- struct ftrace_func_entry *entry, *iter;
- static struct ftrace_ops tmp_ops = {
- .func = ftrace_stub,
- .flags = FTRACE_OPS_FL_STUB,
- };
- int i, size;
- int err;
-
- lockdep_assert_held_once(&direct_mutex);
-
- /* Enable the tmp_ops to have the same functions as the direct ops */
- ftrace_ops_init(&tmp_ops);
- tmp_ops.func_hash = ops->func_hash;
- tmp_ops.direct_call = addr;
-
- err = register_ftrace_function_nolock(&tmp_ops);
- if (err)
- return err;
-
- /*
- * Now the ftrace_ops_list_func() is called to do the direct callers.
- * We can safely change the direct functions attached to each entry.
- */
- mutex_lock(&ftrace_lock);
-
- hash = ops->func_hash->filter_hash;
- size = 1 << hash->size_bits;
- for (i = 0; i < size; i++) {
- hlist_for_each_entry(iter, &hash->buckets[i], hlist) {
- entry = __ftrace_lookup_ip(direct_functions, iter->ip);
- if (!entry)
- continue;
- entry->direct = addr;
- }
- }
- /* Prevent store tearing if a trampoline concurrently accesses the value */
- WRITE_ONCE(ops->direct_call, addr);
-
- mutex_unlock(&ftrace_lock);
-
- /* Removing the tmp_ops will add the updated direct callers to the functions */
- unregister_ftrace_function(&tmp_ops);
-
- return err;
-}
-
-/**
- * modify_ftrace_direct_nolock - Modify an existing direct 'multi' call
- * to call something else
- * @ops: The address of the struct ftrace_ops object
- * @addr: The address of the new trampoline to call at @ops functions
- *
- * This is used to unregister currently registered direct caller and
- * register new one @addr on functions registered in @ops object.
- *
- * Note there's window between ftrace_shutdown and ftrace_startup calls
- * where there will be no callbacks called.
- *
- * Caller should already have direct_mutex locked, so we don't lock
- * direct_mutex here.
- *
- * Returns: zero on success. Non zero on error, which includes:
- * -EINVAL - The @ops object was not properly registered.
- */
-int modify_ftrace_direct_nolock(struct ftrace_ops *ops, unsigned long addr)
-{
- if (check_direct_multi(ops))
- return -EINVAL;
- if (!(ops->flags & FTRACE_OPS_FL_ENABLED))
- return -EINVAL;
-
- return __modify_ftrace_direct(ops, addr);
-}
-EXPORT_SYMBOL_GPL(modify_ftrace_direct_nolock);
-
/**
* modify_ftrace_direct - Modify an existing direct 'multi' call
* to call something else
@@ -6203,18 +6040,17 @@ EXPORT_SYMBOL_GPL(modify_ftrace_direct_nolock);
* Returns: zero on success. Non zero on error, which includes:
* -EINVAL - The @ops object was not properly registered.
*/
-int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)
+int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long addr, bool lock_direct_mutex)
{
+ struct ftrace_hash *hash;
int err;
- if (check_direct_multi(ops))
- return -EINVAL;
- if (!(ops->flags & FTRACE_OPS_FL_ENABLED))
- return -EINVAL;
+ hash = hash_from_ip(ip, addr);
+ if (!hash)
+ return -ENOMEM;
- mutex_lock(&direct_mutex);
- err = __modify_ftrace_direct(ops, addr);
- mutex_unlock(&direct_mutex);
+ err = modify_ftrace_direct_hash(ops, hash, lock_direct_mutex);
+ free_ftrace_hash(hash);
return err;
}
EXPORT_SYMBOL_GPL(modify_ftrace_direct);
diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c
index d88c44f1dfa5..37f5eb1f252b 100644
--- a/kernel/trace/trace_selftest.c
+++ b/kernel/trace/trace_selftest.c
@@ -1135,8 +1135,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,
* Register direct function together with graph tracer
* and make sure we get graph trace.
*/
- ftrace_set_filter_ip(&direct, (unsigned long)DYN_FTRACE_TEST_NAME, 0, 0);
- ret = register_ftrace_direct(&direct,
+ ret = register_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_NAME,
(unsigned long)ftrace_stub_direct_tramp);
if (ret)
goto out;
@@ -1159,7 +1158,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,
unregister_ftrace_graph(&fgraph_ops);
- ret = unregister_ftrace_direct(&direct,
+ ret = unregister_ftrace_direct(&direct, (unsigned long)DYN_FTRACE_TEST_NAME,
(unsigned long)ftrace_stub_direct_tramp,
true);
if (ret)
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 7/9] bpf: Add trampoline ip hash table
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (5 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 6/9] ftrace: Use direct hash interface in direct functions Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 8/9] ftrace: Factor ftrace_ops ops_func interface Jiri Olsa
2025-09-23 21:51 ` [PATCH 9/9] bpf, x86: Use single ftrace_ops for direct calls Jiri Olsa
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Following changes need to lookup trampoline based on its ip address,
adding hash table for that.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/bpf.h | 7 +++++--
kernel/bpf/trampoline.c | 30 +++++++++++++++++++-----------
2 files changed, 24 insertions(+), 13 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index dfc1a27b56d5..9d97c64cbf47 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1271,14 +1271,17 @@ struct bpf_tramp_image {
};
struct bpf_trampoline {
- /* hlist for trampoline_table */
- struct hlist_node hlist;
+ /* hlist for trampoline_key_table */
+ struct hlist_node hlist_key;
+ /* hlist for trampoline_ip_table */
+ struct hlist_node hlist_ip;
struct ftrace_ops *fops;
/* serializes access to fields of this trampoline */
struct mutex mutex;
refcount_t refcnt;
u32 flags;
u64 key;
+ unsigned long ip;
struct {
struct btf_func_model model;
void *addr;
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 97df6e482bf4..bdebaa94ca37 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -24,9 +24,10 @@ const struct bpf_prog_ops bpf_extension_prog_ops = {
#define TRAMPOLINE_HASH_BITS 10
#define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS)
-static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE];
+static struct hlist_head trampoline_key_table[TRAMPOLINE_TABLE_SIZE];
+static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE];
-/* serializes access to trampoline_table */
+/* serializes access to trampoline tables */
static DEFINE_MUTEX(trampoline_mutex);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
@@ -135,15 +136,15 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
PAGE_SIZE, true, ksym->name);
}
-static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
+static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
{
struct bpf_trampoline *tr;
struct hlist_head *head;
int i;
mutex_lock(&trampoline_mutex);
- head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
- hlist_for_each_entry(tr, head, hlist) {
+ head = &trampoline_key_table[hash_64(key, TRAMPOLINE_HASH_BITS)];
+ hlist_for_each_entry(tr, head, hlist_key) {
if (tr->key == key) {
refcount_inc(&tr->refcnt);
goto out;
@@ -164,8 +165,12 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
#endif
tr->key = key;
- INIT_HLIST_NODE(&tr->hlist);
- hlist_add_head(&tr->hlist, head);
+ tr->ip = ip;
+ INIT_HLIST_NODE(&tr->hlist_key);
+ INIT_HLIST_NODE(&tr->hlist_ip);
+ hlist_add_head(&tr->hlist_key, head);
+ head = &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)];
+ hlist_add_head(&tr->hlist_ip, head);
refcount_set(&tr->refcnt, 1);
mutex_init(&tr->mutex);
for (i = 0; i < BPF_TRAMP_MAX; i++)
@@ -800,7 +805,7 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog)
prog->aux->attach_btf_id);
bpf_lsm_find_cgroup_shim(prog, &bpf_func);
- tr = bpf_trampoline_lookup(key);
+ tr = bpf_trampoline_lookup(key, 0);
if (WARN_ON_ONCE(!tr))
return;
@@ -820,7 +825,7 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key,
{
struct bpf_trampoline *tr;
- tr = bpf_trampoline_lookup(key);
+ tr = bpf_trampoline_lookup(key, tgt_info->tgt_addr);
if (!tr)
return NULL;
@@ -856,7 +861,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
* fexit progs. The fentry-only trampoline will be freed via
* multiple rcu callbacks.
*/
- hlist_del(&tr->hlist);
+ hlist_del(&tr->hlist_key);
+ hlist_del(&tr->hlist_ip);
if (tr->fops) {
ftrace_free_filter(tr->fops);
kfree(tr->fops);
@@ -1129,7 +1135,9 @@ static int __init init_trampolines(void)
int i;
for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
- INIT_HLIST_HEAD(&trampoline_table[i]);
+ INIT_HLIST_HEAD(&trampoline_key_table[i]);
+ for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++)
+ INIT_HLIST_HEAD(&trampoline_ip_table[i]);
return 0;
}
late_initcall(init_trampolines);
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 8/9] ftrace: Factor ftrace_ops ops_func interface
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (6 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 7/9] bpf: Add trampoline ip hash table Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 9/9] bpf, x86: Use single ftrace_ops for direct calls Jiri Olsa
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
We are going to remove "ftrace_ops->private == bpf_trampoline" setup
in following changes.
Adding ip argument to ftrace_ops_func_t callback function, so we can
use it to look up the trampoline.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
include/linux/ftrace.h | 2 +-
kernel/bpf/trampoline.c | 3 ++-
kernel/trace/ftrace.c | 6 +++---
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 42da4dd08759..98e983d3d6b3 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -394,7 +394,7 @@ enum ftrace_ops_cmd {
* Negative on failure. The return value is dependent on the
* callback.
*/
-typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, enum ftrace_ops_cmd cmd);
+typedef int (*ftrace_ops_func_t)(struct ftrace_ops *op, unsigned long ip, enum ftrace_ops_cmd cmd);
#ifdef CONFIG_DYNAMIC_FTRACE
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index bdebaa94ca37..3464859189a2 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -33,7 +33,8 @@ static DEFINE_MUTEX(trampoline_mutex);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
-static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, enum ftrace_ops_cmd cmd)
+static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
+ enum ftrace_ops_cmd cmd)
{
struct bpf_trampoline *tr = ops->private;
int ret = 0;
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 6e9d5975477d..2af1304d1a83 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2036,7 +2036,7 @@ static int __ftrace_hash_update_ipmodify(struct ftrace_ops *ops,
*/
if (!ops->ops_func)
return -EBUSY;
- ret = ops->ops_func(ops, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF);
+ ret = ops->ops_func(ops, rec->ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF);
if (ret)
return ret;
} else if (is_ipmodify) {
@@ -8740,7 +8740,7 @@ static int prepare_direct_functions_for_ipmodify(struct ftrace_ops *ops)
if (!op->ops_func)
return -EBUSY;
- ret = op->ops_func(op, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER);
+ ret = op->ops_func(op, ip, FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_PEER);
if (ret)
return ret;
}
@@ -8787,7 +8787,7 @@ static void cleanup_direct_functions_after_ipmodify(struct ftrace_ops *ops)
/* The cleanup is optional, ignore any errors */
if (found_op && op->ops_func)
- op->ops_func(op, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER);
+ op->ops_func(op, ip, FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER);
}
}
mutex_unlock(&direct_mutex);
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 9/9] bpf, x86: Use single ftrace_ops for direct calls
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
` (7 preceding siblings ...)
2025-09-23 21:51 ` [PATCH 8/9] ftrace: Factor ftrace_ops ops_func interface Jiri Olsa
@ 2025-09-23 21:51 ` Jiri Olsa
8 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-23 21:51 UTC (permalink / raw)
To: Steven Rostedt, Florent Revest, Mark Rutland
Cc: bpf, linux-kernel, linux-trace-kernel, linux-arm-kernel,
Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Menglong Dong
Using single ftrace_ops for direct calls update instead of allocating
ftrace_ops obejct for each trampoline.
At the moment we can enable this only on x86 arch, because arm relies
on ftrace_ops object representing just single trampoline image (stored
in ftrace_ops::direct_call).
Adding HAVE_SINGLE_FTRACE_DIRECT_OPS config option to be enabled on
each arch that supports this.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
arch/x86/Kconfig | 1 +
kernel/bpf/trampoline.c | 85 +++++++++++++++++++++++++++++++++++------
kernel/trace/Kconfig | 3 ++
kernel/trace/ftrace.c | 15 +++++++-
4 files changed, 92 insertions(+), 12 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 52c8910ba2ef..bfc9115baa6f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -330,6 +330,7 @@ config X86
imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI
select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
select ARCH_SUPPORTS_PT_RECLAIM if X86_64
+ select HAVE_SINGLE_FTRACE_DIRECT_OPS if X86_64 && DYNAMIC_FTRACE_WITH_DIRECT_CALLS
config INSTRUCTION_DECODER
def_bool y
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 3464859189a2..e3b721773dfb 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -33,12 +33,40 @@ static DEFINE_MUTEX(trampoline_mutex);
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex);
+#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
+static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
+{
+ struct hlist_head *head_ip;
+ struct bpf_trampoline *tr;
+
+ mutex_lock(&trampoline_mutex);
+ head_ip = &trampoline_ip_table[hash_64(ip, TRAMPOLINE_HASH_BITS)];
+ hlist_for_each_entry(tr, head_ip, hlist_ip) {
+ if (tr->func.addr == (void *) ip)
+ goto out;
+ }
+ tr = NULL;
+out:
+ mutex_unlock(&trampoline_mutex);
+ return tr;
+}
+#else
+static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip)
+{
+ return ops->private;
+}
+#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */
+
static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip,
enum ftrace_ops_cmd cmd)
{
- struct bpf_trampoline *tr = ops->private;
+ struct bpf_trampoline *tr;
int ret = 0;
+ tr = direct_ops_ip_lookup(ops, ip);
+ if (!tr)
+ return -EINVAL;
+
if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) {
/* This is called inside register_ftrace_direct_multi(), so
* tr->mutex is already locked.
@@ -137,6 +165,48 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym)
PAGE_SIZE, true, ksym->name);
}
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
+struct ftrace_ops direct_ops = {
+ .ops_func = bpf_tramp_ftrace_ops_func,
+};
+
+static int direct_ops_get(struct bpf_trampoline *tr)
+{
+ tr->fops = &direct_ops;
+ return 0;
+}
+static void direct_ops_clear(struct bpf_trampoline *tr) { }
+static void direct_ops_free(struct bpf_trampoline *tr) { }
+#else
+static int direct_ops_get(struct bpf_trampoline *tr)
+{
+ tr->fops = kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL);
+ if (!tr->fops)
+ return -1;
+ tr->fops->private = tr;
+ tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
+ return 0;
+}
+
+static void direct_ops_clear(struct bpf_trampoline *tr)
+{
+ tr->fops->func = NULL;
+ tr->fops->trampoline = 0;
+}
+
+static void direct_ops_free(struct bpf_trampoline *tr)
+{
+ if (tr->fops) {
+ ftrace_free_filter(tr->fops);
+ kfree(tr->fops);
+ }
+}
+#endif /* CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */
+#else
+static void direct_ops_free(struct bpf_trampoline *tr) { }
+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */
+
static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
{
struct bpf_trampoline *tr;
@@ -155,14 +225,11 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip)
if (!tr)
goto out;
#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
- tr->fops = kzalloc(sizeof(struct ftrace_ops), GFP_KERNEL);
- if (!tr->fops) {
+ if (direct_ops_get(tr)) {
kfree(tr);
tr = NULL;
goto out;
}
- tr->fops->private = tr;
- tr->fops->ops_func = bpf_tramp_ftrace_ops_func;
#endif
tr->key = key;
@@ -482,8 +549,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
* trampoline again, and retry register.
*/
/* reset fops->func and fops->trampoline for re-register */
- tr->fops->func = NULL;
- tr->fops->trampoline = 0;
+ direct_ops_clear(tr);
/* free im memory and reallocate later */
bpf_tramp_image_free(im);
@@ -864,10 +930,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr)
*/
hlist_del(&tr->hlist_key);
hlist_del(&tr->hlist_ip);
- if (tr->fops) {
- ftrace_free_filter(tr->fops);
- kfree(tr->fops);
- }
+ direct_ops_free(tr);
kfree(tr);
out:
mutex_unlock(&trampoline_mutex);
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index d2c79da81e4f..4bf5beb04a5b 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -50,6 +50,9 @@ config HAVE_DYNAMIC_FTRACE_WITH_REGS
config HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
bool
+config HAVE_SINGLE_FTRACE_DIRECT_OPS
+ bool
+
config HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS
bool
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 2af1304d1a83..c4b969fb1010 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2592,8 +2592,13 @@ unsigned long ftrace_find_rec_direct(unsigned long ip)
static void call_direct_funcs(unsigned long ip, unsigned long pip,
struct ftrace_ops *ops, struct ftrace_regs *fregs)
{
- unsigned long addr = READ_ONCE(ops->direct_call);
+ unsigned long addr;
+#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
+ addr = ftrace_find_rec_direct(ip);
+#else
+ addr = READ_ONCE(ops->direct_call);
+#endif
if (!addr)
return;
@@ -5986,6 +5991,10 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned lo
if (!hash)
return -ENOMEM;
+#ifndef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
+ ops->direct_call = addr;
+#endif
+
err = register_ftrace_direct_hash(ops, hash);
free_ftrace_hash(hash);
return err;
@@ -6050,6 +6059,10 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long ip, unsigned long
return -ENOMEM;
err = modify_ftrace_direct_hash(ops, hash, lock_direct_mutex);
+#ifndef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
+ if (!err)
+ ops->direct_call = addr;
+#endif
free_ftrace_hash(hash);
return err;
}
--
2.51.0
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function
2025-09-23 21:51 ` [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function Jiri Olsa
@ 2025-09-24 9:04 ` Steven Rostedt
2025-09-24 14:37 ` Jiri Olsa
0 siblings, 1 reply; 14+ messages in thread
From: Steven Rostedt @ 2025-09-24 9:04 UTC (permalink / raw)
To: Jiri Olsa
Cc: Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong
On Tue, 23 Sep 2025 23:51:40 +0200
Jiri Olsa <jolsa@kernel.org> wrote:
> Adding register_ftrace_direct_hash function that registers
> all entries (ip -> direct) provided in hash argument.
>
> The difference to current register_ftrace_direct is
> - hash argument that allows to register multiple ip -> direct
> entries at once
I'm a bit confused. How is this different? Doesn't
register_ftrace_direct() register multiple ip -> direct entries at once
too? But instead of using a passed in hash, it uses the hash from
within the ftrace_ops.
> - we can call register_ftrace_direct_hash multiple times on the
> same ftrace_ops object, becase after first registration with
> register_ftrace_function_nolock, it uses ftrace_update_ops to
> update the ftrace_ops object
OK, I don't like the name "register" here. "register" should be for the
first instance and then it is registered. If you call it multiple times
on the same ops without "unregister" it should give an error.
Perhaps call this "update_ftrace_direct()" where it can update a direct
ftrace_ops from?
>
> This change will allow us to have simple ftrace_ops for all bpf
> direct interface users in following changes.
After applying all the patches, I have this:
$ git grep register_ftrace_direct_hash
include/linux/ftrace.h:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
include/linux/ftrace.h:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
include/linux/ftrace.h:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
include/linux/ftrace.h:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
kernel/trace/ftrace.c: err = register_ftrace_direct_hash(ops, hash);
kernel/trace/ftrace.c: err = unregister_ftrace_direct_hash(ops, hash);
kernel/trace/ftrace.c:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
kernel/trace/ftrace.c:EXPORT_SYMBOL_GPL(register_ftrace_direct_hash);
kernel/trace/ftrace.c:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
kernel/trace/ftrace.c:EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash);
Where I do not see it is used outside of ftrace.c. Why is it exported?
-- Steve
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function
2025-09-24 9:04 ` Steven Rostedt
@ 2025-09-24 14:37 ` Jiri Olsa
2025-09-24 15:07 ` Steven Rostedt
0 siblings, 1 reply; 14+ messages in thread
From: Jiri Olsa @ 2025-09-24 14:37 UTC (permalink / raw)
To: Steven Rostedt
Cc: Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong
On Wed, Sep 24, 2025 at 05:04:15AM -0400, Steven Rostedt wrote:
> On Tue, 23 Sep 2025 23:51:40 +0200
> Jiri Olsa <jolsa@kernel.org> wrote:
>
> > Adding register_ftrace_direct_hash function that registers
> > all entries (ip -> direct) provided in hash argument.
> >
> > The difference to current register_ftrace_direct is
> > - hash argument that allows to register multiple ip -> direct
> > entries at once
>
> I'm a bit confused. How is this different? Doesn't
> register_ftrace_direct() register multiple ip -> direct entries at once
> too? But instead of using a passed in hash, it uses the hash from
> within the ftrace_ops.
right, but that assumes that we can touch the hash in ftrace_ops directly,
but register_ftrace_direct_hash semantics is bit different, because it allows
to register new (ip,addr) entries on already 'running' ftrace_ops, in which
case you can't change the ftrace_ops hash directly
>
> > - we can call register_ftrace_direct_hash multiple times on the
> > same ftrace_ops object, becase after first registration with
> > register_ftrace_function_nolock, it uses ftrace_update_ops to
> > update the ftrace_ops object
>
> OK, I don't like the name "register" here. "register" should be for the
> first instance and then it is registered. If you call it multiple times
> on the same ops without "unregister" it should give an error.
>
> Perhaps call this "update_ftrace_direct()" where it can update a direct
> ftrace_ops from?
I agree the 'register' naming is confusing in here.. but we still need to
use 3 functions for register/unregister/modify operations, so perhaps:
update_ftrace_direct_add(ops, hash)
update_ftrace_direct_del(ops, hash)
update_ftrace_direct_mod(ops, hash)
?
>
> >
> > This change will allow us to have simple ftrace_ops for all bpf
> > direct interface users in following changes.
>
> After applying all the patches, I have this:
>
> $ git grep register_ftrace_direct_hash
> include/linux/ftrace.h:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
> include/linux/ftrace.h:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash);
> include/linux/ftrace.h:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
> include/linux/ftrace.h:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
> kernel/trace/ftrace.c: err = register_ftrace_direct_hash(ops, hash);
> kernel/trace/ftrace.c: err = unregister_ftrace_direct_hash(ops, hash);
> kernel/trace/ftrace.c:int register_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
> kernel/trace/ftrace.c:EXPORT_SYMBOL_GPL(register_ftrace_direct_hash);
> kernel/trace/ftrace.c:int unregister_ftrace_direct_hash(struct ftrace_ops *ops, struct ftrace_hash *hash)
> kernel/trace/ftrace.c:EXPORT_SYMBOL_GPL(unregister_ftrace_direct_hash);
>
> Where I do not see it is used outside of ftrace.c. Why is it exported?
I have bpf changes using this that I did not post yet, but even with that
there's probably no reason to export this.. will remove
thanks,
jirka
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function
2025-09-24 14:37 ` Jiri Olsa
@ 2025-09-24 15:07 ` Steven Rostedt
2025-09-24 16:00 ` Jiri Olsa
0 siblings, 1 reply; 14+ messages in thread
From: Steven Rostedt @ 2025-09-24 15:07 UTC (permalink / raw)
To: Jiri Olsa
Cc: Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong
On Wed, 24 Sep 2025 16:37:03 +0200
Jiri Olsa <olsajiri@gmail.com> wrote:
> I have bpf changes using this that I did not post yet, but even with that
> there's probably no reason to export this.. will remove
I'm interested in seeing these patches, as the ftrace hashes were
supposed to be opaque from other parts of the kernel.
-- Steve
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function
2025-09-24 15:07 ` Steven Rostedt
@ 2025-09-24 16:00 ` Jiri Olsa
0 siblings, 0 replies; 14+ messages in thread
From: Jiri Olsa @ 2025-09-24 16:00 UTC (permalink / raw)
To: Steven Rostedt
Cc: Jiri Olsa, Florent Revest, Mark Rutland, bpf, linux-kernel,
linux-trace-kernel, linux-arm-kernel, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Menglong Dong
On Wed, Sep 24, 2025 at 11:07:03AM -0400, Steven Rostedt wrote:
> On Wed, 24 Sep 2025 16:37:03 +0200
> Jiri Olsa <olsajiri@gmail.com> wrote:
>
> > I have bpf changes using this that I did not post yet, but even with that
> > there's probably no reason to export this.. will remove
>
> I'm interested in seeing these patches, as the ftrace hashes were
> supposed to be opaque from other parts of the kernel.
branch:
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/log/?h=bpf/tracing_multi
used in this commit:
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf/tracing_multi&id=8814108949537edaae84fbeee1cbc28280590b7f
background.. it's poc code for bpf tracing-multi attachment. Most likely we will
go with Menglong change instead [1], but it could use this direct interface in a
same way for speeding up the attachment
jirka
[1] https://lore.kernel.org/bpf/20250528034712.138701-1-dongml2@chinatelecom.cn/
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2025-09-24 16:00 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-23 21:51 [PATCH 0/9] ftrace,bpf: Use single direct ops for bpf trampolines Jiri Olsa
2025-09-23 21:51 ` [PATCH 1/9] ftrace: Make alloc_and_copy_ftrace_hash direct friendly Jiri Olsa
2025-09-23 21:51 ` [PATCH 2/9] ftrace: Add register_ftrace_direct_hash function Jiri Olsa
2025-09-24 9:04 ` Steven Rostedt
2025-09-24 14:37 ` Jiri Olsa
2025-09-24 15:07 ` Steven Rostedt
2025-09-24 16:00 ` Jiri Olsa
2025-09-23 21:51 ` [PATCH 3/9] ftrace: Add unregister_ftrace_direct_hash function Jiri Olsa
2025-09-23 21:51 ` [PATCH 4/9] ftrace: Add modify_ftrace_direct_hash function Jiri Olsa
2025-09-23 21:51 ` [PATCH 5/9] ftrace: Export some of hash related functions Jiri Olsa
2025-09-23 21:51 ` [PATCH 6/9] ftrace: Use direct hash interface in direct functions Jiri Olsa
2025-09-23 21:51 ` [PATCH 7/9] bpf: Add trampoline ip hash table Jiri Olsa
2025-09-23 21:51 ` [PATCH 8/9] ftrace: Factor ftrace_ops ops_func interface Jiri Olsa
2025-09-23 21:51 ` [PATCH 9/9] bpf, x86: Use single ftrace_ops for direct calls Jiri Olsa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).