[PATCHv2 bpf-next 00/24] bpf: Add multi uprobe link

[PATCHv2 bpf-next 00/24] bpf: Add multi uprobe link

[Jiri Olsa]

hi,
this patchset is adding support to attach multiple uprobes and usdt probes
through new uprobe_multi link.

The current uprobe is attached through the perf event and attaching many
uprobes takes a lot of time because of that.

The main reason is that we need to install perf event for each probed function
and profile shows perf event installation (perf_install_in_context) as culprit.

The new uprobe_multi link just creates raw uprobes and attaches the bpf
program to them without perf event being involved.

In addition to being faster we also save file descriptors. For the current
uprobe attach we use extra perf event fd for each probed function. The new
link just need one fd that covers all the functions we are attaching to.

v2 changes:
  - allow to run sleepable programs [Alexei]
  - allow user to specify only single path [Andrii]
  - pid filter support [Andrii]
  - add 'u[ret]probe.multi.s' section defs [Andrii]
  - add acked [Andrii]
  - remove WARN_ON_ONCE in bpf_uprobe_multi_cookie [Andrii]
  - merge tools uapi change with kernel change [Andrii]
  - drop ref_ctr_offset from struct bpf_uprobe [Andrii]
  - use generic string parsing instead of specific pattern matching
    in attach_uprobe_multi [Andrii]

followup changes:
  - move elf specific code into separate C object [Andrii]
  - will send fill_info support when [3] is merged [Andrii]

Also available at:
  https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  uprobe_multi

The support for bpftrace [2] and tetragon [1] is updated to v2 API changes.

thanks,
jirka


[1] https://github.com/cilium/tetragon/pull/936
[2] https://github.com/iovisor/bpftrace/compare/master...olsajiri:bpftrace:uprobe_multi
[3] https://lore.kernel.org/bpf/20230612151608.99661-1-laoar.shao@gmail.com/
[4] https://lore.kernel.org/bpf/20230613113119.2348619-1-jolsa@kernel.org/
---
Jiri Olsa (24):
      bpf: Add multi uprobe link
      bpf: Add cookies support for uprobe_multi link
      bpf: Add pid filter support for uprobe_multi link
      bpf: Add bpf_get_func_ip helper support for uprobe link
      libbpf: Add uprobe_multi attach type and link names
      libbpf: Add elf symbol iterator
      libbpf: Add open_elf/close_elf functions
      libbpf: Add elf_find_multi_func_offset function
      libbpf: Add elf_find_pattern_func_offset function
      libbpf: Add bpf_link_create support for multi uprobes
      libbpf: Add bpf_program__attach_uprobe_multi_opts function
      libbpf: Add support for u[ret]probe.multi[.s] program sections
      libbpf: Add uprobe multi link detection
      libbpf: Add uprobe multi link support to bpf_program__attach_usdt
      selftests/bpf: Add uprobe_multi skel test
      selftests/bpf: Add uprobe_multi api test
      selftests/bpf: Add uprobe_multi link test
      selftests/bpf: Add uprobe_multi test program
      selftests/bpf: Add uprobe_multi bench test
      selftests/bpf: Add usdt_multi test program
      selftests/bpf: Add usdt_multi bench test
      selftests/bpf: Add uprobe_multi cookie test
      selftests/bpf: Add uprobe_multi pid filter tests
      selftests/bpf: Add extra link to uprobe_multi tests

 include/linux/trace_events.h                               |   6 ++
 include/uapi/linux/bpf.h                                   |  16 ++++
 kernel/bpf/syscall.c                                       |  14 +++-
 kernel/trace/bpf_trace.c                                   | 349 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 tools/include/uapi/linux/bpf.h                             |  16 ++++
 tools/lib/bpf/bpf.c                                        |  11 +++
 tools/lib/bpf/bpf.h                                        |  11 ++-
 tools/lib/bpf/libbpf.c                                     | 645 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------
 tools/lib/bpf/libbpf.h                                     |  31 +++++++
 tools/lib/bpf/libbpf.map                                   |   1 +
 tools/lib/bpf/libbpf_internal.h                            |   7 ++
 tools/lib/bpf/usdt.c                                       | 120 ++++++++++++++++++++-------
 tools/testing/selftests/bpf/Makefile                       |  10 +++
 tools/testing/selftests/bpf/prog_tests/bpf_cookie.c        |  78 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c | 449 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/testing/selftests/bpf/progs/uprobe_multi.c           | 110 +++++++++++++++++++++++++
 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c      |  16 ++++
 tools/testing/selftests/bpf/uprobe_multi.c                 |  53 ++++++++++++
 tools/testing/selftests/bpf/usdt_multi.c                   |  24 ++++++
 19 files changed, 1841 insertions(+), 126 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c
 create mode 100644 tools/testing/selftests/bpf/uprobe_multi.c
 create mode 100644 tools/testing/selftests/bpf/usdt_multi.c

[PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

Adding new multi uprobe link that allows to attach bpf program
to multiple uprobes.

Uprobes to attach are specified via new link_create uprobe_multi
union:

  struct {
          __u32           flags;
          __u32           cnt;
          __aligned_u64   path;
          __aligned_u64   offsets;
          __aligned_u64   ref_ctr_offsets;
  } uprobe_multi;

Uprobes are defined for single binary specified in path and multiple
calling sites specified in offsets array with optional reference
counters specified in ref_ctr_offsets array. All specified arrays
have length of 'cnt'.

The 'flags' supports single bit for now that marks the uprobe as
return probe.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/linux/trace_events.h   |   6 +
 include/uapi/linux/bpf.h       |  14 ++
 kernel/bpf/syscall.c           |  12 +-
 kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  14 ++
 5 files changed, 281 insertions(+), 2 deletions(-)

diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
index 0e373222a6df..b0db245fc0f5 100644
--- a/include/linux/trace_events.h
+++ b/include/linux/trace_events.h
@@ -749,6 +749,7 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id,
 			    u32 *fd_type, const char **buf,
 			    u64 *probe_offset, u64 *probe_addr);
 int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
 #else
 static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx)
 {
@@ -795,6 +796,11 @@ bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
 {
 	return -EOPNOTSUPP;
 }
+static inline int
+bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 enum {
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index a7b5e91dd768..bfbc1246b220 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1035,6 +1035,7 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TRACE_UPROBE_MULTI,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,6 +1053,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
+	BPF_LINK_TYPE_UPROBE_MULTI = 11,
 
 	MAX_BPF_LINK_TYPE,
 };
@@ -1169,6 +1171,11 @@ enum bpf_link_type {
  */
 #define BPF_F_KPROBE_MULTI_RETURN	(1U << 0)
 
+/* link_create.uprobe_multi.flags used in LINK_CREATE command for
+ * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
+ */
+#define BPF_F_UPROBE_MULTI_RETURN	(1U << 0)
+
 /* When BPF ldimm64's insn[0].src_reg != 0 then this can have
  * the following extensions:
  *
@@ -1578,6 +1585,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				__u32		flags;
+				__u32		cnt;
+				__aligned_u64	path;
+				__aligned_u64	offsets;
+				__aligned_u64	ref_ctr_offsets;
+			} uprobe_multi;
 		};
 	} link_create;
 
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index a75c54b6f8a3..a96e46cd407e 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
 		return prog->enforce_expected_attach_type &&
 			prog->expected_attach_type != attach_type ?
 			-EINVAL : 0;
+	case BPF_PROG_TYPE_KPROBE:
+		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+		    attach_type != BPF_TRACE_KPROBE_MULTI)
+			return -EINVAL;
+		fallthrough;
 	default:
 		return 0;
 	}
@@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 		break;
 	case BPF_PROG_TYPE_KPROBE:
 		if (attr->link_create.attach_type != BPF_PERF_EVENT &&
-		    attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
+		    attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
+		    attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
 			ret = -EINVAL;
 			goto out;
 		}
@@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 	case BPF_PROG_TYPE_KPROBE:
 		if (attr->link_create.attach_type == BPF_PERF_EVENT)
 			ret = bpf_perf_link_attach(attr, prog);
-		else
+		else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
 			ret = bpf_kprobe_multi_link_attach(attr, prog);
+		else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
+			ret = bpf_uprobe_multi_link_attach(attr, prog);
 		break;
 	default:
 		ret = -EINVAL;
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 2bc41e6ac9fe..806ea9fd210d 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -23,6 +23,7 @@
 #include <linux/sort.h>
 #include <linux/key.h>
 #include <linux/verification.h>
+#include <linux/namei.h>
 
 #include <net/bpf_sk_storage.h>
 
@@ -2912,3 +2913,239 @@ static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
 	return 0;
 }
 #endif
+
+#ifdef CONFIG_UPROBES
+struct bpf_uprobe_multi_link;
+
+struct bpf_uprobe {
+	struct bpf_uprobe_multi_link *link;
+	loff_t offset;
+	struct uprobe_consumer consumer;
+};
+
+struct bpf_uprobe_multi_link {
+	struct path path;
+	struct bpf_link link;
+	u32 cnt;
+	struct bpf_uprobe *uprobes;
+};
+
+struct bpf_uprobe_multi_run_ctx {
+	struct bpf_run_ctx run_ctx;
+	unsigned long entry_ip;
+};
+
+static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
+				  u32 cnt)
+{
+	u32 i;
+
+	for (i = 0; i < cnt; i++) {
+		uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
+				  &uprobes[i].consumer);
+	}
+}
+
+static void bpf_uprobe_multi_link_release(struct bpf_link *link)
+{
+	struct bpf_uprobe_multi_link *umulti_link;
+
+	umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+	bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
+	path_put(&umulti_link->path);
+}
+
+static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
+{
+	struct bpf_uprobe_multi_link *umulti_link;
+
+	umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
+	kvfree(umulti_link->uprobes);
+	kfree(umulti_link);
+}
+
+static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
+	.release = bpf_uprobe_multi_link_release,
+	.dealloc = bpf_uprobe_multi_link_dealloc,
+};
+
+static int uprobe_prog_run(struct bpf_uprobe *uprobe,
+			   unsigned long entry_ip,
+			   struct pt_regs *regs)
+{
+	struct bpf_uprobe_multi_link *link = uprobe->link;
+	struct bpf_uprobe_multi_run_ctx run_ctx = {
+		.entry_ip = entry_ip,
+	};
+	struct bpf_prog *prog = link->link.prog;
+	struct bpf_run_ctx *old_run_ctx;
+	int err = 0;
+
+	might_fault();
+
+	rcu_read_lock_trace();
+	migrate_disable();
+
+	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
+		goto out;
+
+	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
+
+	if (!prog->aux->sleepable)
+		rcu_read_lock();
+
+	err = bpf_prog_run(link->link.prog, regs);
+
+	if (!prog->aux->sleepable)
+		rcu_read_unlock();
+
+	bpf_reset_run_ctx(old_run_ctx);
+
+out:
+	__this_cpu_dec(bpf_prog_active);
+	migrate_enable();
+	rcu_read_unlock_trace();
+	return err;
+}
+
+static int
+uprobe_multi_link_handler(struct uprobe_consumer *con, struct pt_regs *regs)
+{
+	struct bpf_uprobe *uprobe;
+
+	uprobe = container_of(con, struct bpf_uprobe, consumer);
+	return uprobe_prog_run(uprobe, instruction_pointer(regs), regs);
+}
+
+static int
+uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, struct pt_regs *regs)
+{
+	struct bpf_uprobe *uprobe;
+
+	uprobe = container_of(con, struct bpf_uprobe, consumer);
+	return uprobe_prog_run(uprobe, func, regs);
+}
+
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	unsigned long __user *uref_ctr_offsets, ref_ctr_offset = 0;
+	struct bpf_uprobe_multi_link *link = NULL;
+	unsigned long __user *uoffsets, offset;
+	unsigned long *ref_ctr_offsets = NULL;
+	struct bpf_link_primer link_primer;
+	struct bpf_uprobe *uprobes = NULL;
+	void __user *upath;
+	u32 flags, cnt, i;
+	struct path path;
+	char *name;
+	int err;
+
+	/* no support for 32bit archs yet */
+	if (sizeof(u64) != sizeof(void *))
+		return -EOPNOTSUPP;
+
+	if (prog->expected_attach_type != BPF_TRACE_UPROBE_MULTI)
+		return -EINVAL;
+
+	flags = attr->link_create.uprobe_multi.flags;
+	if (flags & ~BPF_F_UPROBE_MULTI_RETURN)
+		return -EINVAL;
+
+	/*
+	 * path, offsets and cnt are mandatory,
+	 * ref_ctr_offsets is optional
+	 */
+	upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
+	uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
+	cnt = attr->link_create.uprobe_multi.cnt;
+	if (!upath || !uoffsets || !cnt)
+		return -EINVAL;
+
+	uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
+
+	name = strndup_user(upath, PATH_MAX);
+	if (IS_ERR(name)) {
+		err = PTR_ERR(name);
+		return err;
+	}
+
+	err = kern_path(name, LOOKUP_FOLLOW, &path);
+	kfree(name);
+	if (err)
+		return err;
+
+	if (!d_is_reg(path.dentry)) {
+		err = -EINVAL;
+		goto error_path_put;
+	}
+
+	err = -ENOMEM;
+
+	link = kzalloc(sizeof(*link), GFP_KERNEL);
+	uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
+	ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);
+
+	if (!uprobes || !ref_ctr_offsets || !link)
+		goto error_free;
+
+	for (i = 0; i < cnt; i++) {
+		if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
+			err = -EFAULT;
+			goto error_free;
+		}
+		if (__get_user(offset, uoffsets + i)) {
+			err = -EFAULT;
+			goto error_free;
+		}
+
+		uprobes[i].offset = offset;
+		uprobes[i].link = link;
+
+		if (flags & BPF_F_UPROBE_MULTI_RETURN)
+			uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
+		else
+			uprobes[i].consumer.handler = uprobe_multi_link_handler;
+
+		ref_ctr_offsets[i] = ref_ctr_offset;
+	}
+
+	link->cnt = cnt;
+	link->uprobes = uprobes;
+	link->path = path;
+
+	bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
+		      &bpf_uprobe_multi_link_lops, prog);
+
+	err = bpf_link_prime(&link->link, &link_primer);
+	if (err)
+		goto error_free;
+
+	for (i = 0; i < cnt; i++) {
+		err = uprobe_register_refctr(d_real_inode(link->path.dentry),
+					     uprobes[i].offset, ref_ctr_offsets[i],
+					     &uprobes[i].consumer);
+		if (err) {
+			bpf_uprobe_unregister(&path, uprobes, i);
+			bpf_link_cleanup(&link_primer);
+			kvfree(ref_ctr_offsets);
+			return err;
+		}
+	}
+
+	kvfree(ref_ctr_offsets);
+	return bpf_link_settle(&link_primer);
+
+error_free:
+	kvfree(ref_ctr_offsets);
+	kvfree(uprobes);
+	kfree(link);
+error_path_put:
+	path_put(&path);
+	return err;
+}
+#else /* !CONFIG_UPROBES */
+int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+	return -EOPNOTSUPP;
+}
+#endif /* CONFIG_UPROBES */
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index a7b5e91dd768..bfbc1246b220 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1035,6 +1035,7 @@ enum bpf_attach_type {
 	BPF_TRACE_KPROBE_MULTI,
 	BPF_LSM_CGROUP,
 	BPF_STRUCT_OPS,
+	BPF_TRACE_UPROBE_MULTI,
 	__MAX_BPF_ATTACH_TYPE
 };
 
@@ -1052,6 +1053,7 @@ enum bpf_link_type {
 	BPF_LINK_TYPE_KPROBE_MULTI = 8,
 	BPF_LINK_TYPE_STRUCT_OPS = 9,
 	BPF_LINK_TYPE_NETFILTER = 10,
+	BPF_LINK_TYPE_UPROBE_MULTI = 11,
 
 	MAX_BPF_LINK_TYPE,
 };
@@ -1169,6 +1171,11 @@ enum bpf_link_type {
  */
 #define BPF_F_KPROBE_MULTI_RETURN	(1U << 0)
 
+/* link_create.uprobe_multi.flags used in LINK_CREATE command for
+ * BPF_TRACE_UPROBE_MULTI attach type to create return probe.
+ */
+#define BPF_F_UPROBE_MULTI_RETURN	(1U << 0)
+
 /* When BPF ldimm64's insn[0].src_reg != 0 then this can have
  * the following extensions:
  *
@@ -1578,6 +1585,13 @@ union bpf_attr {
 				__s32		priority;
 				__u32		flags;
 			} netfilter;
+			struct {
+				__u32		flags;
+				__u32		cnt;
+				__aligned_u64	path;
+				__aligned_u64	offsets;
+				__aligned_u64	ref_ctr_offsets;
+			} uprobe_multi;
 		};
 	} link_create;
 
-- 
2.41.0

[PATCHv2 bpf-next 02/24] bpf: Add cookies support for uprobe_multi link

[Jiri Olsa]

Adding support to specify cookies array for uprobe_multi link.

The cookies array share indexes and length with other uprobe_multi
arrays (offsets/ref_ctr_offsets).

The cookies[i] value defines cookie for i-the uprobe and will be
returned by bpf_get_attach_cookie helper when called from ebpf
program hooked to that specific uprobe.

Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 47 insertions(+), 5 deletions(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index bfbc1246b220..12d4174fce8f 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1591,6 +1591,7 @@ union bpf_attr {
 				__aligned_u64	path;
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
+				__aligned_u64	cookies;
 			} uprobe_multi;
 		};
 	} link_create;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index a96e46cd407e..3ae444898c15 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -4651,7 +4651,7 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 
-#define BPF_LINK_CREATE_LAST_FIELD link_create.kprobe_multi.cookies
+#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.cookies
 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 {
 	enum bpf_prog_type ptype;
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 806ea9fd210d..f9cd7d283426 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -87,6 +87,8 @@ static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
 static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx);
 static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
 
+static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx);
+
 /**
  * trace_call_bpf - invoke BPF program
  * @call: tracepoint event
@@ -1089,6 +1091,18 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_kmulti = {
 	.arg1_type	= ARG_PTR_TO_CTX,
 };
 
+BPF_CALL_1(bpf_get_attach_cookie_uprobe_multi, struct pt_regs *, regs)
+{
+	return bpf_uprobe_multi_cookie(current->bpf_ctx);
+}
+
+static const struct bpf_func_proto bpf_get_attach_cookie_proto_umulti = {
+	.func		= bpf_get_attach_cookie_uprobe_multi,
+	.gpl_only	= false,
+	.ret_type	= RET_INTEGER,
+	.arg1_type	= ARG_PTR_TO_CTX,
+};
+
 BPF_CALL_1(bpf_get_attach_cookie_trace, void *, ctx)
 {
 	struct bpf_trace_run_ctx *run_ctx;
@@ -1535,9 +1549,11 @@ kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 			&bpf_get_func_ip_proto_kprobe_multi :
 			&bpf_get_func_ip_proto_kprobe;
 	case BPF_FUNC_get_attach_cookie:
-		return prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI ?
-			&bpf_get_attach_cookie_proto_kmulti :
-			&bpf_get_attach_cookie_proto_trace;
+		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI)
+			return &bpf_get_attach_cookie_proto_kmulti;
+		if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI)
+			return &bpf_get_attach_cookie_proto_umulti;
+		return &bpf_get_attach_cookie_proto_trace;
 	default:
 		return bpf_tracing_func_proto(func_id, prog);
 	}
@@ -2920,6 +2936,7 @@ struct bpf_uprobe_multi_link;
 struct bpf_uprobe {
 	struct bpf_uprobe_multi_link *link;
 	loff_t offset;
+	u64 cookie;
 	struct uprobe_consumer consumer;
 };
 
@@ -2933,6 +2950,7 @@ struct bpf_uprobe_multi_link {
 struct bpf_uprobe_multi_run_ctx {
 	struct bpf_run_ctx run_ctx;
 	unsigned long entry_ip;
+	struct bpf_uprobe *uprobe;
 };
 
 static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
@@ -2976,6 +2994,7 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	struct bpf_uprobe_multi_link *link = uprobe->link;
 	struct bpf_uprobe_multi_run_ctx run_ctx = {
 		.entry_ip = entry_ip,
+		.uprobe = uprobe,
 	};
 	struct bpf_prog *prog = link->link.prog;
 	struct bpf_run_ctx *old_run_ctx;
@@ -3026,6 +3045,16 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
 	return uprobe_prog_run(uprobe, func, regs);
 }
 
+static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
+{
+	struct bpf_uprobe_multi_run_ctx *run_ctx;
+
+	if (!ctx)
+		return 0;
+	run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
+	return run_ctx->uprobe->cookie;
+}
+
 int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
 {
 	unsigned long __user *uref_ctr_offsets, ref_ctr_offset = 0;
@@ -3034,6 +3063,7 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 	unsigned long *ref_ctr_offsets = NULL;
 	struct bpf_link_primer link_primer;
 	struct bpf_uprobe *uprobes = NULL;
+	u64 __user *ucookies, cookie = 0;
 	void __user *upath;
 	u32 flags, cnt, i;
 	struct path path;
@@ -3053,7 +3083,7 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 
 	/*
 	 * path, offsets and cnt are mandatory,
-	 * ref_ctr_offsets is optional
+	 * ref_ctr_offsets and cookies are optional
 	 */
 	upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path);
 	uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets);
@@ -3062,6 +3092,7 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 		return -EINVAL;
 
 	uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets);
+	ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies);
 
 	name = strndup_user(upath, PATH_MAX);
 	if (IS_ERR(name)) {
@@ -3089,6 +3120,10 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 		goto error_free;
 
 	for (i = 0; i < cnt; i++) {
+		if (ucookies && __get_user(cookie, ucookies + i)) {
+			err = -EFAULT;
+			goto error_free;
+		}
 		if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
 			err = -EFAULT;
 			goto error_free;
@@ -3099,6 +3134,7 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 		}
 
 		uprobes[i].offset = offset;
+		uprobes[i].cookie = cookie;
 		uprobes[i].link = link;
 
 		if (flags & BPF_F_UPROBE_MULTI_RETURN)
@@ -3148,4 +3184,8 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 {
 	return -EOPNOTSUPP;
 }
+static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
+{
+	return 0;
+}
 #endif /* CONFIG_UPROBES */
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index bfbc1246b220..12d4174fce8f 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1591,6 +1591,7 @@ union bpf_attr {
 				__aligned_u64	path;
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
+				__aligned_u64	cookies;
 			} uprobe_multi;
 		};
 	} link_create;
-- 
2.41.0

[PATCHv2 bpf-next 03/24] bpf: Add pid filter support for uprobe_multi link

[Jiri Olsa]

Adding support to specify pid for uprobe_multi link and the uprobes
are created only for task with given pid value.

Using the consumer.filter filter callback for that, so the task gets
filtered during the uprobe installation.

We still need to check the task during runtime in the uprobe handler,
because the handler could get executed if there's another system
wide consumer on the same uprobe (thanks Oleg for the insight).

Cc: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/uapi/linux/bpf.h       |  1 +
 kernel/bpf/syscall.c           |  2 +-
 kernel/trace/bpf_trace.c       | 33 +++++++++++++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 12d4174fce8f..117acca838f1 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1592,6 +1592,7 @@ union bpf_attr {
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
 				__aligned_u64	cookies;
+				__u32		pid;
 			} uprobe_multi;
 		};
 	} link_create;
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 3ae444898c15..b871f0eb8a05 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -4651,7 +4651,7 @@ static int bpf_map_do_batch(const union bpf_attr *attr,
 	return err;
 }
 
-#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.cookies
+#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid
 static int link_create(union bpf_attr *attr, bpfptr_t uattr)
 {
 	enum bpf_prog_type ptype;
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index f9cd7d283426..5c5b543d7d03 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -2945,6 +2945,7 @@ struct bpf_uprobe_multi_link {
 	struct bpf_link link;
 	u32 cnt;
 	struct bpf_uprobe *uprobes;
+	struct task_struct *task;
 };
 
 struct bpf_uprobe_multi_run_ctx {
@@ -2971,6 +2972,8 @@ static void bpf_uprobe_multi_link_release(struct bpf_link *link)
 	umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
 	bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
 	path_put(&umulti_link->path);
+	if (umulti_link->task)
+		put_task_struct(umulti_link->task);
 }
 
 static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
@@ -3000,6 +3003,9 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	struct bpf_run_ctx *old_run_ctx;
 	int err = 0;
 
+	if (link->task && current != link->task)
+		return 0;
+
 	might_fault();
 
 	rcu_read_lock_trace();
@@ -3027,6 +3033,16 @@ static int uprobe_prog_run(struct bpf_uprobe *uprobe,
 	return err;
 }
 
+static bool
+uprobe_multi_link_filter(struct uprobe_consumer *con, enum uprobe_filter_ctx ctx,
+			 struct mm_struct *mm)
+{
+	struct bpf_uprobe *uprobe;
+
+	uprobe = container_of(con, struct bpf_uprobe, consumer);
+	return uprobe->link->task->mm == mm;
+}
+
 static int
 uprobe_multi_link_handler(struct uprobe_consumer *con, struct pt_regs *regs)
 {
@@ -3064,10 +3080,12 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 	struct bpf_link_primer link_primer;
 	struct bpf_uprobe *uprobes = NULL;
 	u64 __user *ucookies, cookie = 0;
+	struct task_struct *task = NULL;
 	void __user *upath;
 	u32 flags, cnt, i;
 	struct path path;
 	char *name;
+	pid_t pid;
 	int err;
 
 	/* no support for 32bit archs yet */
@@ -3110,6 +3128,15 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 		goto error_path_put;
 	}
 
+	pid = attr->link_create.uprobe_multi.pid;
+	if (pid) {
+		rcu_read_lock();
+		task = get_pid_task(find_vpid(pid), PIDTYPE_PID);
+		rcu_read_unlock();
+		if (!task)
+			goto error_path_put;
+	}
+
 	err = -ENOMEM;
 
 	link = kzalloc(sizeof(*link), GFP_KERNEL);
@@ -3143,11 +3170,15 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 			uprobes[i].consumer.handler = uprobe_multi_link_handler;
 
 		ref_ctr_offsets[i] = ref_ctr_offset;
+
+		if (pid)
+			uprobes[i].consumer.filter = uprobe_multi_link_filter;
 	}
 
 	link->cnt = cnt;
 	link->uprobes = uprobes;
 	link->path = path;
+	link->task = task;
 
 	bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
 		      &bpf_uprobe_multi_link_lops, prog);
@@ -3175,6 +3206,8 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
 	kvfree(ref_ctr_offsets);
 	kvfree(uprobes);
 	kfree(link);
+	if (task)
+		put_task_struct(task);
 error_path_put:
 	path_put(&path);
 	return err;
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 12d4174fce8f..117acca838f1 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1592,6 +1592,7 @@ union bpf_attr {
 				__aligned_u64	offsets;
 				__aligned_u64	ref_ctr_offsets;
 				__aligned_u64	cookies;
+				__u32		pid;
 			} uprobe_multi;
 		};
 	} link_create;
-- 
2.41.0

[PATCHv2 bpf-next 04/24] bpf: Add bpf_get_func_ip helper support for uprobe link

[Jiri Olsa]

Adding support for bpf_get_func_ip helper being called from
ebpf program attached by uprobe_multi link.

It returns the ip of the uprobe.

Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 kernel/trace/bpf_trace.c | 33 ++++++++++++++++++++++++++++++---
 1 file changed, 30 insertions(+), 3 deletions(-)

diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 5c5b543d7d03..dfba0bb479a2 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -88,6 +88,7 @@ static u64 bpf_kprobe_multi_cookie(struct bpf_run_ctx *ctx);
 static u64 bpf_kprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
 
 static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx);
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx);
 
 /**
  * trace_call_bpf - invoke BPF program
@@ -1091,6 +1092,18 @@ static const struct bpf_func_proto bpf_get_attach_cookie_proto_kmulti = {
 	.arg1_type	= ARG_PTR_TO_CTX,
 };
 
+BPF_CALL_1(bpf_get_func_ip_uprobe_multi, struct pt_regs *, regs)
+{
+	return bpf_uprobe_multi_entry_ip(current->bpf_ctx);
+}
+
+static const struct bpf_func_proto bpf_get_func_ip_proto_uprobe_multi = {
+	.func		= bpf_get_func_ip_uprobe_multi,
+	.gpl_only	= false,
+	.ret_type	= RET_INTEGER,
+	.arg1_type	= ARG_PTR_TO_CTX,
+};
+
 BPF_CALL_1(bpf_get_attach_cookie_uprobe_multi, struct pt_regs *, regs)
 {
 	return bpf_uprobe_multi_cookie(current->bpf_ctx);
@@ -1545,9 +1558,11 @@ kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return &bpf_override_return_proto;
 #endif
 	case BPF_FUNC_get_func_ip:
-		return prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI ?
-			&bpf_get_func_ip_proto_kprobe_multi :
-			&bpf_get_func_ip_proto_kprobe;
+		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI)
+			return &bpf_get_func_ip_proto_kprobe_multi;
+		if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI)
+			return &bpf_get_func_ip_proto_uprobe_multi;
+		return &bpf_get_func_ip_proto_kprobe;
 	case BPF_FUNC_get_attach_cookie:
 		if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI)
 			return &bpf_get_attach_cookie_proto_kmulti;
@@ -3061,6 +3076,14 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
 	return uprobe_prog_run(uprobe, func, regs);
 }
 
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
+{
+	struct bpf_uprobe_multi_run_ctx *run_ctx;
+
+	run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
+	return run_ctx->entry_ip;
+}
+
 static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
 {
 	struct bpf_uprobe_multi_run_ctx *run_ctx;
@@ -3221,4 +3244,8 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
 {
 	return 0;
 }
+static u64 bpf_uprobe_multi_entry_ip(struct bpf_run_ctx *ctx)
+{
+	return 0;
+}
 #endif /* CONFIG_UPROBES */
-- 
2.41.0

[PATCHv2 bpf-next 05/24] libbpf: Add uprobe_multi attach type and link names

[Jiri Olsa]

Adding new uprobe_multi attach type and link names,
so the functions can resolve the new values.

Acked-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 47632606b06d..af52188daa80 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -117,6 +117,7 @@ static const char * const attach_type_name[] = {
 	[BPF_PERF_EVENT]		= "perf_event",
 	[BPF_TRACE_KPROBE_MULTI]	= "trace_kprobe_multi",
 	[BPF_STRUCT_OPS]		= "struct_ops",
+	[BPF_TRACE_UPROBE_MULTI]	= "trace_uprobe_multi",
 };
 
 static const char * const link_type_name[] = {
@@ -131,6 +132,7 @@ static const char * const link_type_name[] = {
 	[BPF_LINK_TYPE_KPROBE_MULTI]		= "kprobe_multi",
 	[BPF_LINK_TYPE_STRUCT_OPS]		= "struct_ops",
 	[BPF_LINK_TYPE_NETFILTER]		= "netfilter",
+	[BPF_LINK_TYPE_UPROBE_MULTI]		= "uprobe_multi",
 };
 
 static const char * const map_type_name[] = {
-- 
2.41.0

[PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

[Jiri Olsa]

Adding elf symbol iterator object (and some functions) that follow
open-coded iterator pattern and some functions to ease up iterating
elf object symbols.

The idea is to iterate single symbol section with:

  struct elf_symbol_iter iter;
  struct elf_symbol *sym;

  if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))
        goto error;

  while ((sym = elf_symbol_iter_next(&iter))) {
        ...
  }

I considered opening the elf inside the iterator and iterate all symbol
sections, but then it gets more complicated wrt user checks for when
the next section is processed.

Plus side is the we don't need 'exit' function, because caller/user is
in charge of that.

The returned iterated symbol object from elf_symbol_iter_next function
is placed inside the struct elf_symbol_iter, so no extra allocation or
argument is needed.

Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
 1 file changed, 114 insertions(+), 65 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index af52188daa80..cdac368c7ce1 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
 	return NULL;
 }
 
+struct elf_symbol {
+	const char *name;
+	unsigned long offset;
+	int bind;
+};
+
+struct elf_symbol_iter {
+	Elf *elf;
+	Elf_Data *symbols;
+	size_t nr_syms;
+	size_t strtabidx;
+	size_t idx;
+	struct elf_symbol sym;
+};
+
+static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
+			       Elf *elf, const char *binary_path,
+			       int sh_type)
+{
+	Elf_Scn *scn = NULL;
+	GElf_Ehdr ehdr;
+	GElf_Shdr sh;
+
+	memset(iter, 0, sizeof(*iter));
+
+	if (!gelf_getehdr(elf, &ehdr)) {
+		pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+
+	scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
+	if (!scn) {
+		pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
+			 binary_path);
+		return -EINVAL;
+	}
+
+	if (!gelf_getshdr(scn, &sh))
+		return -EINVAL;
+
+	iter->strtabidx = sh.sh_link;
+	iter->symbols = elf_getdata(scn, 0);
+	if (!iter->symbols) {
+		pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
+			binary_path, elf_errmsg(-1));
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
+	iter->elf = elf;
+	return 0;
+}
+
+static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
+{
+	struct elf_symbol *ret = &iter->sym;
+	unsigned long offset = 0;
+	const char *name = NULL;
+	GElf_Shdr sym_sh;
+	Elf_Scn *sym_scn;
+	GElf_Sym sym;
+	size_t idx;
+
+	for (idx = iter->idx; idx < iter->nr_syms; idx++) {
+		if (!gelf_getsym(iter->symbols, idx, &sym))
+			continue;
+		if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
+			continue;
+		name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
+		if (!name)
+			continue;
+
+		/* Transform symbol's virtual address (absolute for
+		 * binaries and relative for shared libs) into file
+		 * offset, which is what kernel is expecting for
+		 * uprobe/uretprobe attachment.
+		 * See Documentation/trace/uprobetracer.rst for more
+		 * details.
+		 * This is done by looking up symbol's containing
+		 * section's header and using iter's virtual address
+		 * (sh_addr) and corresponding file offset (sh_offset)
+		 * to transform sym.st_value (virtual address) into
+		 * desired final file offset.
+		 */
+		sym_scn = elf_getscn(iter->elf, sym.st_shndx);
+		if (!sym_scn)
+			continue;
+		if (!gelf_getshdr(sym_scn, &sym_sh))
+			continue;
+
+		offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
+		break;
+	}
+
+	/* we reached the last symbol */
+	if (idx == iter->nr_syms)
+		return NULL;
+	iter->idx = idx + 1;
+	ret->name = name;
+	ret->bind = GELF_ST_BIND(sym.st_info);
+	ret->offset = offset;
+	return ret;
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library
@@ -10855,94 +10958,40 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
 	 * reported as a warning/error.
 	 */
 	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
-		size_t nr_syms, strtabidx, idx;
-		Elf_Data *symbols = NULL;
-		Elf_Scn *scn = NULL;
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
 		int last_bind = -1;
-		const char *sname;
-		GElf_Shdr sh;
 
-		scn = elf_find_next_scn_by_type(elf, sh_types[i], NULL);
-		if (!scn) {
-			pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
-				 binary_path);
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
 			continue;
-		}
-		if (!gelf_getshdr(scn, &sh))
-			continue;
-		strtabidx = sh.sh_link;
-		symbols = elf_getdata(scn, 0);
-		if (!symbols) {
-			pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
-				binary_path, elf_errmsg(-1));
-			ret = -LIBBPF_ERRNO__FORMAT;
-			goto out;
-		}
-		nr_syms = symbols->d_size / sh.sh_entsize;
-
-		for (idx = 0; idx < nr_syms; idx++) {
-			int curr_bind;
-			GElf_Sym sym;
-			Elf_Scn *sym_scn;
-			GElf_Shdr sym_sh;
-
-			if (!gelf_getsym(symbols, idx, &sym))
-				continue;
-
-			if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
-				continue;
-
-			sname = elf_strptr(elf, strtabidx, sym.st_name);
-			if (!sname)
-				continue;
-
-			curr_bind = GELF_ST_BIND(sym.st_info);
 
+		while ((sym = elf_symbol_iter_next(&iter))) {
 			/* User can specify func, func@@LIB or func@@LIB_VERSION. */
-			if (strncmp(sname, name, name_len) != 0)
+			if (strncmp(sym->name, name, name_len) != 0)
 				continue;
 			/* ...but we don't want a search for "foo" to match 'foo2" also, so any
 			 * additional characters in sname should be of the form "@@LIB".
 			 */
-			if (!is_name_qualified && sname[name_len] != '\0' && sname[name_len] != '@')
+			if (!is_name_qualified && sym->name[name_len] != '\0' && sym->name[name_len] != '@')
 				continue;
 
 			if (ret >= 0) {
 				/* handle multiple matches */
-				if (last_bind != STB_WEAK && curr_bind != STB_WEAK) {
+				if (last_bind != STB_WEAK && sym->bind != STB_WEAK) {
 					/* Only accept one non-weak bind. */
 					pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
-						sname, name, binary_path);
+						sym->name, name, binary_path);
 					ret = -LIBBPF_ERRNO__FORMAT;
 					goto out;
-				} else if (curr_bind == STB_WEAK) {
+				} else if (sym->bind == STB_WEAK) {
 					/* already have a non-weak bind, and
 					 * this is a weak bind, so ignore.
 					 */
 					continue;
 				}
 			}
-
-			/* Transform symbol's virtual address (absolute for
-			 * binaries and relative for shared libs) into file
-			 * offset, which is what kernel is expecting for
-			 * uprobe/uretprobe attachment.
-			 * See Documentation/trace/uprobetracer.rst for more
-			 * details.
-			 * This is done by looking up symbol's containing
-			 * section's header and using it's virtual address
-			 * (sh_addr) and corresponding file offset (sh_offset)
-			 * to transform sym.st_value (virtual address) into
-			 * desired final file offset.
-			 */
-			sym_scn = elf_getscn(elf, sym.st_shndx);
-			if (!sym_scn)
-				continue;
-			if (!gelf_getshdr(sym_scn, &sym_sh))
-				continue;
-
-			ret = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
-			last_bind = curr_bind;
+			last_bind = sym->bind;
+			ret = sym->offset;
 		}
 		if (ret > 0)
 			break;
-- 
2.41.0

[PATCHv2 bpf-next 07/24] libbpf: Add open_elf/close_elf functions

[Jiri Olsa]

Adding open_elf/close_elf functions and using it in
elf_find_func_offset_from_file function. It will be
used in following changes to save some code.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
 1 file changed, 44 insertions(+), 18 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index cdac368c7ce1..30d9e3b69114 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
 	return ret;
 }
 
+struct elf_fd {
+	Elf *elf;
+	int fd;
+};
+
+static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
+{
+	char errmsg[STRERR_BUFSIZE];
+	int fd, ret;
+	Elf *elf;
+
+	if (elf_version(EV_CURRENT) == EV_NONE) {
+		pr_warn("failed to init libelf for %s\n", binary_path);
+		return -LIBBPF_ERRNO__LIBELF;
+	}
+	fd = open(binary_path, O_RDONLY | O_CLOEXEC);
+	if (fd < 0) {
+		ret = -errno;
+		pr_warn("failed to open %s: %s\n", binary_path,
+			libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
+		return ret;
+	}
+	elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
+	if (!elf) {
+		pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
+		close(fd);
+		return -LIBBPF_ERRNO__FORMAT;
+	}
+	elf_fd->fd = fd;
+	elf_fd->elf = elf;
+	return 0;
+}
+
+static void close_elf(struct elf_fd *elf_fd)
+{
+	elf_end(elf_fd->elf);
+	close(elf_fd->fd);
+}
+
 /* Find offset of function name in the provided ELF object. "binary_path" is
  * the path to the ELF binary represented by "elf", and only used for error
  * reporting matters. "name" matches symbol name or name@@LIB for library
@@ -11019,28 +11058,15 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
  */
 static long elf_find_func_offset_from_file(const char *binary_path, const char *name)
 {
-	char errmsg[STRERR_BUFSIZE];
+	struct elf_fd elf_fd = {};
 	long ret = -ENOENT;
-	Elf *elf;
-	int fd;
 
-	fd = open(binary_path, O_RDONLY | O_CLOEXEC);
-	if (fd < 0) {
-		ret = -errno;
-		pr_warn("failed to open %s: %s\n", binary_path,
-			libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
 		return ret;
-	}
-	elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
-	if (!elf) {
-		pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
-		close(fd);
-		return -LIBBPF_ERRNO__FORMAT;
-	}
 
-	ret = elf_find_func_offset(elf, binary_path, name);
-	elf_end(elf);
-	close(fd);
+	ret = elf_find_func_offset(elf_fd.elf, binary_path, name);
+	close_elf(&elf_fd);
 	return ret;
 }
 
-- 
2.41.0

[PATCHv2 bpf-next 08/24] libbpf: Add elf_find_multi_func_offset function

[Jiri Olsa]

Adding elf_find_multi_func_offset function that looks up
offsets for symbols specified in syms array argument.

Offsets are returned in allocated array with the 'cnt' size,
that needs to be released by the caller.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |   2 +
 2 files changed, 114 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 30d9e3b69114..1c310b718961 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
 	return ret;
 }
 
+struct elf_symbol_offset {
+	const char *name;
+	unsigned long offset;
+	int bind;
+	int idx;
+};
+
+static int cmp_func_offset(const void *_a, const void *_b)
+{
+	const struct elf_symbol_offset *a = _a;
+	const struct elf_symbol_offset *b = _b;
+
+	return strcmp(a->name, b->name);
+}
+
+static int
+__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
+			     const char **syms, unsigned long **poffsets)
+{
+	int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+	struct elf_symbol_offset *func_offs;
+	int err = 0, i, idx, cnt_done = 0;
+	unsigned long *offsets = NULL;
+
+	func_offs = calloc(cnt, sizeof(*func_offs));
+	if (!func_offs)
+		return -ENOMEM;
+
+	for (i = 0; i < cnt; i++) {
+		func_offs[i].name = syms[i];
+		func_offs[i].idx = i;
+	}
+
+	qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
+
+	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
+
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+			continue;
+
+		while ((sym = elf_symbol_iter_next(&iter))) {
+			struct elf_symbol_offset *fo, tmp = {
+				.name = sym->name,
+			};
+
+			fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
+				     cmp_func_offset);
+			if (!fo)
+				continue;
+
+			if (fo->offset > 0) {
+				/* same offset, no problem */
+				if (fo->offset == sym->offset)
+					continue;
+				/* handle multiple matches */
+				if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
+					/* Only accept one non-weak bind. */
+					pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
+						sym->name, fo->name, binary_path);
+					err = -LIBBPF_ERRNO__FORMAT;
+					goto out;
+				} else if (sym->bind == STB_WEAK) {
+					/* already have a non-weak bind, and
+					 * this is a weak bind, so ignore.
+					 */
+					continue;
+				}
+			}
+			if (!fo->offset)
+				cnt_done++;
+			fo->offset = sym->offset;
+			fo->bind = sym->bind;
+		}
+	}
+
+	if (cnt != cnt_done) {
+		err = -ENOENT;
+		goto out;
+	}
+	offsets = calloc(cnt, sizeof(*offsets));
+	if (!offsets) {
+		err = -ENOMEM;
+		goto out;
+	}
+	for (i = 0; i < cnt; i++) {
+		idx = func_offs[i].idx;
+		offsets[idx] = func_offs[i].offset;
+	}
+
+out:
+	*poffsets = offsets;
+	free(func_offs);
+	return err;
+}
+
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+			       const char **syms, unsigned long **poffsets)
+{
+	struct elf_fd elf_fd = {};
+	long ret = -ENOENT;
+
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
+		return ret;
+
+	ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);
+	close_elf(&elf_fd);
+	return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index e4d05662a96c..13d5c12fbd0b 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -577,4 +577,6 @@ static inline bool is_pow_of_2(size_t x)
 #define PROG_LOAD_ATTEMPTS 5
 int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
+int elf_find_multi_func_offset(const char *binary_path, int cnt,
+			       const char **syms, unsigned long **poffsets);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
-- 
2.41.0

[PATCHv2 bpf-next 09/24] libbpf: Add elf_find_pattern_func_offset function

[Jiri Olsa]

Adding elf_find_pattern_func_offset function that looks up
offsets for symbols specified by pattern argument.

The 'pattern' argument allows wildcards (*?' supported).

Offsets are returned in allocated array together with its
size and needs to be released by the caller.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  3 ++
 2 files changed, 81 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 1c310b718961..3e5c88caf5d5 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
 	return ret;
 }
 
+static int
+__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
+			       const char ***pnames, unsigned long **poffsets, size_t *pcnt)
+{
+	int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
+	struct elf_symbol_offset *func_offs = NULL;
+	unsigned long *offsets = NULL;
+	const char **names = NULL;
+	size_t func_offs_cnt = 0;
+	size_t func_offs_cap = 0;
+	int err = 0, i;
+
+	for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
+		struct elf_symbol_iter iter;
+		struct elf_symbol *sym;
+
+		if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
+			continue;
+
+		while ((sym = elf_symbol_iter_next(&iter))) {
+			if (!glob_match(sym->name, pattern))
+				continue;
+
+			err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
+						sizeof(*func_offs), func_offs_cnt + 1);
+			if (err)
+				goto out;
+
+			func_offs[func_offs_cnt].offset = sym->offset;
+			func_offs[func_offs_cnt].name = strdup(sym->name);
+			func_offs_cnt++;
+		}
+
+		/* If we found anything in the first symbol section,
+		 * do not search others to avoid duplicates.
+		 */
+		if (func_offs_cnt)
+			break;
+	}
+
+	offsets = calloc(func_offs_cnt, sizeof(*offsets));
+	names = calloc(func_offs_cnt, sizeof(*names));
+	if (!offsets || !names) {
+		free(offsets);
+		free(names);
+		err = -ENOMEM;
+		goto out;
+	}
+
+	for (i = 0; i < func_offs_cnt; i++) {
+		offsets[i] = func_offs[i].offset;
+		names[i] = func_offs[i].name;
+	}
+
+	*pnames = names;
+	*poffsets = offsets;
+	*pcnt = func_offs_cnt;
+out:
+	free(func_offs);
+	return err;
+}
+
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+				 const char ***pnames, unsigned long **poffsets,
+				 size_t *pcnt)
+{
+	struct elf_fd elf_fd = {};
+	long ret = -ENOENT;
+
+	ret = open_elf(binary_path, &elf_fd);
+	if (ret)
+		return ret;
+
+	ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);
+	close_elf(&elf_fd);
+	return ret;
+}
+
 /* Find offset of function name in ELF object specified by path. "name" matches
  * symbol name or name@@LIB for library functions.
  */
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index 13d5c12fbd0b..22b0834e7fe1 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
 
 int elf_find_multi_func_offset(const char *binary_path, int cnt,
 			       const char **syms, unsigned long **poffsets);
+int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
+				 const char ***pnames, unsigned long **poffsets,
+				 size_t *pcnt);
 #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
-- 
2.41.0

[PATCHv2 bpf-next 10/24] libbpf: Add bpf_link_create support for multi uprobes

[Jiri Olsa]

Adding new uprobe_multi struct to bpf_link_create_opts object
to pass multiple uprobe data to link_create attr uapi.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/bpf.c | 11 +++++++++++
 tools/lib/bpf/bpf.h | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
index ed86b37d8024..0fd35c91f50c 100644
--- a/tools/lib/bpf/bpf.c
+++ b/tools/lib/bpf/bpf.c
@@ -733,6 +733,17 @@ int bpf_link_create(int prog_fd, int target_fd,
 		if (!OPTS_ZEROED(opts, kprobe_multi))
 			return libbpf_err(-EINVAL);
 		break;
+	case BPF_TRACE_UPROBE_MULTI:
+		attr.link_create.uprobe_multi.flags = OPTS_GET(opts, uprobe_multi.flags, 0);
+		attr.link_create.uprobe_multi.cnt = OPTS_GET(opts, uprobe_multi.cnt, 0);
+		attr.link_create.uprobe_multi.path = ptr_to_u64(OPTS_GET(opts, uprobe_multi.path, 0));
+		attr.link_create.uprobe_multi.offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.offsets, 0));
+		attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
+		attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
+		attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
+		if (!OPTS_ZEROED(opts, uprobe_multi))
+			return libbpf_err(-EINVAL);
+		break;
 	case BPF_TRACE_FENTRY:
 	case BPF_TRACE_FEXIT:
 	case BPF_MODIFY_RETURN:
diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
index 9aa0ee473754..82979b4f2769 100644
--- a/tools/lib/bpf/bpf.h
+++ b/tools/lib/bpf/bpf.h
@@ -346,13 +346,22 @@ struct bpf_link_create_opts {
 			const unsigned long *addrs;
 			const __u64 *cookies;
 		} kprobe_multi;
+		struct {
+			__u32 flags;
+			__u32 cnt;
+			const char *path;
+			const unsigned long *offsets;
+			const unsigned long *ref_ctr_offsets;
+			const __u64 *cookies;
+			__u32 pid;
+		} uprobe_multi;
 		struct {
 			__u64 cookie;
 		} tracing;
 	};
 	size_t :0;
 };
-#define bpf_link_create_opts__last_field kprobe_multi.cookies
+#define bpf_link_create_opts__last_field uprobe_multi.pid
 
 LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
 			       enum bpf_attach_type attach_type,
-- 
2.41.0

[PATCHv2 bpf-next 11/24] libbpf: Add bpf_program__attach_uprobe_multi_opts function

[Jiri Olsa]

Adding bpf_program__attach_uprobe_multi_opts function that
allows to attach multiple uprobes with uprobe_multi link.

The user can specify uprobes with direct arguments:

  binary_path/func_pattern

or with struct bpf_uprobe_multi_opts opts argument fields:

  const char *path;
  const char **syms;
  const unsigned long *offsets;
  const unsigned long *ref_ctr_offsets;

User can specify 3 mutually exclusive set of incputs:

 1) use only binary_path/func_pattern aruments

 2) use only opts argument with allowed combinations of:
    path/offsets/ref_ctr_offsets/cookies/cnt

 3) use binary_path with allowed combinations of:
    syms/offsets/ref_ctr_offsets/cookies/cnt

Any other usage results in error.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf.h   |  31 +++++++++
 tools/lib/bpf/libbpf.map |   1 +
 3 files changed, 163 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 3e5c88caf5d5..d972cea4c658 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
 	return -ENOENT;
 }
 
+struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+				      pid_t pid,
+				      const char *binary_path,
+				      const char *func_pattern,
+				      const struct bpf_uprobe_multi_opts *opts)
+{
+	const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
+	LIBBPF_OPTS(bpf_link_create_opts, lopts);
+	unsigned long *resolved_offsets = NULL;
+	const char **resolved_symbols = NULL;
+	int err = 0, link_fd, prog_fd;
+	struct bpf_link *link = NULL;
+	char errmsg[STRERR_BUFSIZE];
+	const char *path, **syms;
+	char full_path[PATH_MAX];
+	const __u64 *cookies;
+	bool has_pattern;
+	bool retprobe;
+	size_t cnt;
+
+	if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
+		return libbpf_err_ptr(-EINVAL);
+
+	path = OPTS_GET(opts, path, NULL);
+	syms = OPTS_GET(opts, syms, NULL);
+	offsets = OPTS_GET(opts, offsets, NULL);
+	ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
+	cookies = OPTS_GET(opts, cookies, NULL);
+	cnt = OPTS_GET(opts, cnt, 0);
+
+	/*
+	 * User can specify 3 mutually exclusive set of incputs:
+	 *
+	 * 1) use only binary_path/func_pattern aruments
+	 *
+	 * 2) use only opts argument with allowed combinations of:
+	 *    path/offsets/ref_ctr_offsets/cookies/cnt
+	 *
+	 * 3) use binary_path with allowed combinations of:
+	 *    syms/offsets/ref_ctr_offsets/cookies/cnt
+	 *
+	 * Any other usage results in error.
+	 */
+
+	if (!binary_path && !func_pattern && !cnt)
+		return libbpf_err_ptr(-EINVAL);
+	if (func_pattern && !binary_path)
+		return libbpf_err_ptr(-EINVAL);
+
+	has_pattern = binary_path && func_pattern;
+
+	if (has_pattern) {
+		if (path || syms || offsets || ref_ctr_offsets || cookies || cnt)
+			return libbpf_err_ptr(-EINVAL);
+	} else {
+		if (!cnt)
+			return libbpf_err_ptr(-EINVAL);
+		if (!!path == !!binary_path)
+			return libbpf_err_ptr(-EINVAL);
+		if (!!syms == !!offsets)
+			return libbpf_err_ptr(-EINVAL);
+		if (path && syms)
+			return libbpf_err_ptr(-EINVAL);
+	}
+
+	if (has_pattern) {
+		if (!strchr(binary_path, '/')) {
+			err = resolve_full_path(binary_path, full_path, sizeof(full_path));
+			if (err) {
+				pr_warn("prog '%s': failed to resolve full path for '%s': %d\n",
+					prog->name, binary_path, err);
+				return libbpf_err_ptr(err);
+			}
+			binary_path = full_path;
+		}
+
+		err = elf_find_pattern_func_offset(binary_path, func_pattern,
+						   &resolved_symbols, &resolved_offsets,
+						   &cnt);
+		if (err < 0)
+			return libbpf_err_ptr(err);
+		offsets = resolved_offsets;
+	} else if (syms) {
+		err = elf_find_multi_func_offset(binary_path, cnt, syms, &resolved_offsets);
+		if (err < 0)
+			return libbpf_err_ptr(err);
+		offsets = resolved_offsets;
+	}
+
+	retprobe = OPTS_GET(opts, retprobe, false);
+
+	lopts.uprobe_multi.path = path ?: binary_path;
+	lopts.uprobe_multi.offsets = offsets;
+	lopts.uprobe_multi.ref_ctr_offsets = ref_ctr_offsets;
+	lopts.uprobe_multi.cookies = cookies;
+	lopts.uprobe_multi.cnt = cnt;
+	lopts.uprobe_multi.flags = retprobe ? BPF_F_UPROBE_MULTI_RETURN : 0;
+
+	if (pid == 0)
+		pid = getpid();
+	if (pid > 0)
+		lopts.uprobe_multi.pid = pid;
+
+	link = calloc(1, sizeof(*link));
+	if (!link) {
+		err = -ENOMEM;
+		goto error;
+	}
+	link->detach = &bpf_link__detach_fd;
+
+	prog_fd = bpf_program__fd(prog);
+	link_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &lopts);
+	if (link_fd < 0) {
+		err = -errno;
+		pr_warn("prog '%s': failed to attach: %s\n",
+			prog->name, libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+		goto error;
+	}
+	link->fd = link_fd;
+	free(resolved_offsets);
+	free(resolved_symbols);
+	return link;
+
+error:
+	free(resolved_offsets);
+	free(resolved_symbols);
+	free(link);
+	return libbpf_err_ptr(err);
+}
+
 LIBBPF_API struct bpf_link *
 bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
 				const char *binary_path, size_t func_offset,
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index 754da73c643b..b6ff7d69a1d7 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -529,6 +529,37 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
 				      const char *pattern,
 				      const struct bpf_kprobe_multi_opts *opts);
 
+struct bpf_uprobe_multi_opts {
+	/* size of this struct, for forward/backward compatibility */
+	size_t sz;
+	/* path to attach */
+	const char *path;
+	/* array of function symbols to attach */
+	const char **syms;
+	/* array of function addresses to attach */
+	const unsigned long *offsets;
+	/* array of refctr offsets to attach */
+	const unsigned long *ref_ctr_offsets;
+	/* array of user-provided values fetchable through bpf_get_attach_cookie */
+	const __u64 *cookies;
+	/* number of elements in syms/addrs/cookies arrays */
+	size_t cnt;
+	/* create return uprobes */
+	bool retprobe;
+	/* pid filter */
+	int pid;
+	size_t :0;
+};
+
+#define bpf_uprobe_multi_opts__last_field pid
+
+LIBBPF_API struct bpf_link *
+bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
+				      pid_t pid,
+				      const char *binary_path,
+				      const char *func_pattern,
+				      const struct bpf_uprobe_multi_opts *opts);
+
 struct bpf_ksyscall_opts {
 	/* size of this struct, for forward/backward compatibility */
 	size_t sz;
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index 7521a2fb7626..81558ef1bc38 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
 		bpf_link_get_info_by_fd;
 		bpf_map_get_info_by_fd;
 		bpf_prog_get_info_by_fd;
+		bpf_program__attach_uprobe_multi_opts;
 } LIBBPF_1.1.0;
 
 LIBBPF_1.3.0 {
-- 
2.41.0

[PATCHv2 bpf-next 12/24] libbpf: Add support for u[ret]probe.multi[.s] program sections

[Jiri Olsa]

Adding support for several uprobe_multi program sections
to allow auto attach of multi_uprobe programs.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index d972cea4c658..e42080258ec7 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
 
@@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("kprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kretprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
+	SEC_DEF("uprobe.multi+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+	SEC_DEF("uretprobe.multi+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
+	SEC_DEF("uprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
+	SEC_DEF("uretprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
 	SEC_DEF("ksyscall+",		KPROBE,	0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
@@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
 	return libbpf_get_error(*link);
 }
 
+static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
+{
+	char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	int n, ret = -EINVAL;
+
+	*link = NULL;
+
+	n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
+		   &probe_type, &binary_path, &func_name);
+	switch (n) {
+	case 1:
+		/* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
+		ret = 0;
+		break;
+	case 2:
+		pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",
+			prog->name, prog->sec_name);
+		break;
+	case 3:
+		opts.retprobe = strcmp(probe_type, "uretprobe.multi");
+		*link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
+		ret = libbpf_get_error(*link);
+		break;
+	default:
+		pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
+			prog->sec_name);
+		break;
+	}
+	free(probe_type);
+	free(binary_path);
+	free(func_name);
+	return ret;
+}
+
 static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
 					 const char *binary_path, uint64_t offset)
 {
-- 
2.41.0

[PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

[Jiri Olsa]

Adding uprobe-multi link detection. It will be used later in
bpf_program__attach_usdt function to check and use uprobe_multi
link over standard uprobe links.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 31 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index e42080258ec7..3d570898459e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
 	return link_fd < 0 && err == -EBADF;
 }
 
+static int probe_uprobe_multi_link(void)
+{
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, link_fd, err;
+
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+				insns, ARRAY_SIZE(insns), NULL);
+	if (prog_fd < 0)
+		return -errno;
+
+	/* No need to specify attach function. If the link is not supported
+	 * we will get -EOPNOTSUPP error before any other check is performed.
+	 */
+	link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
+	err = -errno; /* close() can clobber errno */
+
+	if (link_fd >= 0)
+		close(link_fd);
+	close(prog_fd);
+
+	return link_fd < 0 && err != -EOPNOTSUPP;
+}
+
 static int probe_kern_bpf_cookie(void)
 {
 	struct bpf_insn insns[] = {
@@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
 	[FEAT_SYSCALL_WRAPPER] = {
 		"Kernel using syscall wrapper", probe_kern_syscall_wrapper,
 	},
+	[FEAT_UPROBE_LINK] = {
+		"BPF uprobe multi link support", probe_uprobe_multi_link,
+	},
 };
 
 bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index 22b0834e7fe1..a257eb81af25 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -354,6 +354,8 @@ enum kern_feature_id {
 	FEAT_BTF_ENUM64,
 	/* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
 	FEAT_SYSCALL_WRAPPER,
+	/* BPF uprobe_multi link support */
+	FEAT_UPROBE_LINK,
 	__FEAT_CNT,
 };
 
-- 
2.41.0

[PATCHv2 bpf-next 14/24] libbpf: Add uprobe multi link support to bpf_program__attach_usdt

[Jiri Olsa]

Adding support for usdt_manager_attach_usdt to use uprobe_multi
link to attach to usdt probes.

The uprobe_multi support is detected before the usdt program is
loaded and its expected_attach_type is set accordingly.

If uprobe_multi support is detected the usdt_manager_attach_usdt
gathers uprobes info and calls bpf_program__attach_uprobe_opts to
create all needed uprobes.

If uprobe_multi support is not detected the old behaviour stays.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/bpf/libbpf.c |  12 ++++-
 tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
 2 files changed, 99 insertions(+), 33 deletions(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 3d570898459e..9c7a67c5cbe8 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -363,6 +363,8 @@ enum sec_def_flags {
 	SEC_SLEEPABLE = 8,
 	/* BPF program support non-linear XDP buffer */
 	SEC_XDP_FRAGS = 16,
+	/* Setup proper attach type for usdt probes. */
+	SEC_USDT = 32,
 };
 
 struct bpf_sec_def {
@@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
 	if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
 		opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
 
+	/* special check for usdt to use uprobe_multi link */
+	if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
+		prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;
+
 	if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
 		int btf_obj_fd = 0, btf_type_id = 0, err;
 		const char *attach_name;
@@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 	if (!insns || !insns_cnt)
 		return -EINVAL;
 
-	load_attr.expected_attach_type = prog->expected_attach_type;
 	if (kernel_supports(obj, FEAT_PROG_NAME))
 		prog_name = prog->name;
 	load_attr.attach_prog_fd = prog->attach_prog_fd;
@@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
 		insns_cnt = prog->insns_cnt;
 	}
 
+	/* allow prog_prepare_load_fn to change expected_attach_type */
+	load_attr.expected_attach_type = prog->expected_attach_type;
+
 	if (obj->gen_loader) {
 		bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
 				   license, insns, insns_cnt, &load_attr,
@@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("uretprobe.multi.s+",	KPROBE,	BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
 	SEC_DEF("ksyscall+",		KPROBE,	0, SEC_NONE, attach_ksyscall),
 	SEC_DEF("kretsyscall+",		KPROBE, 0, SEC_NONE, attach_ksyscall),
-	SEC_DEF("usdt+",		KPROBE,	0, SEC_NONE, attach_usdt),
+	SEC_DEF("usdt+",		KPROBE,	0, SEC_USDT, attach_usdt),
 	SEC_DEF("tc",			SCHED_CLS, 0, SEC_NONE),
 	SEC_DEF("classifier",		SCHED_CLS, 0, SEC_NONE),
 	SEC_DEF("action",		SCHED_ACT, 0, SEC_NONE),
diff --git a/tools/lib/bpf/usdt.c b/tools/lib/bpf/usdt.c
index f1a141555f08..33f0a2b4cc1c 100644
--- a/tools/lib/bpf/usdt.c
+++ b/tools/lib/bpf/usdt.c
@@ -808,6 +808,16 @@ struct bpf_link_usdt {
 		long abs_ip;
 		struct bpf_link *link;
 	} *uprobes;
+
+	bool has_uprobe_multi;
+
+	struct {
+		char *path;
+		unsigned long *offsets;
+		unsigned long *ref_ctr_offsets;
+		__u64 *cookies;
+		struct bpf_link *link;
+	} uprobe_multi;
 };
 
 static int bpf_link_usdt_detach(struct bpf_link *link)
@@ -816,19 +826,23 @@ static int bpf_link_usdt_detach(struct bpf_link *link)
 	struct usdt_manager *man = usdt_link->usdt_man;
 	int i;
 
-	for (i = 0; i < usdt_link->uprobe_cnt; i++) {
-		/* detach underlying uprobe link */
-		bpf_link__destroy(usdt_link->uprobes[i].link);
-		/* there is no need to update specs map because it will be
-		 * unconditionally overwritten on subsequent USDT attaches,
-		 * but if BPF cookies are not used we need to remove entry
-		 * from ip_to_spec_id map, otherwise we'll run into false
-		 * conflicting IP errors
-		 */
-		if (!man->has_bpf_cookie) {
-			/* not much we can do about errors here */
-			(void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
-						  &usdt_link->uprobes[i].abs_ip);
+	if (usdt_link->has_uprobe_multi) {
+		bpf_link__destroy(usdt_link->uprobe_multi.link);
+	} else {
+		for (i = 0; i < usdt_link->uprobe_cnt; i++) {
+			/* detach underlying uprobe link */
+			bpf_link__destroy(usdt_link->uprobes[i].link);
+			/* there is no need to update specs map because it will be
+			 * unconditionally overwritten on subsequent USDT attaches,
+			 * but if BPF cookies are not used we need to remove entry
+			 * from ip_to_spec_id map, otherwise we'll run into false
+			 * conflicting IP errors
+			 */
+			if (!man->has_bpf_cookie) {
+				/* not much we can do about errors here */
+				(void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
+							  &usdt_link->uprobes[i].abs_ip);
+			}
 		}
 	}
 
@@ -868,9 +882,15 @@ static void bpf_link_usdt_dealloc(struct bpf_link *link)
 {
 	struct bpf_link_usdt *usdt_link = container_of(link, struct bpf_link_usdt, link);
 
-	free(usdt_link->spec_ids);
-	free(usdt_link->uprobes);
-	free(usdt_link);
+	if (usdt_link->has_uprobe_multi) {
+		free(usdt_link->uprobe_multi.offsets);
+		free(usdt_link->uprobe_multi.ref_ctr_offsets);
+		free(usdt_link->uprobe_multi.cookies);
+	} else {
+		free(usdt_link->spec_ids);
+		free(usdt_link->uprobes);
+		free(usdt_link);
+	}
 }
 
 static size_t specs_hash_fn(long key, void *ctx)
@@ -943,11 +963,13 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
 					  const char *usdt_provider, const char *usdt_name,
 					  __u64 usdt_cookie)
 {
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts_multi);
 	int i, fd, err, spec_map_fd, ip_map_fd;
 	LIBBPF_OPTS(bpf_uprobe_opts, opts);
 	struct hashmap *specs_hash = NULL;
 	struct bpf_link_usdt *link = NULL;
 	struct usdt_target *targets = NULL;
+	struct bpf_link *uprobe_link;
 	size_t target_cnt;
 	Elf *elf;
 
@@ -1003,16 +1025,29 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
 	link->usdt_man = man;
 	link->link.detach = &bpf_link_usdt_detach;
 	link->link.dealloc = &bpf_link_usdt_dealloc;
+	link->has_uprobe_multi = bpf_program__expected_attach_type(prog) == BPF_TRACE_UPROBE_MULTI;
 
-	link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
-	if (!link->uprobes) {
-		err = -ENOMEM;
-		goto err_out;
+	if (link->has_uprobe_multi) {
+		link->uprobe_multi.offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.offsets));
+		link->uprobe_multi.ref_ctr_offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.ref_ctr_offsets));
+		link->uprobe_multi.cookies = calloc(target_cnt, sizeof(*link->uprobe_multi.cookies));
+
+		if (!link->uprobe_multi.offsets ||
+		    !link->uprobe_multi.ref_ctr_offsets ||
+		    !link->uprobe_multi.cookies) {
+			err = -ENOMEM;
+			goto err_out;
+		}
+	} else {
+		link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
+		if (!link->uprobes) {
+			err = -ENOMEM;
+			goto err_out;
+		}
 	}
 
 	for (i = 0; i < target_cnt; i++) {
 		struct usdt_target *target = &targets[i];
-		struct bpf_link *uprobe_link;
 		bool is_new;
 		int spec_id;
 
@@ -1048,20 +1083,43 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
 			goto err_out;
 		}
 
-		opts.ref_ctr_offset = target->sema_off;
-		opts.bpf_cookie = man->has_bpf_cookie ? spec_id : 0;
-		uprobe_link = bpf_program__attach_uprobe_opts(prog, pid, path,
-							      target->rel_ip, &opts);
+		if (link->has_uprobe_multi) {
+			link->uprobe_multi.offsets[i] = target->rel_ip;
+			link->uprobe_multi.ref_ctr_offsets[i] = target->sema_off;
+			link->uprobe_multi.cookies[i] = spec_id;
+		} else {
+			opts.ref_ctr_offset = target->sema_off;
+			opts.bpf_cookie = man->has_bpf_cookie ? spec_id : 0;
+			uprobe_link = bpf_program__attach_uprobe_opts(prog, pid, path,
+								      target->rel_ip, &opts);
+			err = libbpf_get_error(uprobe_link);
+			if (err) {
+				pr_warn("usdt: failed to attach uprobe #%d for '%s:%s' in '%s': %d\n",
+					i, usdt_provider, usdt_name, path, err);
+				goto err_out;
+			}
+
+			link->uprobes[i].link = uprobe_link;
+			link->uprobes[i].abs_ip = target->abs_ip;
+			link->uprobe_cnt++;
+		}
+	}
+
+	if (link->has_uprobe_multi) {
+		opts_multi.cnt = target_cnt;
+		opts_multi.path = path;
+		opts_multi.offsets = link->uprobe_multi.offsets;
+		opts_multi.ref_ctr_offsets = link->uprobe_multi.ref_ctr_offsets;
+		opts_multi.cookies = link->uprobe_multi.cookies;
+
+		uprobe_link = bpf_program__attach_uprobe_multi_opts(prog, pid, NULL, NULL, &opts_multi);
 		err = libbpf_get_error(uprobe_link);
 		if (err) {
-			pr_warn("usdt: failed to attach uprobe #%d for '%s:%s' in '%s': %d\n",
-				i, usdt_provider, usdt_name, path, err);
+			pr_warn("usdt: failed to attach uprobe multi for '%s:%s' in '%s': %d\n",
+				usdt_provider, usdt_name, path, err);
 			goto err_out;
 		}
-
-		link->uprobes[i].link = uprobe_link;
-		link->uprobes[i].abs_ip = target->abs_ip;
-		link->uprobe_cnt++;
+		link->uprobe_multi.link = uprobe_link;
 	}
 
 	free(targets);
-- 
2.41.0

[PATCHv2 bpf-next 15/24] selftests/bpf: Add uprobe_multi skel test

[Jiri Olsa]

Adding uprobe_multi test for skeleton load/attach functions,
to test skeleton auto attach for uprobe_multi link.

Test that bpf_get_func_ip works properly for uprobe_multi
attachment.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 76 ++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        | 91 +++++++++++++++++++
 2 files changed, 167 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi.c

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
new file mode 100644
index 000000000000..5cd1116bbb62
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <unistd.h>
+#include <test_progs.h>
+#include "uprobe_multi.skel.h"
+
+static char test_data[] = "test_data";
+
+noinline void uprobe_multi_func_1(void)
+{
+	asm volatile ("");
+}
+
+noinline void uprobe_multi_func_2(void)
+{
+	asm volatile ("");
+}
+
+noinline void uprobe_multi_func_3(void)
+{
+	asm volatile ("");
+}
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel)
+{
+	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
+	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
+	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
+
+	skel->bss->user_ptr = test_data;
+	skel->bss->pid = getpid();
+
+	/* trigger all probes */
+	uprobe_multi_func_1();
+	uprobe_multi_func_2();
+	uprobe_multi_func_3();
+
+	/*
+	 * There are 2 entry and 2 exit probe called for each uprobe_multi_func_[123]
+	 * function and each slepable probe (6) increments uprobe_multi_sleep_result.
+	 */
+	ASSERT_EQ(skel->bss->uprobe_multi_func_1_result, 2, "uprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_2_result, 2, "uprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_3_result, 2, "uprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_1_result, 2, "uretprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_2_result, 2, "uretprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 2, "uretprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uprobe_multi_sleep_result, 6, "uprobe_multi_sleep_result");
+}
+
+static void test_skel_api(void)
+{
+	struct uprobe_multi *skel = NULL;
+	int err;
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	err = uprobe_multi__attach(skel);
+	if (!ASSERT_OK(err, "uprobe_multi__attach"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	uprobe_multi__destroy(skel);
+}
+
+void test_uprobe_multi_test(void)
+{
+	if (test__start_subtest("skel_api"))
+		test_skel_api();
+}
diff --git a/tools/testing/selftests/bpf/progs/uprobe_multi.c b/tools/testing/selftests/bpf/progs/uprobe_multi.c
new file mode 100644
index 000000000000..1eeb9b7b9cad
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/uprobe_multi.c
@@ -0,0 +1,91 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <stdbool.h>
+
+char _license[] SEC("license") = "GPL";
+
+__u64 uprobe_multi_func_1_addr = 0;
+__u64 uprobe_multi_func_2_addr = 0;
+__u64 uprobe_multi_func_3_addr = 0;
+
+__u64 uprobe_multi_func_1_result = 0;
+__u64 uprobe_multi_func_2_result = 0;
+__u64 uprobe_multi_func_3_result = 0;
+
+__u64 uretprobe_multi_func_1_result = 0;
+__u64 uretprobe_multi_func_2_result = 0;
+__u64 uretprobe_multi_func_3_result = 0;
+
+__u64 uprobe_multi_sleep_result = 0;
+
+int pid = 0;
+bool test_cookie = false;
+void *user_ptr = 0;
+
+static __always_inline bool verify_sleepable_user_copy(void)
+{
+	char data[9];
+
+	bpf_copy_from_user(data, sizeof(data), user_ptr);
+	return bpf_strncmp(data, sizeof(data), "test_data") == 0;
+}
+
+static void uprobe_multi_check(void *ctx, bool is_return, bool is_sleep)
+{
+	if (bpf_get_current_pid_tgid() >> 32 != pid)
+		return;
+
+	__u64 cookie = test_cookie ? bpf_get_attach_cookie(ctx) : 0;
+	__u64 addr = bpf_get_func_ip(ctx);
+
+#define SET(__var, __addr, __cookie) ({			\
+	if (addr == __addr &&				\
+	   (!test_cookie || (cookie == __cookie)))	\
+		__var += 1;				\
+})
+
+	if (is_return) {
+		SET(uretprobe_multi_func_1_result, uprobe_multi_func_1_addr, 2);
+		SET(uretprobe_multi_func_2_result, uprobe_multi_func_2_addr, 3);
+		SET(uretprobe_multi_func_3_result, uprobe_multi_func_3_addr, 1);
+	} else {
+		SET(uprobe_multi_func_1_result, uprobe_multi_func_1_addr, 3);
+		SET(uprobe_multi_func_2_result, uprobe_multi_func_2_addr, 1);
+		SET(uprobe_multi_func_3_result, uprobe_multi_func_3_addr, 2);
+	}
+
+#undef SET
+
+	if (is_sleep && verify_sleepable_user_copy())
+		uprobe_multi_sleep_result += 1;
+}
+
+SEC("uprobe.multi//proc/self/exe:uprobe_multi_func_*")
+int test_uprobe(struct pt_regs *ctx)
+{
+	uprobe_multi_check(ctx, false, false);
+	return 0;
+}
+
+SEC("uretprobe.multi//proc/self/exe:uprobe_multi_func_*")
+int test_uretprobe(struct pt_regs *ctx)
+{
+	uprobe_multi_check(ctx, true, false);
+	return 0;
+}
+
+SEC("uprobe.multi.s//proc/self/exe:uprobe_multi_func_*")
+int test_uprobe_sleep(struct pt_regs *ctx)
+{
+	uprobe_multi_check(ctx, false, true);
+	return 0;
+}
+
+SEC("uretprobe.multi.s//proc/self/exe:uprobe_multi_func_*")
+int test_uretprobe_sleep(struct pt_regs *ctx)
+{
+	uprobe_multi_check(ctx, true, true);
+	return 0;
+}
-- 
2.41.0

[PATCHv2 bpf-next 16/24] selftests/bpf: Add uprobe_multi api test

[Jiri Olsa]

Adding uprobe_multi test for bpf_program__attach_uprobe_multi_opts
attach function.

Testing attachment using glob patterns and via bpf_uprobe_multi_opts
paths/syms fields.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 71 +++++++++++++++++++
 1 file changed, 71 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index 5cd1116bbb62..28b8e8d451fb 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -69,8 +69,79 @@ static void test_skel_api(void)
 	uprobe_multi__destroy(skel);
 }
 
+static void
+test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+{
+	struct bpf_link *link1 = NULL, *link2 = NULL;
+	struct bpf_link *link3 = NULL, *link4 = NULL;
+	struct uprobe_multi *skel = NULL;
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	opts->retprobe = false;
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	opts->retprobe = true;
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	opts->retprobe = false;
+	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	opts->retprobe = true;
+	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	bpf_link__destroy(link4);
+	bpf_link__destroy(link3);
+	bpf_link__destroy(link2);
+	bpf_link__destroy(link1);
+	uprobe_multi__destroy(skel);
+}
+
+static void test_attach_api_pattern(void)
+{
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+
+	test_attach_api("/proc/self/exe", "uprobe_multi_func_*", &opts);
+	test_attach_api("/proc/self/exe", "uprobe_multi_func_?", &opts);
+}
+
+static void test_attach_api_syms(void)
+{
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+
+	opts.syms = syms;
+	opts.cnt = ARRAY_SIZE(syms);
+	test_attach_api("/proc/self/exe", NULL, &opts);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
 		test_skel_api();
+	if (test__start_subtest("attach_api_pattern"))
+		test_attach_api_pattern();
+	if (test__start_subtest("attach_api_syms"))
+		test_attach_api_syms();
 }
-- 
2.41.0

[PATCHv2 bpf-next 17/24] selftests/bpf: Add uprobe_multi link test

[Jiri Olsa]

Adding uprobe_multi test for bpf_link_create attach function.

Testing attachment using the struct bpf_link_create_opts.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 68 +++++++++++++++++++
 1 file changed, 68 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index 28b8e8d451fb..fb5247a86559 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -3,6 +3,7 @@
 #include <unistd.h>
 #include <test_progs.h>
 #include "uprobe_multi.skel.h"
+#include "bpf/libbpf_internal.h"
 
 static char test_data[] = "test_data";
 
@@ -136,6 +137,71 @@ static void test_attach_api_syms(void)
 	test_attach_api("/proc/self/exe", NULL, &opts);
 }
 
+static void test_link_api(void)
+{
+	int prog_fd, link1_fd = -1, link2_fd = -1, link3_fd = -1, link4_fd = -1;
+	LIBBPF_OPTS(bpf_link_create_opts, opts);
+	const char *path = "/proc/self/exe";
+	struct uprobe_multi *skel = NULL;
+	unsigned long *offsets = NULL;
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+	int err;
+
+	err = elf_find_multi_func_offset(path, 3, syms, (unsigned long **) &offsets);
+	if (!ASSERT_OK(err, "elf_find_multi_func_offset"))
+		return;
+
+	opts.uprobe_multi.path = path;
+	opts.uprobe_multi.offsets = offsets;
+	opts.uprobe_multi.cnt = ARRAY_SIZE(syms);
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe);
+	link1_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link1_fd, 0, "link1_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = BPF_F_UPROBE_MULTI_RETURN;
+	prog_fd = bpf_program__fd(skel->progs.test_uretprobe);
+	link2_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link2_fd, 0, "link2_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe_sleep);
+	link3_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link1_fd, 0, "link3_fd"))
+		goto cleanup;
+
+	opts.kprobe_multi.flags = BPF_F_UPROBE_MULTI_RETURN;
+	prog_fd = bpf_program__fd(skel->progs.test_uretprobe_sleep);
+	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
+		goto cleanup;
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	if (link1_fd >= 0)
+		close(link1_fd);
+	if (link2_fd >= 0)
+		close(link2_fd);
+	if (link3_fd >= 0)
+		close(link3_fd);
+	if (link4_fd >= 0)
+		close(link4_fd);
+
+	uprobe_multi__destroy(skel);
+	free(offsets);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -144,4 +210,6 @@ void test_uprobe_multi_test(void)
 		test_attach_api_pattern();
 	if (test__start_subtest("attach_api_syms"))
 		test_attach_api_syms();
+	if (test__start_subtest("link_api"))
+		test_link_api();
 }
-- 
2.41.0

[PATCHv2 bpf-next 18/24] selftests/bpf: Add uprobe_multi test program

[Jiri Olsa]

Adding uprobe_multi test program that defines 50k uprobe_multi_func_*
functions and will serve as attach point for uprobe_multi bench test
in following patch.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/testing/selftests/bpf/Makefile       |  5 ++
 tools/testing/selftests/bpf/uprobe_multi.c | 53 ++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/uprobe_multi.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 538df8fb8c42..26ecdd117f71 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -567,6 +567,7 @@ TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/liburandom_read.so			\
 		       $(OUTPUT)/xdp_synproxy				\
 		       $(OUTPUT)/sign-file				\
+		       $(OUTPUT)/uprobe_multi				\
 		       ima_setup.sh 					\
 		       verify_sig_setup.sh				\
 		       $(wildcard progs/btf_dump_test_case_*.c)		\
@@ -670,6 +671,10 @@ $(OUTPUT)/veristat: $(OUTPUT)/veristat.o
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $(filter %.a %.o,$^) $(LDLIBS) -o $@
 
+$(OUTPUT)/uprobe_multi: uprobe_multi.c
+	$(call msg,BINARY,,$@)
+	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
 EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR)	\
 	prog_tests/tests.h map_tests/tests.h verifier/tests.h		\
 	feature bpftool							\
diff --git a/tools/testing/selftests/bpf/uprobe_multi.c b/tools/testing/selftests/bpf/uprobe_multi.c
new file mode 100644
index 000000000000..115a7f6cebfa
--- /dev/null
+++ b/tools/testing/selftests/bpf/uprobe_multi.c
@@ -0,0 +1,53 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <stdio.h>
+
+#define __PASTE(a, b) a##b
+#define PASTE(a, b) __PASTE(a, b)
+
+#define NAME(name, idx) PASTE(name, idx)
+
+#define DEF(name, idx)  int NAME(name, idx)(void) { return 0; }
+#define CALL(name, idx) NAME(name, idx)();
+
+#define F(body, name, idx) body(name, idx)
+
+#define F10(body, name, idx) \
+	F(body, PASTE(name, idx), 0) F(body, PASTE(name, idx), 1) F(body, PASTE(name, idx), 2) \
+	F(body, PASTE(name, idx), 3) F(body, PASTE(name, idx), 4) F(body, PASTE(name, idx), 5) \
+	F(body, PASTE(name, idx), 6) F(body, PASTE(name, idx), 7) F(body, PASTE(name, idx), 8) \
+	F(body, PASTE(name, idx), 9)
+
+#define F100(body, name, idx) \
+	F10(body, PASTE(name, idx), 0) F10(body, PASTE(name, idx), 1) F10(body, PASTE(name, idx), 2) \
+	F10(body, PASTE(name, idx), 3) F10(body, PASTE(name, idx), 4) F10(body, PASTE(name, idx), 5) \
+	F10(body, PASTE(name, idx), 6) F10(body, PASTE(name, idx), 7) F10(body, PASTE(name, idx), 8) \
+	F10(body, PASTE(name, idx), 9)
+
+#define F1000(body, name, idx) \
+	F100(body, PASTE(name, idx), 0) F100(body, PASTE(name, idx), 1) F100(body, PASTE(name, idx), 2) \
+	F100(body, PASTE(name, idx), 3) F100(body, PASTE(name, idx), 4) F100(body, PASTE(name, idx), 5) \
+	F100(body, PASTE(name, idx), 6) F100(body, PASTE(name, idx), 7) F100(body, PASTE(name, idx), 8) \
+	F100(body, PASTE(name, idx), 9)
+
+#define F10000(body, name, idx) \
+	F1000(body, PASTE(name, idx), 0) F1000(body, PASTE(name, idx), 1) F1000(body, PASTE(name, idx), 2) \
+	F1000(body, PASTE(name, idx), 3) F1000(body, PASTE(name, idx), 4) F1000(body, PASTE(name, idx), 5) \
+	F1000(body, PASTE(name, idx), 6) F1000(body, PASTE(name, idx), 7) F1000(body, PASTE(name, idx), 8) \
+	F1000(body, PASTE(name, idx), 9)
+
+F10000(DEF, uprobe_multi_func_, 0)
+F10000(DEF, uprobe_multi_func_, 1)
+F10000(DEF, uprobe_multi_func_, 2)
+F10000(DEF, uprobe_multi_func_, 3)
+F10000(DEF, uprobe_multi_func_, 4)
+
+int main(void)
+{
+	F10000(CALL, uprobe_multi_func_, 0)
+	F10000(CALL, uprobe_multi_func_, 1)
+	F10000(CALL, uprobe_multi_func_, 2)
+	F10000(CALL, uprobe_multi_func_, 3)
+	F10000(CALL, uprobe_multi_func_, 4)
+	return 0;
+}
-- 
2.41.0

[PATCHv2 bpf-next 19/24] selftests/bpf: Add uprobe_multi bench test

[Jiri Olsa]

Adding test that attaches 50k uprobes in uprobe_multi binary.

After the attach is done we run the binary and make sure we
get proper amount of hits.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 56 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        |  9 +++
 2 files changed, 65 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index fb5247a86559..9a34c754f1ad 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -202,6 +202,60 @@ static void test_link_api(void)
 	free(offsets);
 }
 
+static inline __u64 get_time_ns(void)
+{
+	struct timespec t;
+
+	clock_gettime(CLOCK_MONOTONIC, &t);
+	return (__u64) t.tv_sec * 1000000000 + t.tv_nsec;
+}
+
+static void test_bench_attach_uprobe(void)
+{
+	long attach_start_ns, attach_end_ns;
+	long detach_start_ns, detach_end_ns;
+	double attach_delta, detach_delta;
+	struct uprobe_multi *skel = NULL;
+	struct bpf_program *prog;
+	int err;
+
+	skel = uprobe_multi__open();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open"))
+		goto cleanup;
+
+	bpf_object__for_each_program(prog, skel->obj)
+		bpf_program__set_autoload(prog, false);
+
+	bpf_program__set_autoload(skel->progs.test_uprobe_bench, true);
+
+	err = uprobe_multi__load(skel);
+	if (!ASSERT_EQ(err, 0, "uprobe_multi__load"))
+		goto cleanup;
+
+	attach_start_ns = get_time_ns();
+
+	err = uprobe_multi__attach(skel);
+	if (!ASSERT_OK(err, "uprobe_multi__attach"))
+		goto cleanup;
+
+	attach_end_ns = get_time_ns();
+
+	system("./uprobe_multi");
+
+	ASSERT_EQ(skel->bss->count, 50000, "uprobes_count");
+
+cleanup:
+	detach_start_ns = get_time_ns();
+	uprobe_multi__destroy(skel);
+	detach_end_ns = get_time_ns();
+
+	attach_delta = (attach_end_ns - attach_start_ns) / 1000000000.0;
+	detach_delta = (detach_end_ns - detach_start_ns) / 1000000000.0;
+
+	printf("%s: attached in %7.3lfs\n", __func__, attach_delta);
+	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -212,4 +266,6 @@ void test_uprobe_multi_test(void)
 		test_attach_api_syms();
 	if (test__start_subtest("link_api"))
 		test_link_api();
+	if (test__start_subtest("bench_uprobe"))
+		test_bench_attach_uprobe();
 }
diff --git a/tools/testing/selftests/bpf/progs/uprobe_multi.c b/tools/testing/selftests/bpf/progs/uprobe_multi.c
index 1eeb9b7b9cad..cd73139dc881 100644
--- a/tools/testing/selftests/bpf/progs/uprobe_multi.c
+++ b/tools/testing/selftests/bpf/progs/uprobe_multi.c
@@ -89,3 +89,12 @@ int test_uretprobe_sleep(struct pt_regs *ctx)
 	uprobe_multi_check(ctx, true, true);
 	return 0;
 }
+
+int count;
+
+SEC("?uprobe.multi/./uprobe_multi:uprobe_multi_func_*")
+int test_uprobe_bench(struct pt_regs *ctx)
+{
+	count++;
+	return 0;
+}
-- 
2.41.0

[PATCHv2 bpf-next 20/24] selftests/bpf: Add usdt_multi test program

[Jiri Olsa]

Adding usdt_multi test program that defines 50k usdts and will
serve as attach point for uprobe_multi usdt bench test in
following patch.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/testing/selftests/bpf/Makefile     |  5 +++++
 tools/testing/selftests/bpf/usdt_multi.c | 24 ++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/usdt_multi.c

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 26ecdd117f71..135b909f7bc9 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -568,6 +568,7 @@ TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko	\
 		       $(OUTPUT)/xdp_synproxy				\
 		       $(OUTPUT)/sign-file				\
 		       $(OUTPUT)/uprobe_multi				\
+		       $(OUTPUT)/usdt_multi				\
 		       ima_setup.sh 					\
 		       verify_sig_setup.sh				\
 		       $(wildcard progs/btf_dump_test_case_*.c)		\
@@ -675,6 +676,10 @@ $(OUTPUT)/uprobe_multi: uprobe_multi.c
 	$(call msg,BINARY,,$@)
 	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
 
+$(OUTPUT)/usdt_multi: usdt_multi.c
+	$(call msg,BINARY,,$@)
+	$(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@
+
 EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR)	\
 	prog_tests/tests.h map_tests/tests.h verifier/tests.h		\
 	feature bpftool							\
diff --git a/tools/testing/selftests/bpf/usdt_multi.c b/tools/testing/selftests/bpf/usdt_multi.c
new file mode 100644
index 000000000000..fedf856bad2b
--- /dev/null
+++ b/tools/testing/selftests/bpf/usdt_multi.c
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <sdt.h>
+
+#define PROBE STAP_PROBE(test, usdt);
+
+#define PROBE10    PROBE PROBE PROBE PROBE PROBE \
+		   PROBE PROBE PROBE PROBE PROBE
+#define PROBE100   PROBE10 PROBE10 PROBE10 PROBE10 PROBE10 \
+		   PROBE10 PROBE10 PROBE10 PROBE10 PROBE10
+#define PROBE1000  PROBE100 PROBE100 PROBE100 PROBE100 PROBE100 \
+		   PROBE100 PROBE100 PROBE100 PROBE100 PROBE100
+#define PROBE10000 PROBE1000 PROBE1000 PROBE1000 PROBE1000 PROBE1000 \
+		   PROBE1000 PROBE1000 PROBE1000 PROBE1000 PROBE1000
+
+int main(void)
+{
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	PROBE10000
+	return 0;
+}
-- 
2.41.0

[PATCHv2 bpf-next 21/24] selftests/bpf: Add usdt_multi bench test

[Jiri Olsa]

Adding test that attaches 50k usdt probes in usdt_multi binary.

After the attach is done we run the binary and make sure we get
proper amount of hits.

With current uprobes:

  # perf stat --null ./test_progs -n 254/6
  #254/6   uprobe_multi_test/bench_usdt:OK
  #254     uprobe_multi_test:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED

   Performance counter stats for './test_progs -n 254/6':

      1353.659680562 seconds time elapsed

With uprobe_multi link:

  # perf stat --null ./test_progs -n 254/6
  #254/6   uprobe_multi_test/bench_usdt:OK
  #254     uprobe_multi_test:OK
  Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED

   Performance counter stats for './test_progs -n 254/6':

         0.322046364 seconds time elapsed

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 50 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi_usdt.c   | 16 ++++++
 2 files changed, 66 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index 9a34c754f1ad..a9d5a1773407 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -4,6 +4,7 @@
 #include <test_progs.h>
 #include "uprobe_multi.skel.h"
 #include "bpf/libbpf_internal.h"
+#include "uprobe_multi_usdt.skel.h"
 
 static char test_data[] = "test_data";
 
@@ -256,6 +257,53 @@ static void test_bench_attach_uprobe(void)
 	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
 }
 
+static void test_bench_attach_usdt(void)
+{
+	struct uprobe_multi_usdt *skel = NULL;
+	long attach_start_ns, attach_end_ns;
+	long detach_start_ns, detach_end_ns;
+	double attach_delta, detach_delta;
+	struct bpf_program *prog;
+	int err;
+
+	skel = uprobe_multi_usdt__open();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open"))
+		goto cleanup;
+
+	bpf_object__for_each_program(prog, skel->obj)
+		bpf_program__set_autoload(prog, false);
+
+	bpf_program__set_autoload(skel->progs.usdt0, true);
+
+	err = uprobe_multi_usdt__load(skel);
+	if (!ASSERT_EQ(err, 0, "uprobe_multi_usdt__load"))
+		goto cleanup;
+
+	attach_start_ns = get_time_ns();
+
+	skel->links.usdt0 = bpf_program__attach_usdt(skel->progs.usdt0, -1, "./usdt_multi",
+						     "test", "usdt", NULL);
+	if (!ASSERT_OK_PTR(skel->links.usdt0, "bpf_program__attach_usdt"))
+		goto cleanup;
+
+	attach_end_ns = get_time_ns();
+
+	system("./usdt_multi");
+
+	ASSERT_EQ(skel->bss->count, 50000, "usdt_count");
+
+cleanup:
+	detach_start_ns = get_time_ns();
+	uprobe_multi_usdt__destroy(skel);
+	detach_end_ns = get_time_ns();
+
+	attach_delta = (attach_end_ns - attach_start_ns) / 1000000000.0;
+	detach_delta = (detach_end_ns - detach_start_ns) / 1000000000.0;
+
+	printf("%s: attached in %7.3lfs\n", __func__, attach_delta);
+	printf("%s: detached in %7.3lfs\n", __func__, detach_delta);
+}
+
 void test_uprobe_multi_test(void)
 {
 	if (test__start_subtest("skel_api"))
@@ -268,4 +316,6 @@ void test_uprobe_multi_test(void)
 		test_link_api();
 	if (test__start_subtest("bench_uprobe"))
 		test_bench_attach_uprobe();
+	if (test__start_subtest("bench_usdt"))
+		test_bench_attach_usdt();
 }
diff --git a/tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c b/tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c
new file mode 100644
index 000000000000..9e1c33d0bd2f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/uprobe_multi_usdt.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/usdt.bpf.h>
+
+char _license[] SEC("license") = "GPL";
+
+int count;
+
+SEC("usdt")
+int usdt0(struct pt_regs *ctx)
+{
+	count++;
+	return 0;
+}
-- 
2.41.0

[PATCHv2 bpf-next 22/24] selftests/bpf: Add uprobe_multi cookie test

[Jiri Olsa]

Adding test for cookies setup/retrieval in uprobe_link uprobes
and making sure bpf_get_attach_cookie works properly.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../selftests/bpf/prog_tests/bpf_cookie.c     | 78 +++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c
index 26b2d1bffdfd..47f692a64cef 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_cookie.c
@@ -11,6 +11,7 @@
 #include <bpf/btf.h>
 #include "test_bpf_cookie.skel.h"
 #include "kprobe_multi.skel.h"
+#include "uprobe_multi.skel.h"
 
 /* uprobe attach point */
 static noinline void trigger_func(void)
@@ -239,6 +240,81 @@ static void kprobe_multi_attach_api_subtest(void)
 	bpf_link__destroy(link1);
 	kprobe_multi__destroy(skel);
 }
+
+/* defined in prog_tests/uprobe_multi_test.c */
+void uprobe_multi_func_1(void);
+void uprobe_multi_func_2(void);
+void uprobe_multi_func_3(void);
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel)
+{
+	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
+	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
+	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
+
+	skel->bss->pid = getpid();
+	skel->bss->test_cookie = true;
+
+	uprobe_multi_func_1();
+	uprobe_multi_func_2();
+	uprobe_multi_func_3();
+
+	ASSERT_EQ(skel->bss->uprobe_multi_func_1_result, 1, "uprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_2_result, 1, "uprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uprobe_multi_func_3_result, 1, "uprobe_multi_func_3_result");
+
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_1_result, 1, "uretprobe_multi_func_1_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_2_result, 1, "uretprobe_multi_func_2_result");
+	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 1, "uretprobe_multi_func_3_result");
+}
+
+static void uprobe_multi_attach_api_subtest(void)
+{
+	struct bpf_link *link1 = NULL, *link2 = NULL;
+	struct uprobe_multi *skel = NULL;
+	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
+	const char *syms[3] = {
+		"uprobe_multi_func_1",
+		"uprobe_multi_func_2",
+		"uprobe_multi_func_3",
+	};
+	__u64 cookies[3];
+
+	cookies[0] = 3; /* uprobe_multi_func_1 */
+	cookies[1] = 1; /* uprobe_multi_func_2 */
+	cookies[2] = 2; /* uprobe_multi_func_3 */
+
+	opts.syms = syms;
+	opts.cnt = ARRAY_SIZE(syms);
+	opts.cookies = &cookies[0];
+
+	skel = uprobe_multi__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "uprobe_multi"))
+		goto cleanup;
+
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+						      "/proc/self/exe", NULL, &opts);
+	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
+	cookies[0] = 2; /* uprobe_multi_func_1 */
+	cookies[1] = 3; /* uprobe_multi_func_2 */
+	cookies[2] = 1; /* uprobe_multi_func_3 */
+
+	opts.retprobe = true;
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+						      "/proc/self/exe", NULL, &opts);
+	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
+		goto cleanup;
+
+	uprobe_multi_test_run(skel);
+
+cleanup:
+	bpf_link__destroy(link2);
+	bpf_link__destroy(link1);
+	uprobe_multi__destroy(skel);
+}
+
 static void uprobe_subtest(struct test_bpf_cookie *skel)
 {
 	DECLARE_LIBBPF_OPTS(bpf_uprobe_opts, opts);
@@ -515,6 +591,8 @@ void test_bpf_cookie(void)
 		kprobe_multi_attach_api_subtest();
 	if (test__start_subtest("uprobe"))
 		uprobe_subtest(skel);
+	if (test__start_subtest("multi_uprobe_attach_api"))
+		uprobe_multi_attach_api_subtest();
 	if (test__start_subtest("tracepoint"))
 		tp_subtest(skel);
 	if (test__start_subtest("perf_event"))
-- 
2.41.0

[PATCHv2 bpf-next 23/24] selftests/bpf: Add uprobe_multi pid filter tests

[Jiri Olsa]

Running api and link tests also with pid filter and checking
the probe gets executed only for specific pid.

Spawning extra process to trigger attached uprobes and checking
we get correct counts from executed programs.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 131 ++++++++++++++++--
 .../selftests/bpf/progs/uprobe_multi.c        |   6 +-
 2 files changed, 125 insertions(+), 12 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index a9d5a1773407..efa7b5d49940 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -23,14 +23,86 @@ noinline void uprobe_multi_func_3(void)
 	asm volatile ("");
 }
 
-static void uprobe_multi_test_run(struct uprobe_multi *skel)
+struct child {
+	int go[2];
+	int pid;
+};
+
+static void release_child(struct child *child)
+{
+	int child_status;
+
+	if (!child)
+		return;
+	close(child->go[1]);
+	close(child->go[0]);
+	if (child->pid > 0)
+		waitpid(child->pid, &child_status, 0);
+}
+
+static void kick_child(struct child *child)
+{
+	char c = 1;
+
+	if (child) {
+		write(child->go[1], &c, 1);
+		release_child(child);
+	}
+	fflush(NULL);
+}
+
+static struct child *spawn_child(void)
+{
+	static struct child child;
+	int err;
+	int c;
+
+	/* pid filter */
+	if (!ASSERT_OK(pipe(child.go), "pipe"))
+		return NULL;
+
+	child.pid = fork();
+	if (child.pid < 0) {
+		release_child(&child);
+		return NULL;
+	}
+
+	/* child */
+	if (child.pid == 0) {
+		close(child.go[1]);
+		fflush(NULL);
+		/* wait for parent's kick */
+		err = read(child.go[0], &c, 1);
+		if (!ASSERT_EQ(err, 1, "child_read_pipe"))
+			exit(err);
+
+		uprobe_multi_func_1();
+		uprobe_multi_func_2();
+		uprobe_multi_func_3();
+
+		exit(errno);
+	}
+
+	return &child;
+}
+
+static void uprobe_multi_test_run(struct uprobe_multi *skel, struct child *child)
 {
 	skel->bss->uprobe_multi_func_1_addr = (__u64) uprobe_multi_func_1;
 	skel->bss->uprobe_multi_func_2_addr = (__u64) uprobe_multi_func_2;
 	skel->bss->uprobe_multi_func_3_addr = (__u64) uprobe_multi_func_3;
 
 	skel->bss->user_ptr = test_data;
-	skel->bss->pid = getpid();
+
+	/*
+	 * Disable pid check in bpf program if we are pid filter test,
+	 * because the probe should be executed only by child->pid
+	 * passed at the probe attach.
+	 */
+	skel->bss->pid = child ? 0 : getpid();
+
+	if (child)
+		kick_child(child);
 
 	/* trigger all probes */
 	uprobe_multi_func_1();
@@ -50,6 +122,9 @@ static void uprobe_multi_test_run(struct uprobe_multi *skel)
 	ASSERT_EQ(skel->bss->uretprobe_multi_func_3_result, 2, "uretprobe_multi_func_3_result");
 
 	ASSERT_EQ(skel->bss->uprobe_multi_sleep_result, 6, "uprobe_multi_sleep_result");
+
+	if (child)
+		ASSERT_EQ(skel->bss->child_pid, child->pid, "uprobe_multi_child_pid");
 }
 
 static void test_skel_api(void)
@@ -65,17 +140,19 @@ static void test_skel_api(void)
 	if (!ASSERT_OK(err, "uprobe_multi__attach"))
 		goto cleanup;
 
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, NULL);
 
 cleanup:
 	uprobe_multi__destroy(skel);
 }
 
 static void
-test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+__test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts,
+		  struct child *child)
 {
 	struct bpf_link *link1 = NULL, *link2 = NULL;
 	struct bpf_link *link3 = NULL, *link4 = NULL;
+	pid_t pid = child ? child->pid : -1;
 	struct uprobe_multi *skel = NULL;
 
 	skel = uprobe_multi__open_and_load();
@@ -83,30 +160,30 @@ test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi
 		goto cleanup;
 
 	opts->retprobe = false;
-	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, -1,
+	link1 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
 		goto cleanup;
 
 	opts->retprobe = true;
-	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, -1,
+	link2 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
 	opts->retprobe = false;
-	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, -1,
+	link3 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_sleep, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link1, "bpf_program__attach_uprobe_multi_opts"))
 		goto cleanup;
 
 	opts->retprobe = true;
-	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, -1,
+	link4 = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uretprobe_sleep, pid,
 						      binary, pattern, opts);
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, child);
 
 cleanup:
 	bpf_link__destroy(link4);
@@ -116,6 +193,22 @@ test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi
 	uprobe_multi__destroy(skel);
 }
 
+static void
+test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_multi_opts *opts)
+{
+	struct child *child;
+
+	/* no pid filter */
+	__test_attach_api(binary, pattern, opts, NULL);
+
+	/* pid filter */
+	child = spawn_child();
+	if (!child)
+		return;
+
+	__test_attach_api(binary, pattern, opts, child);
+}
+
 static void test_attach_api_pattern(void)
 {
 	LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
@@ -138,7 +231,7 @@ static void test_attach_api_syms(void)
 	test_attach_api("/proc/self/exe", NULL, &opts);
 }
 
-static void test_link_api(void)
+static void __test_link_api(struct child *child)
 {
 	int prog_fd, link1_fd = -1, link2_fd = -1, link3_fd = -1, link4_fd = -1;
 	LIBBPF_OPTS(bpf_link_create_opts, opts);
@@ -159,6 +252,7 @@ static void test_link_api(void)
 	opts.uprobe_multi.path = path;
 	opts.uprobe_multi.offsets = offsets;
 	opts.uprobe_multi.cnt = ARRAY_SIZE(syms);
+	opts.uprobe_multi.pid = child ? child->pid : 0;
 
 	skel = uprobe_multi__open_and_load();
 	if (!ASSERT_OK_PTR(skel, "uprobe_multi__open_and_load"))
@@ -187,7 +281,7 @@ static void test_link_api(void)
 	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
 	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
 		goto cleanup;
-	uprobe_multi_test_run(skel);
+	uprobe_multi_test_run(skel, child);
 
 cleanup:
 	if (link1_fd >= 0)
@@ -203,6 +297,21 @@ static void test_link_api(void)
 	free(offsets);
 }
 
+void test_link_api(void)
+{
+	struct child *child;
+
+	/* no pid filter */
+	__test_link_api(NULL);
+
+	/* pid filter */
+	child = spawn_child();
+	if (!child)
+		return;
+
+	__test_link_api(child);
+}
+
 static inline __u64 get_time_ns(void)
 {
 	struct timespec t;
diff --git a/tools/testing/selftests/bpf/progs/uprobe_multi.c b/tools/testing/selftests/bpf/progs/uprobe_multi.c
index cd73139dc881..1086312b5d1e 100644
--- a/tools/testing/selftests/bpf/progs/uprobe_multi.c
+++ b/tools/testing/selftests/bpf/progs/uprobe_multi.c
@@ -21,6 +21,8 @@ __u64 uretprobe_multi_func_3_result = 0;
 __u64 uprobe_multi_sleep_result = 0;
 
 int pid = 0;
+int child_pid = 0;
+
 bool test_cookie = false;
 void *user_ptr = 0;
 
@@ -34,7 +36,9 @@ static __always_inline bool verify_sleepable_user_copy(void)
 
 static void uprobe_multi_check(void *ctx, bool is_return, bool is_sleep)
 {
-	if (bpf_get_current_pid_tgid() >> 32 != pid)
+	child_pid = bpf_get_current_pid_tgid() >> 32;
+
+	if (pid && child_pid != pid)
 		return;
 
 	__u64 cookie = test_cookie ? bpf_get_attach_cookie(ctx) : 0;
-- 
2.41.0

[PATCHv2 bpf-next 24/24] selftests/bpf: Add extra link to uprobe_multi tests

[Jiri Olsa]

Attaching extra program to same functions system wide for api
and link tests.

This way we can test the pid filter works properly when there's
extra system wide consumer on the same uprobe that will trigger
the original uprobe handler.

We expect to have the same counts as before.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../bpf/prog_tests/uprobe_multi_test.c        | 19 +++++++++++++++++++
 .../selftests/bpf/progs/uprobe_multi.c        |  6 ++++++
 2 files changed, 25 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
index efa7b5d49940..9057a228c12d 100644
--- a/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/uprobe_multi_test.c
@@ -153,6 +153,7 @@ __test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_mul
 	struct bpf_link *link1 = NULL, *link2 = NULL;
 	struct bpf_link *link3 = NULL, *link4 = NULL;
 	pid_t pid = child ? child->pid : -1;
+	struct bpf_link *link_extra = NULL;
 	struct uprobe_multi *skel = NULL;
 
 	skel = uprobe_multi__open_and_load();
@@ -183,9 +184,16 @@ __test_attach_api(const char *binary, const char *pattern, struct bpf_uprobe_mul
 	if (!ASSERT_OK_PTR(link2, "bpf_program__attach_uprobe_multi_opts_retprobe"))
 		goto cleanup;
 
+	opts->retprobe = false;
+	link_extra = bpf_program__attach_uprobe_multi_opts(skel->progs.test_uprobe_extra, -1,
+						      binary, pattern, opts);
+	if (!ASSERT_OK_PTR(link_extra, "bpf_program__attach_uprobe_multi_opts"))
+		goto cleanup;
+
 	uprobe_multi_test_run(skel, child);
 
 cleanup:
+	bpf_link__destroy(link_extra);
 	bpf_link__destroy(link4);
 	bpf_link__destroy(link3);
 	bpf_link__destroy(link2);
@@ -243,6 +251,7 @@ static void __test_link_api(struct child *child)
 		"uprobe_multi_func_2",
 		"uprobe_multi_func_3",
 	};
+	int link_extra_fd = -1;
 	int err;
 
 	err = elf_find_multi_func_offset(path, 3, syms, (unsigned long **) &offsets);
@@ -281,6 +290,14 @@ static void __test_link_api(struct child *child)
 	link4_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
 	if (!ASSERT_GE(link2_fd, 0, "link4_fd"))
 		goto cleanup;
+
+	opts.kprobe_multi.flags = 0;
+	opts.uprobe_multi.pid = 0;
+	prog_fd = bpf_program__fd(skel->progs.test_uprobe_extra);
+	link_extra_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_UPROBE_MULTI, &opts);
+	if (!ASSERT_GE(link_extra_fd, 0, "link_extra_fd"))
+		goto cleanup;
+
 	uprobe_multi_test_run(skel, child);
 
 cleanup:
@@ -292,6 +309,8 @@ static void __test_link_api(struct child *child)
 		close(link3_fd);
 	if (link4_fd >= 0)
 		close(link4_fd);
+	if (link_extra_fd >= 0)
+		close(link_extra_fd);
 
 	uprobe_multi__destroy(skel);
 	free(offsets);
diff --git a/tools/testing/selftests/bpf/progs/uprobe_multi.c b/tools/testing/selftests/bpf/progs/uprobe_multi.c
index 1086312b5d1e..8ee957670303 100644
--- a/tools/testing/selftests/bpf/progs/uprobe_multi.c
+++ b/tools/testing/selftests/bpf/progs/uprobe_multi.c
@@ -102,3 +102,9 @@ int test_uprobe_bench(struct pt_regs *ctx)
 	count++;
 	return 0;
 }
+
+SEC("uprobe.multi//proc/self/exe:uprobe_multi_func_*")
+int test_uprobe_extra(struct pt_regs *ctx)
+{
+	return 0;
+}
-- 
2.41.0

Re: [PATCHv2 bpf-next 03/24] bpf: Add pid filter support for uprobe_multi link

[Oleg Nesterov]

On 06/20, Jiri Olsa wrote:
>
> We still need to check the task during runtime in the uprobe handler,
> because the handler could get executed if there's another system
> wide consumer on the same uprobe (thanks Oleg for the insight).

... or link->task's sub-thread hits this uprobe.

Reviewed-by: Oleg Nesterov <oleg@redhat.com>

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Alexei Starovoitov]

On Tue, Jun 20, 2023 at 10:35:27AM +0200, Jiri Olsa wrote:
> +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> +			   unsigned long entry_ip,
> +			   struct pt_regs *regs)
> +{
> +	struct bpf_uprobe_multi_link *link = uprobe->link;
> +	struct bpf_uprobe_multi_run_ctx run_ctx = {
> +		.entry_ip = entry_ip,
> +	};
> +	struct bpf_prog *prog = link->link.prog;
> +	struct bpf_run_ctx *old_run_ctx;
> +	int err = 0;
> +
> +	might_fault();
> +
> +	rcu_read_lock_trace();
> +	migrate_disable();
> +
> +	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
> +		goto out;

bpf_prog_run_array_sleepable() doesn't do such things.
Such 'proteciton' will actively hurt.
The sleepable prog below will block all kprobes on this cpu.
please remove.

> +
> +	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
> +
> +	if (!prog->aux->sleepable)
> +		rcu_read_lock();
> +
> +	err = bpf_prog_run(link->link.prog, regs);
> +
> +	if (!prog->aux->sleepable)
> +		rcu_read_unlock();
> +
> +	bpf_reset_run_ctx(old_run_ctx);
> +
> +out:
> +	__this_cpu_dec(bpf_prog_active);
> +	migrate_enable();
> +	rcu_read_unlock_trace();
> +	return err;

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

On Tue, Jun 20, 2023 at 10:11:15AM -0700, Alexei Starovoitov wrote:
> On Tue, Jun 20, 2023 at 10:35:27AM +0200, Jiri Olsa wrote:
> > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > +			   unsigned long entry_ip,
> > +			   struct pt_regs *regs)
> > +{
> > +	struct bpf_uprobe_multi_link *link = uprobe->link;
> > +	struct bpf_uprobe_multi_run_ctx run_ctx = {
> > +		.entry_ip = entry_ip,
> > +	};
> > +	struct bpf_prog *prog = link->link.prog;
> > +	struct bpf_run_ctx *old_run_ctx;
> > +	int err = 0;
> > +
> > +	might_fault();
> > +
> > +	rcu_read_lock_trace();
> > +	migrate_disable();
> > +
> > +	if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
> > +		goto out;
> 
> bpf_prog_run_array_sleepable() doesn't do such things.
> Such 'proteciton' will actively hurt.
> The sleepable prog below will block all kprobes on this cpu.
> please remove.

ok makes sense, can't recall the reason why I added it

jirka

> 
> > +
> > +	old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
> > +
> > +	if (!prog->aux->sleepable)
> > +		rcu_read_lock();
> > +
> > +	err = bpf_prog_run(link->link.prog, regs);
> > +
> > +	if (!prog->aux->sleepable)
> > +		rcu_read_unlock();
> > +
> > +	bpf_reset_run_ctx(old_run_ctx);
> > +
> > +out:
> > +	__this_cpu_dec(bpf_prog_active);
> > +	migrate_enable();
> > +	rcu_read_unlock_trace();
> > +	return err;

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding new multi uprobe link that allows to attach bpf program
> to multiple uprobes.
>
> Uprobes to attach are specified via new link_create uprobe_multi
> union:
>
>   struct {
>           __u32           flags;
>           __u32           cnt;
>           __aligned_u64   path;
>           __aligned_u64   offsets;
>           __aligned_u64   ref_ctr_offsets;
>   } uprobe_multi;
>
> Uprobes are defined for single binary specified in path and multiple
> calling sites specified in offsets array with optional reference
> counters specified in ref_ctr_offsets array. All specified arrays
> have length of 'cnt'.
>
> The 'flags' supports single bit for now that marks the uprobe as
> return probe.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  include/linux/trace_events.h   |   6 +
>  include/uapi/linux/bpf.h       |  14 ++
>  kernel/bpf/syscall.c           |  12 +-
>  kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
>  tools/include/uapi/linux/bpf.h |  14 ++
>  5 files changed, 281 insertions(+), 2 deletions(-)
>

[...]

> diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> index a75c54b6f8a3..a96e46cd407e 100644
> --- a/kernel/bpf/syscall.c
> +++ b/kernel/bpf/syscall.c
> @@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
>                 return prog->enforce_expected_attach_type &&
>                         prog->expected_attach_type != attach_type ?
>                         -EINVAL : 0;
> +       case BPF_PROG_TYPE_KPROBE:
> +               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
> +                   attach_type != BPF_TRACE_KPROBE_MULTI)

should this be UPROBE_MULTI? this looks like your recent bug fix,
which already landed

> +                       return -EINVAL;
> +               fallthrough;

and I replaced this with `return 0;` ;)
>         default:
>                 return 0;
>         }
> @@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
>                 break;
>         case BPF_PROG_TYPE_KPROBE:
>                 if (attr->link_create.attach_type != BPF_PERF_EVENT &&
> -                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
> +                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
> +                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
>                         ret = -EINVAL;
>                         goto out;
>                 }

should this be moved into bpf_prog_attach_check_attach_type() and
unify these checks?

> @@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
>         case BPF_PROG_TYPE_KPROBE:
>                 if (attr->link_create.attach_type == BPF_PERF_EVENT)
>                         ret = bpf_perf_link_attach(attr, prog);
> -               else
> +               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
>                         ret = bpf_kprobe_multi_link_attach(attr, prog);
> +               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
> +                       ret = bpf_uprobe_multi_link_attach(attr, prog);
>                 break;
>         default:
>                 ret = -EINVAL;

[...]

> +static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
> +                                 u32 cnt)
> +{
> +       u32 i;
> +
> +       for (i = 0; i < cnt; i++) {
> +               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
> +                                 &uprobes[i].consumer);
> +       }
> +}
> +
> +static void bpf_uprobe_multi_link_release(struct bpf_link *link)
> +{
> +       struct bpf_uprobe_multi_link *umulti_link;
> +
> +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> +       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
> +       path_put(&umulti_link->path);
> +}
> +
> +static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
> +{
> +       struct bpf_uprobe_multi_link *umulti_link;
> +
> +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> +       kvfree(umulti_link->uprobes);
> +       kfree(umulti_link);
> +}
> +
> +static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
> +       .release = bpf_uprobe_multi_link_release,
> +       .dealloc = bpf_uprobe_multi_link_dealloc,
> +};
> +
> +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> +                          unsigned long entry_ip,
> +                          struct pt_regs *regs)
> +{
> +       struct bpf_uprobe_multi_link *link = uprobe->link;
> +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> +               .entry_ip = entry_ip,
> +       };
> +       struct bpf_prog *prog = link->link.prog;
> +       struct bpf_run_ctx *old_run_ctx;
> +       int err = 0;
> +
> +       might_fault();
> +
> +       rcu_read_lock_trace();

we don't need this if uprobe is not sleepable, right? why unconditional then?

> +       migrate_disable();
> +
> +       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
> +               goto out;
> +
> +       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
> +
> +       if (!prog->aux->sleepable)
> +               rcu_read_lock();
> +
> +       err = bpf_prog_run(link->link.prog, regs);
> +
> +       if (!prog->aux->sleepable)
> +               rcu_read_unlock();
> +
> +       bpf_reset_run_ctx(old_run_ctx);
> +
> +out:
> +       __this_cpu_dec(bpf_prog_active);
> +       migrate_enable();
> +       rcu_read_unlock_trace();
> +       return err;
> +}
> +

[...]

> +
> +       err = kern_path(name, LOOKUP_FOLLOW, &path);
> +       kfree(name);
> +       if (err)
> +               return err;
> +
> +       if (!d_is_reg(path.dentry)) {
> +               err = -EINVAL;
> +               goto error_path_put;
> +       }
> +
> +       err = -ENOMEM;
> +
> +       link = kzalloc(sizeof(*link), GFP_KERNEL);
> +       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
> +       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);

ref_ctr_offsets is optional, but we'll unconditionally allocate this array?

> +
> +       if (!uprobes || !ref_ctr_offsets || !link)
> +               goto error_free;
> +
> +       for (i = 0; i < cnt; i++) {
> +               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
> +                       err = -EFAULT;
> +                       goto error_free;
> +               }
> +               if (__get_user(offset, uoffsets + i)) {
> +                       err = -EFAULT;
> +                       goto error_free;
> +               }
> +
> +               uprobes[i].offset = offset;
> +               uprobes[i].link = link;
> +
> +               if (flags & BPF_F_UPROBE_MULTI_RETURN)
> +                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
> +               else
> +                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
> +
> +               ref_ctr_offsets[i] = ref_ctr_offset;
> +       }
> +
> +       link->cnt = cnt;
> +       link->uprobes = uprobes;
> +       link->path = path;
> +
> +       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
> +                     &bpf_uprobe_multi_link_lops, prog);
> +
> +       err = bpf_link_prime(&link->link, &link_primer);
> +       if (err)
> +               goto error_free;
> +
> +       for (i = 0; i < cnt; i++) {
> +               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
> +                                            uprobes[i].offset, ref_ctr_offsets[i],
> +                                            &uprobes[i].consumer);
> +               if (err) {
> +                       bpf_uprobe_unregister(&path, uprobes, i);

bpf_link_cleanup() will do this through
bpf_uprobe_multi_link_release(), no? So you are double unregistering?
Either drop cnt to zero, or just don't do this here? Latter is better,
IMO.

> +                       bpf_link_cleanup(&link_primer);
> +                       kvfree(ref_ctr_offsets);
> +                       return err;
> +               }
> +       }
> +
> +       kvfree(ref_ctr_offsets);
> +       return bpf_link_settle(&link_primer);
> +
> +error_free:
> +       kvfree(ref_ctr_offsets);
> +       kvfree(uprobes);
> +       kfree(link);
> +error_path_put:
> +       path_put(&path);
> +       return err;
> +}
> +#else /* !CONFIG_UPROBES */
> +int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
> +{
> +       return -EOPNOTSUPP;
> +}

[...]

Re: [PATCHv2 bpf-next 02/24] bpf: Add cookies support for uprobe_multi link

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding support to specify cookies array for uprobe_multi link.
>
> The cookies array share indexes and length with other uprobe_multi
> arrays (offsets/ref_ctr_offsets).
>
> The cookies[i] value defines cookie for i-the uprobe and will be
> returned by bpf_get_attach_cookie helper when called from ebpf
> program hooked to that specific uprobe.
>
> Acked-by: Andrii Nakryiko <andrii@kernel.org>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  include/uapi/linux/bpf.h       |  1 +
>  kernel/bpf/syscall.c           |  2 +-
>  kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
>  tools/include/uapi/linux/bpf.h |  1 +
>  4 files changed, 47 insertions(+), 5 deletions(-)
>

[...]

> @@ -3026,6 +3045,16 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
>         return uprobe_prog_run(uprobe, func, regs);
>  }
>
> +static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
> +{
> +       struct bpf_uprobe_multi_run_ctx *run_ctx;
> +
> +       if (!ctx)
> +               return 0;

can't happen, let's crash if it does happen?


> +       run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
> +       return run_ctx->uprobe->cookie;
> +}
> +

[...]

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding elf symbol iterator object (and some functions) that follow
> open-coded iterator pattern and some functions to ease up iterating
> elf object symbols.
>
> The idea is to iterate single symbol section with:
>
>   struct elf_symbol_iter iter;
>   struct elf_symbol *sym;
>
>   if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))
>         goto error;
>
>   while ((sym = elf_symbol_iter_next(&iter))) {
>         ...
>   }
>
> I considered opening the elf inside the iterator and iterate all symbol
> sections, but then it gets more complicated wrt user checks for when
> the next section is processed.
>
> Plus side is the we don't need 'exit' function, because caller/user is
> in charge of that.
>
> The returned iterated symbol object from elf_symbol_iter_next function
> is placed inside the struct elf_symbol_iter, so no extra allocation or
> argument is needed.
>
> Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
>  1 file changed, 114 insertions(+), 65 deletions(-)
>

This is great. Left a few nits below. I'm thinkin maybe we should add
a separate elf.c file for all these ELF-related helpers and start
offloading code from libbpf.c, which got pretty big already. WDYT?


> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index af52188daa80..cdac368c7ce1 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
>         return NULL;
>  }
>
> +struct elf_symbol {
> +       const char *name;
> +       unsigned long offset;
> +       int bind;
> +};
> +
> +struct elf_symbol_iter {

naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)

> +       Elf *elf;
> +       Elf_Data *symbols;

syms :-P

> +       size_t nr_syms;
> +       size_t strtabidx;
> +       size_t idx;

next_sym_idx?

> +       struct elf_symbol sym;
> +};
> +
> +static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
> +                              Elf *elf, const char *binary_path,
> +                              int sh_type)
> +{
> +       Elf_Scn *scn = NULL;
> +       GElf_Ehdr ehdr;
> +       GElf_Shdr sh;
> +
> +       memset(iter, 0, sizeof(*iter));
> +
> +       if (!gelf_getehdr(elf, &ehdr)) {
> +               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
> +               return -LIBBPF_ERRNO__FORMAT;
> +       }
> +
> +       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
> +       if (!scn) {
> +               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
> +                        binary_path);
> +               return -EINVAL;
> +       }
> +
> +       if (!gelf_getshdr(scn, &sh))
> +               return -EINVAL;
> +
> +       iter->strtabidx = sh.sh_link;
> +       iter->symbols = elf_getdata(scn, 0);
> +       if (!iter->symbols) {
> +               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
> +                       binary_path, elf_errmsg(-1));
> +               return -LIBBPF_ERRNO__FORMAT;
> +       }
> +       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
> +       iter->elf = elf;
> +       return 0;
> +}
> +
> +static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
> +{
> +       struct elf_symbol *ret = &iter->sym;
> +       unsigned long offset = 0;
> +       const char *name = NULL;
> +       GElf_Shdr sym_sh;
> +       Elf_Scn *sym_scn;
> +       GElf_Sym sym;
> +       size_t idx;
> +
> +       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
> +               if (!gelf_getsym(iter->symbols, idx, &sym))
> +                       continue;
> +               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
> +                       continue;

it would be more generic if this symbol type filter was a parameter to
iterator, instead of hard-coding it?

> +               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
> +               if (!name)
> +                       continue;
> +
> +               /* Transform symbol's virtual address (absolute for
> +                * binaries and relative for shared libs) into file
> +                * offset, which is what kernel is expecting for
> +                * uprobe/uretprobe attachment.
> +                * See Documentation/trace/uprobetracer.rst for more
> +                * details.
> +                * This is done by looking up symbol's containing
> +                * section's header and using iter's virtual address
> +                * (sh_addr) and corresponding file offset (sh_offset)
> +                * to transform sym.st_value (virtual address) into
> +                * desired final file offset.
> +                */
> +               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
> +               if (!sym_scn)
> +                       continue;
> +               if (!gelf_getshdr(sym_scn, &sym_sh))
> +                       continue;
> +
> +               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;

I think this part is not really generic "let's iterate ELF symbols",
maybe let users of iterator do this? We can have a helper to do
translation if we need to do it in few different places.

> +               break;
> +       }
> +
> +       /* we reached the last symbol */
> +       if (idx == iter->nr_syms)
> +               return NULL;
> +       iter->idx = idx + 1;
> +       ret->name = name;
> +       ret->bind = GELF_ST_BIND(sym.st_info);
> +       ret->offset = offset;

Why not just return entire GElf_Sym information and let user process
it as desired. So basically for each symbol you'll give back its name,
GElf_Sym info, and I'd return symbol index as well. That will keep
this very generic for future uses.

> +       return ret;

I'd structure this a bit different. If we got out of loop, just return
NULL. Then inside the for loop, when we found the symbol, fill out ret
and return from inside the for loop. I think it's more
straightforward.

> +}
> +
>  /* Find offset of function name in the provided ELF object. "binary_path" is
>   * the path to the ELF binary represented by "elf", and only used for error
>   * reporting matters. "name" matches symbol name or name@@LIB for library

[...]

Re: [PATCHv2 bpf-next 07/24] libbpf: Add open_elf/close_elf functions

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding open_elf/close_elf functions and using it in
> elf_find_func_offset_from_file function. It will be
> used in following changes to save some code.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
>  1 file changed, 44 insertions(+), 18 deletions(-)
>

we should definitely move all this into separate elf.c file

> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index cdac368c7ce1..30d9e3b69114 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
>         return ret;
>  }
>
> +struct elf_fd {
> +       Elf *elf;
> +       int fd;
> +};
> +
> +static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
> +{
> +       char errmsg[STRERR_BUFSIZE];
> +       int fd, ret;
> +       Elf *elf;
> +
> +       if (elf_version(EV_CURRENT) == EV_NONE) {
> +               pr_warn("failed to init libelf for %s\n", binary_path);
> +               return -LIBBPF_ERRNO__LIBELF;
> +       }
> +       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
> +       if (fd < 0) {
> +               ret = -errno;
> +               pr_warn("failed to open %s: %s\n", binary_path,
> +                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));

let's add "elf: " prefix for consistency?

> +               return ret;
> +       }
> +       elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
> +       if (!elf) {
> +               pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
> +               close(fd);
> +               return -LIBBPF_ERRNO__FORMAT;
> +       }
> +       elf_fd->fd = fd;
> +       elf_fd->elf = elf;
> +       return 0;
> +}
> +
> +static void close_elf(struct elf_fd *elf_fd)
> +{
> +       elf_end(elf_fd->elf);
> +       close(elf_fd->fd);
> +}
> +
>  /* Find offset of function name in the provided ELF object. "binary_path" is
>   * the path to the ELF binary represented by "elf", and only used for error
>   * reporting matters. "name" matches symbol name or name@@LIB for library
> @@ -11019,28 +11058,15 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
>   */
>  static long elf_find_func_offset_from_file(const char *binary_path, const char *name)
>  {
> -       char errmsg[STRERR_BUFSIZE];
> +       struct elf_fd elf_fd = {};
>         long ret = -ENOENT;
> -       Elf *elf;
> -       int fd;
>
> -       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
> -       if (fd < 0) {
> -               ret = -errno;
> -               pr_warn("failed to open %s: %s\n", binary_path,
> -                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
> +       ret = open_elf(binary_path, &elf_fd);
> +       if (ret)
>                 return ret;
> -       }
> -       elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
> -       if (!elf) {
> -               pr_warn("elf: could not read elf from %s: %s\n", binary_path, elf_errmsg(-1));
> -               close(fd);
> -               return -LIBBPF_ERRNO__FORMAT;
> -       }
>
> -       ret = elf_find_func_offset(elf, binary_path, name);
> -       elf_end(elf);
> -       close(fd);
> +       ret = elf_find_func_offset(elf_fd.elf, binary_path, name);
> +       close_elf(&elf_fd);
>         return ret;
>  }
>
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 02/24] bpf: Add cookies support for uprobe_multi link

[Jiri Olsa]

On Thu, Jun 22, 2023 at 05:18:10PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding support to specify cookies array for uprobe_multi link.
> >
> > The cookies array share indexes and length with other uprobe_multi
> > arrays (offsets/ref_ctr_offsets).
> >
> > The cookies[i] value defines cookie for i-the uprobe and will be
> > returned by bpf_get_attach_cookie helper when called from ebpf
> > program hooked to that specific uprobe.
> >
> > Acked-by: Andrii Nakryiko <andrii@kernel.org>
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  include/uapi/linux/bpf.h       |  1 +
> >  kernel/bpf/syscall.c           |  2 +-
> >  kernel/trace/bpf_trace.c       | 48 +++++++++++++++++++++++++++++++---
> >  tools/include/uapi/linux/bpf.h |  1 +
> >  4 files changed, 47 insertions(+), 5 deletions(-)
> >
> 
> [...]
> 
> > @@ -3026,6 +3045,16 @@ uprobe_multi_link_ret_handler(struct uprobe_consumer *con, unsigned long func, s
> >         return uprobe_prog_run(uprobe, func, regs);
> >  }
> >
> > +static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx)
> > +{
> > +       struct bpf_uprobe_multi_run_ctx *run_ctx;
> > +
> > +       if (!ctx)
> > +               return 0;
> 
> can't happen, let's crash if it does happen?

ok, will remove

jirka

> 
> 
> > +       run_ctx = container_of(current->bpf_ctx, struct bpf_uprobe_multi_run_ctx, run_ctx);
> > +       return run_ctx->uprobe->cookie;
> > +}
> > +
> 
> [...]

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

[Jiri Olsa]

On Thu, Jun 22, 2023 at 05:31:58PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding elf symbol iterator object (and some functions) that follow
> > open-coded iterator pattern and some functions to ease up iterating
> > elf object symbols.
> >
> > The idea is to iterate single symbol section with:
> >
> >   struct elf_symbol_iter iter;
> >   struct elf_symbol *sym;
> >
> >   if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))
> >         goto error;
> >
> >   while ((sym = elf_symbol_iter_next(&iter))) {
> >         ...
> >   }
> >
> > I considered opening the elf inside the iterator and iterate all symbol
> > sections, but then it gets more complicated wrt user checks for when
> > the next section is processed.
> >
> > Plus side is the we don't need 'exit' function, because caller/user is
> > in charge of that.
> >
> > The returned iterated symbol object from elf_symbol_iter_next function
> > is placed inside the struct elf_symbol_iter, so no extra allocation or
> > argument is needed.
> >
> > Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
> >  1 file changed, 114 insertions(+), 65 deletions(-)
> >
> 
> This is great. Left a few nits below. I'm thinkin maybe we should add
> a separate elf.c file for all these ELF-related helpers and start
> offloading code from libbpf.c, which got pretty big already. WDYT?

yes, I thought doing the move after this is merged might be better,
because it's quite big already

> 
> 
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index af52188daa80..cdac368c7ce1 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
> >         return NULL;
> >  }
> >
> > +struct elf_symbol {
> > +       const char *name;
> > +       unsigned long offset;
> > +       int bind;
> > +};
> > +
> > +struct elf_symbol_iter {
> 
> naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)

ok

> 
> > +       Elf *elf;
> > +       Elf_Data *symbols;
> 
> syms :-P

ook ;-)

> 
> > +       size_t nr_syms;
> > +       size_t strtabidx;
> > +       size_t idx;
> 
> next_sym_idx?

ok

> 
> > +       struct elf_symbol sym;
> > +};
> > +
> > +static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
> > +                              Elf *elf, const char *binary_path,
> > +                              int sh_type)
> > +{
> > +       Elf_Scn *scn = NULL;
> > +       GElf_Ehdr ehdr;
> > +       GElf_Shdr sh;
> > +
> > +       memset(iter, 0, sizeof(*iter));
> > +
> > +       if (!gelf_getehdr(elf, &ehdr)) {
> > +               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
> > +               return -LIBBPF_ERRNO__FORMAT;
> > +       }
> > +
> > +       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
> > +       if (!scn) {
> > +               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
> > +                        binary_path);
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (!gelf_getshdr(scn, &sh))
> > +               return -EINVAL;
> > +
> > +       iter->strtabidx = sh.sh_link;
> > +       iter->symbols = elf_getdata(scn, 0);
> > +       if (!iter->symbols) {
> > +               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
> > +                       binary_path, elf_errmsg(-1));
> > +               return -LIBBPF_ERRNO__FORMAT;
> > +       }
> > +       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
> > +       iter->elf = elf;
> > +       return 0;
> > +}
> > +
> > +static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
> > +{
> > +       struct elf_symbol *ret = &iter->sym;
> > +       unsigned long offset = 0;
> > +       const char *name = NULL;
> > +       GElf_Shdr sym_sh;
> > +       Elf_Scn *sym_scn;
> > +       GElf_Sym sym;
> > +       size_t idx;
> > +
> > +       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
> > +               if (!gelf_getsym(iter->symbols, idx, &sym))
> > +                       continue;
> > +               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
> > +                       continue;
> 
> it would be more generic if this symbol type filter was a parameter to
> iterator, instead of hard-coding it?

ok

> 
> > +               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
> > +               if (!name)
> > +                       continue;
> > +
> > +               /* Transform symbol's virtual address (absolute for
> > +                * binaries and relative for shared libs) into file
> > +                * offset, which is what kernel is expecting for
> > +                * uprobe/uretprobe attachment.
> > +                * See Documentation/trace/uprobetracer.rst for more
> > +                * details.
> > +                * This is done by looking up symbol's containing
> > +                * section's header and using iter's virtual address
> > +                * (sh_addr) and corresponding file offset (sh_offset)
> > +                * to transform sym.st_value (virtual address) into
> > +                * desired final file offset.
> > +                */
> > +               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
> > +               if (!sym_scn)
> > +                       continue;
> > +               if (!gelf_getshdr(sym_scn, &sym_sh))
> > +                       continue;
> > +
> > +               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
> 
> I think this part is not really generic "let's iterate ELF symbols",
> maybe let users of iterator do this? We can have a helper to do
> translation if we need to do it in few different places.

yes this will be called in all the places we use the iterator,
I'll add the helper for it

> 
> > +               break;
> > +       }
> > +
> > +       /* we reached the last symbol */
> > +       if (idx == iter->nr_syms)
> > +               return NULL;
> > +       iter->idx = idx + 1;
> > +       ret->name = name;
> > +       ret->bind = GELF_ST_BIND(sym.st_info);
> > +       ret->offset = offset;
> 
> Why not just return entire GElf_Sym information and let user process
> it as desired. So basically for each symbol you'll give back its name,
> GElf_Sym info, and I'd return symbol index as well. That will keep
> this very generic for future uses.

ok, so you have other users of this iterator in mind already?

> 
> > +       return ret;
> 
> I'd structure this a bit different. If we got out of loop, just return
> NULL. Then inside the for loop, when we found the symbol, fill out ret
> and return from inside the for loop. I think it's more
> straightforward.

ok, will change

thanks,
jirka

> 
> > +}
> > +
> >  /* Find offset of function name in the provided ELF object. "binary_path" is
> >   * the path to the ELF binary represented by "elf", and only used for error
> >   * reporting matters. "name" matches symbol name or name@@LIB for library
> 
> [...]

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

On Thu, Jun 22, 2023 at 05:18:05PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding new multi uprobe link that allows to attach bpf program
> > to multiple uprobes.
> >
> > Uprobes to attach are specified via new link_create uprobe_multi
> > union:
> >
> >   struct {
> >           __u32           flags;
> >           __u32           cnt;
> >           __aligned_u64   path;
> >           __aligned_u64   offsets;
> >           __aligned_u64   ref_ctr_offsets;
> >   } uprobe_multi;
> >
> > Uprobes are defined for single binary specified in path and multiple
> > calling sites specified in offsets array with optional reference
> > counters specified in ref_ctr_offsets array. All specified arrays
> > have length of 'cnt'.
> >
> > The 'flags' supports single bit for now that marks the uprobe as
> > return probe.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  include/linux/trace_events.h   |   6 +
> >  include/uapi/linux/bpf.h       |  14 ++
> >  kernel/bpf/syscall.c           |  12 +-
> >  kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
> >  tools/include/uapi/linux/bpf.h |  14 ++
> >  5 files changed, 281 insertions(+), 2 deletions(-)
> >
> 
> [...]
> 
> > diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> > index a75c54b6f8a3..a96e46cd407e 100644
> > --- a/kernel/bpf/syscall.c
> > +++ b/kernel/bpf/syscall.c
> > @@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
> >                 return prog->enforce_expected_attach_type &&
> >                         prog->expected_attach_type != attach_type ?
> >                         -EINVAL : 0;
> > +       case BPF_PROG_TYPE_KPROBE:
> > +               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
> > +                   attach_type != BPF_TRACE_KPROBE_MULTI)
> 
> should this be UPROBE_MULTI? this looks like your recent bug fix,
> which already landed
> 
> > +                       return -EINVAL;
> > +               fallthrough;
> 
> and I replaced this with `return 0;` ;)

ugh, yes, will fix

> >         default:
> >                 return 0;
> >         }
> > @@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
> >                 break;
> >         case BPF_PROG_TYPE_KPROBE:
> >                 if (attr->link_create.attach_type != BPF_PERF_EVENT &&
> > -                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
> > +                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
> > +                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
> >                         ret = -EINVAL;
> >                         goto out;
> >                 }
> 
> should this be moved into bpf_prog_attach_check_attach_type() and
> unify these checks?

ok, perhaps we could move there the whole switch, will check

> 
> > @@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
> >         case BPF_PROG_TYPE_KPROBE:
> >                 if (attr->link_create.attach_type == BPF_PERF_EVENT)
> >                         ret = bpf_perf_link_attach(attr, prog);
> > -               else
> > +               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
> >                         ret = bpf_kprobe_multi_link_attach(attr, prog);
> > +               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
> > +                       ret = bpf_uprobe_multi_link_attach(attr, prog);
> >                 break;
> >         default:
> >                 ret = -EINVAL;
> 
> [...]
> 
> > +static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
> > +                                 u32 cnt)
> > +{
> > +       u32 i;
> > +
> > +       for (i = 0; i < cnt; i++) {
> > +               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
> > +                                 &uprobes[i].consumer);
> > +       }
> > +}
> > +
> > +static void bpf_uprobe_multi_link_release(struct bpf_link *link)
> > +{
> > +       struct bpf_uprobe_multi_link *umulti_link;
> > +
> > +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> > +       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
> > +       path_put(&umulti_link->path);
> > +}
> > +
> > +static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
> > +{
> > +       struct bpf_uprobe_multi_link *umulti_link;
> > +
> > +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> > +       kvfree(umulti_link->uprobes);
> > +       kfree(umulti_link);
> > +}
> > +
> > +static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
> > +       .release = bpf_uprobe_multi_link_release,
> > +       .dealloc = bpf_uprobe_multi_link_dealloc,
> > +};
> > +
> > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > +                          unsigned long entry_ip,
> > +                          struct pt_regs *regs)
> > +{
> > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > +               .entry_ip = entry_ip,
> > +       };
> > +       struct bpf_prog *prog = link->link.prog;
> > +       struct bpf_run_ctx *old_run_ctx;
> > +       int err = 0;
> > +
> > +       might_fault();
> > +
> > +       rcu_read_lock_trace();
> 
> we don't need this if uprobe is not sleepable, right? why unconditional then?

I won't pretend I understand what rcu_read_lock_trace does ;-)

I tried to follow bpf_prog_run_array_sleepable where it's called
unconditionally for both sleepable and non-sleepable progs

there are conditional rcu_read_un/lock calls later on

I will check

> 
> > +       migrate_disable();
> > +
> > +       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
> > +               goto out;
> > +
> > +       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
> > +
> > +       if (!prog->aux->sleepable)
> > +               rcu_read_lock();
> > +
> > +       err = bpf_prog_run(link->link.prog, regs);
> > +
> > +       if (!prog->aux->sleepable)
> > +               rcu_read_unlock();
> > +
> > +       bpf_reset_run_ctx(old_run_ctx);
> > +
> > +out:
> > +       __this_cpu_dec(bpf_prog_active);
> > +       migrate_enable();
> > +       rcu_read_unlock_trace();
> > +       return err;
> > +}
> > +
> 
> [...]
> 
> > +
> > +       err = kern_path(name, LOOKUP_FOLLOW, &path);
> > +       kfree(name);
> > +       if (err)
> > +               return err;
> > +
> > +       if (!d_is_reg(path.dentry)) {
> > +               err = -EINVAL;
> > +               goto error_path_put;
> > +       }
> > +
> > +       err = -ENOMEM;
> > +
> > +       link = kzalloc(sizeof(*link), GFP_KERNEL);
> > +       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
> > +       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);
> 
> ref_ctr_offsets is optional, but we'll unconditionally allocate this array?

true :-\ will add the uref_ctr_offsets check

> 
> > +
> > +       if (!uprobes || !ref_ctr_offsets || !link)
> > +               goto error_free;
> > +
> > +       for (i = 0; i < cnt; i++) {
> > +               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
> > +                       err = -EFAULT;
> > +                       goto error_free;
> > +               }
> > +               if (__get_user(offset, uoffsets + i)) {
> > +                       err = -EFAULT;
> > +                       goto error_free;
> > +               }
> > +
> > +               uprobes[i].offset = offset;
> > +               uprobes[i].link = link;
> > +
> > +               if (flags & BPF_F_UPROBE_MULTI_RETURN)
> > +                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
> > +               else
> > +                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
> > +
> > +               ref_ctr_offsets[i] = ref_ctr_offset;
> > +       }
> > +
> > +       link->cnt = cnt;
> > +       link->uprobes = uprobes;
> > +       link->path = path;
> > +
> > +       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
> > +                     &bpf_uprobe_multi_link_lops, prog);
> > +
> > +       err = bpf_link_prime(&link->link, &link_primer);
> > +       if (err)
> > +               goto error_free;
> > +
> > +       for (i = 0; i < cnt; i++) {
> > +               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
> > +                                            uprobes[i].offset, ref_ctr_offsets[i],
> > +                                            &uprobes[i].consumer);
> > +               if (err) {
> > +                       bpf_uprobe_unregister(&path, uprobes, i);
> 
> bpf_link_cleanup() will do this through
> bpf_uprobe_multi_link_release(), no? So you are double unregistering?
> Either drop cnt to zero, or just don't do this here? Latter is better,
> IMO.

bpf_link_cleanup path won't call release callback so we have to do that

I think I can add simple selftest to have this path covered

thanks,
jirka

> 
> > +                       bpf_link_cleanup(&link_primer);
> > +                       kvfree(ref_ctr_offsets);
> > +                       return err;
> > +               }
> > +       }
> > +
> > +       kvfree(ref_ctr_offsets);
> > +       return bpf_link_settle(&link_primer);
> > +
> > +error_free:
> > +       kvfree(ref_ctr_offsets);
> > +       kvfree(uprobes);
> > +       kfree(link);
> > +error_path_put:
> > +       path_put(&path);
> > +       return err;
> > +}
> > +#else /* !CONFIG_UPROBES */
> > +int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
> > +{
> > +       return -EOPNOTSUPP;
> > +}
> 
> [...]

Re: [PATCHv2 bpf-next 07/24] libbpf: Add open_elf/close_elf functions

[Jiri Olsa]

On Thu, Jun 22, 2023 at 05:33:33PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding open_elf/close_elf functions and using it in
> > elf_find_func_offset_from_file function. It will be
> > used in following changes to save some code.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c | 62 ++++++++++++++++++++++++++++++------------
> >  1 file changed, 44 insertions(+), 18 deletions(-)
> >
> 
> we should definitely move all this into separate elf.c file

right, also we could use this in usdt_manager_attach_usdt as well


> 
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index cdac368c7ce1..30d9e3b69114 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -10927,6 +10927,45 @@ static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
> >         return ret;
> >  }
> >
> > +struct elf_fd {
> > +       Elf *elf;
> > +       int fd;
> > +};
> > +
> > +static int open_elf(const char *binary_path, struct elf_fd *elf_fd)
> > +{
> > +       char errmsg[STRERR_BUFSIZE];
> > +       int fd, ret;
> > +       Elf *elf;
> > +
> > +       if (elf_version(EV_CURRENT) == EV_NONE) {
> > +               pr_warn("failed to init libelf for %s\n", binary_path);
> > +               return -LIBBPF_ERRNO__LIBELF;
> > +       }
> > +       fd = open(binary_path, O_RDONLY | O_CLOEXEC);
> > +       if (fd < 0) {
> > +               ret = -errno;
> > +               pr_warn("failed to open %s: %s\n", binary_path,
> > +                       libbpf_strerror_r(ret, errmsg, sizeof(errmsg)));
> 
> let's add "elf: " prefix for consistency?

ok

jirka

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Andrii Nakryiko]

On Fri, Jun 23, 2023 at 1:19 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Jun 22, 2023 at 05:18:05PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > >
> > > Adding new multi uprobe link that allows to attach bpf program
> > > to multiple uprobes.
> > >
> > > Uprobes to attach are specified via new link_create uprobe_multi
> > > union:
> > >
> > >   struct {
> > >           __u32           flags;
> > >           __u32           cnt;
> > >           __aligned_u64   path;
> > >           __aligned_u64   offsets;
> > >           __aligned_u64   ref_ctr_offsets;
> > >   } uprobe_multi;
> > >
> > > Uprobes are defined for single binary specified in path and multiple
> > > calling sites specified in offsets array with optional reference
> > > counters specified in ref_ctr_offsets array. All specified arrays
> > > have length of 'cnt'.
> > >
> > > The 'flags' supports single bit for now that marks the uprobe as
> > > return probe.
> > >
> > > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > > ---
> > >  include/linux/trace_events.h   |   6 +
> > >  include/uapi/linux/bpf.h       |  14 ++
> > >  kernel/bpf/syscall.c           |  12 +-
> > >  kernel/trace/bpf_trace.c       | 237 +++++++++++++++++++++++++++++++++
> > >  tools/include/uapi/linux/bpf.h |  14 ++
> > >  5 files changed, 281 insertions(+), 2 deletions(-)
> > >
> >
> > [...]
> >
> > > diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
> > > index a75c54b6f8a3..a96e46cd407e 100644
> > > --- a/kernel/bpf/syscall.c
> > > +++ b/kernel/bpf/syscall.c
> > > @@ -3516,6 +3516,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
> > >                 return prog->enforce_expected_attach_type &&
> > >                         prog->expected_attach_type != attach_type ?
> > >                         -EINVAL : 0;
> > > +       case BPF_PROG_TYPE_KPROBE:
> > > +               if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
> > > +                   attach_type != BPF_TRACE_KPROBE_MULTI)
> >
> > should this be UPROBE_MULTI? this looks like your recent bug fix,
> > which already landed
> >
> > > +                       return -EINVAL;
> > > +               fallthrough;
> >
> > and I replaced this with `return 0;` ;)
>
> ugh, yes, will fix
>
> > >         default:
> > >                 return 0;
> > >         }
> > > @@ -4681,7 +4686,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
> > >                 break;
> > >         case BPF_PROG_TYPE_KPROBE:
> > >                 if (attr->link_create.attach_type != BPF_PERF_EVENT &&
> > > -                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI) {
> > > +                   attr->link_create.attach_type != BPF_TRACE_KPROBE_MULTI &&
> > > +                   attr->link_create.attach_type != BPF_TRACE_UPROBE_MULTI) {
> > >                         ret = -EINVAL;
> > >                         goto out;
> > >                 }
> >
> > should this be moved into bpf_prog_attach_check_attach_type() and
> > unify these checks?
>
> ok, perhaps we could move there the whole switch, will check

+1

>
> >
> > > @@ -4748,8 +4754,10 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr)
> > >         case BPF_PROG_TYPE_KPROBE:
> > >                 if (attr->link_create.attach_type == BPF_PERF_EVENT)
> > >                         ret = bpf_perf_link_attach(attr, prog);
> > > -               else
> > > +               else if (attr->link_create.attach_type == BPF_TRACE_KPROBE_MULTI)
> > >                         ret = bpf_kprobe_multi_link_attach(attr, prog);
> > > +               else if (attr->link_create.attach_type == BPF_TRACE_UPROBE_MULTI)
> > > +                       ret = bpf_uprobe_multi_link_attach(attr, prog);
> > >                 break;
> > >         default:
> > >                 ret = -EINVAL;
> >
> > [...]
> >
> > > +static void bpf_uprobe_unregister(struct path *path, struct bpf_uprobe *uprobes,
> > > +                                 u32 cnt)
> > > +{
> > > +       u32 i;
> > > +
> > > +       for (i = 0; i < cnt; i++) {
> > > +               uprobe_unregister(d_real_inode(path->dentry), uprobes[i].offset,
> > > +                                 &uprobes[i].consumer);
> > > +       }
> > > +}
> > > +
> > > +static void bpf_uprobe_multi_link_release(struct bpf_link *link)
> > > +{
> > > +       struct bpf_uprobe_multi_link *umulti_link;
> > > +
> > > +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> > > +       bpf_uprobe_unregister(&umulti_link->path, umulti_link->uprobes, umulti_link->cnt);
> > > +       path_put(&umulti_link->path);
> > > +}
> > > +
> > > +static void bpf_uprobe_multi_link_dealloc(struct bpf_link *link)
> > > +{
> > > +       struct bpf_uprobe_multi_link *umulti_link;
> > > +
> > > +       umulti_link = container_of(link, struct bpf_uprobe_multi_link, link);
> > > +       kvfree(umulti_link->uprobes);
> > > +       kfree(umulti_link);
> > > +}
> > > +
> > > +static const struct bpf_link_ops bpf_uprobe_multi_link_lops = {
> > > +       .release = bpf_uprobe_multi_link_release,
> > > +       .dealloc = bpf_uprobe_multi_link_dealloc,
> > > +};
> > > +
> > > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > > +                          unsigned long entry_ip,
> > > +                          struct pt_regs *regs)
> > > +{
> > > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > > +               .entry_ip = entry_ip,
> > > +       };
> > > +       struct bpf_prog *prog = link->link.prog;
> > > +       struct bpf_run_ctx *old_run_ctx;
> > > +       int err = 0;
> > > +
> > > +       might_fault();
> > > +
> > > +       rcu_read_lock_trace();
> >
> > we don't need this if uprobe is not sleepable, right? why unconditional then?
>
> I won't pretend I understand what rcu_read_lock_trace does ;-)
>
> I tried to follow bpf_prog_run_array_sleepable where it's called
> unconditionally for both sleepable and non-sleepable progs
>
> there are conditional rcu_read_un/lock calls later on
>
> I will check

hm... Alexei can chime in here, but given here we actually are trying
to run one BPF program (not entire array of them), we do know whether
it's going to be sleepable or not. So we can avoid unnecessary
rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used
when there is going to be sleepable BPF program executed to protect
BPF maps and other resources from being freed too soon. But if we know
that we don't need sleepable, we can avoid that.

>
> >
> > > +       migrate_disable();
> > > +
> > > +       if (unlikely(__this_cpu_inc_return(bpf_prog_active) != 1))
> > > +               goto out;
> > > +
> > > +       old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);
> > > +
> > > +       if (!prog->aux->sleepable)
> > > +               rcu_read_lock();
> > > +
> > > +       err = bpf_prog_run(link->link.prog, regs);
> > > +
> > > +       if (!prog->aux->sleepable)
> > > +               rcu_read_unlock();
> > > +
> > > +       bpf_reset_run_ctx(old_run_ctx);
> > > +
> > > +out:
> > > +       __this_cpu_dec(bpf_prog_active);
> > > +       migrate_enable();
> > > +       rcu_read_unlock_trace();
> > > +       return err;
> > > +}
> > > +
> >
> > [...]
> >
> > > +
> > > +       err = kern_path(name, LOOKUP_FOLLOW, &path);
> > > +       kfree(name);
> > > +       if (err)
> > > +               return err;
> > > +
> > > +       if (!d_is_reg(path.dentry)) {
> > > +               err = -EINVAL;
> > > +               goto error_path_put;
> > > +       }
> > > +
> > > +       err = -ENOMEM;
> > > +
> > > +       link = kzalloc(sizeof(*link), GFP_KERNEL);
> > > +       uprobes = kvcalloc(cnt, sizeof(*uprobes), GFP_KERNEL);
> > > +       ref_ctr_offsets = kvcalloc(cnt, sizeof(*ref_ctr_offsets), GFP_KERNEL);
> >
> > ref_ctr_offsets is optional, but we'll unconditionally allocate this array?
>
> true :-\ will add the uref_ctr_offsets check
>
> >
> > > +
> > > +       if (!uprobes || !ref_ctr_offsets || !link)
> > > +               goto error_free;
> > > +
> > > +       for (i = 0; i < cnt; i++) {
> > > +               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
> > > +                       err = -EFAULT;
> > > +                       goto error_free;
> > > +               }
> > > +               if (__get_user(offset, uoffsets + i)) {
> > > +                       err = -EFAULT;
> > > +                       goto error_free;
> > > +               }
> > > +
> > > +               uprobes[i].offset = offset;
> > > +               uprobes[i].link = link;
> > > +
> > > +               if (flags & BPF_F_UPROBE_MULTI_RETURN)
> > > +                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
> > > +               else
> > > +                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
> > > +
> > > +               ref_ctr_offsets[i] = ref_ctr_offset;
> > > +       }
> > > +
> > > +       link->cnt = cnt;
> > > +       link->uprobes = uprobes;
> > > +       link->path = path;
> > > +
> > > +       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
> > > +                     &bpf_uprobe_multi_link_lops, prog);
> > > +
> > > +       err = bpf_link_prime(&link->link, &link_primer);
> > > +       if (err)
> > > +               goto error_free;
> > > +
> > > +       for (i = 0; i < cnt; i++) {
> > > +               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
> > > +                                            uprobes[i].offset, ref_ctr_offsets[i],
> > > +                                            &uprobes[i].consumer);
> > > +               if (err) {
> > > +                       bpf_uprobe_unregister(&path, uprobes, i);
> >
> > bpf_link_cleanup() will do this through
> > bpf_uprobe_multi_link_release(), no? So you are double unregistering?
> > Either drop cnt to zero, or just don't do this here? Latter is better,
> > IMO.
>
> bpf_link_cleanup path won't call release callback so we have to do that

bpf_link_cleanup() does fput(primer->file); which eventually calls
release callback, no? I'd add printk and simulate failure just to be
sure

>
> I think I can add simple selftest to have this path covered
>
> thanks,
> jirka
>
> >
> > > +                       bpf_link_cleanup(&link_primer);
> > > +                       kvfree(ref_ctr_offsets);
> > > +                       return err;
> > > +               }
> > > +       }
> > > +
> > > +       kvfree(ref_ctr_offsets);
> > > +       return bpf_link_settle(&link_primer);
> > > +
> > > +error_free:
> > > +       kvfree(ref_ctr_offsets);
> > > +       kvfree(uprobes);
> > > +       kfree(link);
> > > +error_path_put:
> > > +       path_put(&path);
> > > +       return err;
> > > +}
> > > +#else /* !CONFIG_UPROBES */
> > > +int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
> > > +{
> > > +       return -EOPNOTSUPP;
> > > +}
> >
> > [...]

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

[Andrii Nakryiko]

On Fri, Jun 23, 2023 at 1:19 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Jun 22, 2023 at 05:31:58PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > >
> > > Adding elf symbol iterator object (and some functions) that follow
> > > open-coded iterator pattern and some functions to ease up iterating
> > > elf object symbols.
> > >
> > > The idea is to iterate single symbol section with:
> > >
> > >   struct elf_symbol_iter iter;
> > >   struct elf_symbol *sym;
> > >
> > >   if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))
> > >         goto error;
> > >
> > >   while ((sym = elf_symbol_iter_next(&iter))) {
> > >         ...
> > >   }
> > >
> > > I considered opening the elf inside the iterator and iterate all symbol
> > > sections, but then it gets more complicated wrt user checks for when
> > > the next section is processed.
> > >
> > > Plus side is the we don't need 'exit' function, because caller/user is
> > > in charge of that.
> > >
> > > The returned iterated symbol object from elf_symbol_iter_next function
> > > is placed inside the struct elf_symbol_iter, so no extra allocation or
> > > argument is needed.
> > >
> > > Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> > > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > > ---
> > >  tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
> > >  1 file changed, 114 insertions(+), 65 deletions(-)
> > >
> >
> > This is great. Left a few nits below. I'm thinkin maybe we should add
> > a separate elf.c file for all these ELF-related helpers and start
> > offloading code from libbpf.c, which got pretty big already. WDYT?
>
> yes, I thought doing the move after this is merged might be better,
> because it's quite big already
>
> >
> >
> > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > index af52188daa80..cdac368c7ce1 100644
> > > --- a/tools/lib/bpf/libbpf.c
> > > +++ b/tools/lib/bpf/libbpf.c
> > > @@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
> > >         return NULL;
> > >  }
> > >
> > > +struct elf_symbol {
> > > +       const char *name;
> > > +       unsigned long offset;
> > > +       int bind;
> > > +};
> > > +
> > > +struct elf_symbol_iter {
> >
> > naming nits: elf_sym and elf_sym_iter? keep it short, keep it cool :)
>
> ok
>
> >
> > > +       Elf *elf;
> > > +       Elf_Data *symbols;
> >
> > syms :-P
>
> ook ;-)
>
> >
> > > +       size_t nr_syms;
> > > +       size_t strtabidx;
> > > +       size_t idx;
> >
> > next_sym_idx?
>
> ok
>
> >
> > > +       struct elf_symbol sym;
> > > +};
> > > +
> > > +static int elf_symbol_iter_new(struct elf_symbol_iter *iter,
> > > +                              Elf *elf, const char *binary_path,
> > > +                              int sh_type)
> > > +{
> > > +       Elf_Scn *scn = NULL;
> > > +       GElf_Ehdr ehdr;
> > > +       GElf_Shdr sh;
> > > +
> > > +       memset(iter, 0, sizeof(*iter));
> > > +
> > > +       if (!gelf_getehdr(elf, &ehdr)) {
> > > +               pr_warn("elf: failed to get ehdr from %s: %s\n", binary_path, elf_errmsg(-1));
> > > +               return -LIBBPF_ERRNO__FORMAT;
> > > +       }
> > > +
> > > +       scn = elf_find_next_scn_by_type(elf, sh_type, NULL);
> > > +       if (!scn) {
> > > +               pr_debug("elf: failed to find symbol table ELF sections in '%s'\n",
> > > +                        binary_path);
> > > +               return -EINVAL;
> > > +       }
> > > +
> > > +       if (!gelf_getshdr(scn, &sh))
> > > +               return -EINVAL;
> > > +
> > > +       iter->strtabidx = sh.sh_link;
> > > +       iter->symbols = elf_getdata(scn, 0);
> > > +       if (!iter->symbols) {
> > > +               pr_warn("elf: failed to get symbols for symtab section in '%s': %s\n",
> > > +                       binary_path, elf_errmsg(-1));
> > > +               return -LIBBPF_ERRNO__FORMAT;
> > > +       }
> > > +       iter->nr_syms = iter->symbols->d_size / sh.sh_entsize;
> > > +       iter->elf = elf;
> > > +       return 0;
> > > +}
> > > +
> > > +static struct elf_symbol *elf_symbol_iter_next(struct elf_symbol_iter *iter)
> > > +{
> > > +       struct elf_symbol *ret = &iter->sym;
> > > +       unsigned long offset = 0;
> > > +       const char *name = NULL;
> > > +       GElf_Shdr sym_sh;
> > > +       Elf_Scn *sym_scn;
> > > +       GElf_Sym sym;
> > > +       size_t idx;
> > > +
> > > +       for (idx = iter->idx; idx < iter->nr_syms; idx++) {
> > > +               if (!gelf_getsym(iter->symbols, idx, &sym))
> > > +                       continue;
> > > +               if (GELF_ST_TYPE(sym.st_info) != STT_FUNC)
> > > +                       continue;
> >
> > it would be more generic if this symbol type filter was a parameter to
> > iterator, instead of hard-coding it?
>
> ok
>
> >
> > > +               name = elf_strptr(iter->elf, iter->strtabidx, sym.st_name);
> > > +               if (!name)
> > > +                       continue;
> > > +
> > > +               /* Transform symbol's virtual address (absolute for
> > > +                * binaries and relative for shared libs) into file
> > > +                * offset, which is what kernel is expecting for
> > > +                * uprobe/uretprobe attachment.
> > > +                * See Documentation/trace/uprobetracer.rst for more
> > > +                * details.
> > > +                * This is done by looking up symbol's containing
> > > +                * section's header and using iter's virtual address
> > > +                * (sh_addr) and corresponding file offset (sh_offset)
> > > +                * to transform sym.st_value (virtual address) into
> > > +                * desired final file offset.
> > > +                */
> > > +               sym_scn = elf_getscn(iter->elf, sym.st_shndx);
> > > +               if (!sym_scn)
> > > +                       continue;
> > > +               if (!gelf_getshdr(sym_scn, &sym_sh))
> > > +                       continue;
> > > +
> > > +               offset = sym.st_value - sym_sh.sh_addr + sym_sh.sh_offset;
> >
> > I think this part is not really generic "let's iterate ELF symbols",
> > maybe let users of iterator do this? We can have a helper to do
> > translation if we need to do it in few different places.
>
> yes this will be called in all the places we use the iterator,
> I'll add the helper for it
>
> >
> > > +               break;
> > > +       }
> > > +
> > > +       /* we reached the last symbol */
> > > +       if (idx == iter->nr_syms)
> > > +               return NULL;
> > > +       iter->idx = idx + 1;
> > > +       ret->name = name;
> > > +       ret->bind = GELF_ST_BIND(sym.st_info);
> > > +       ret->offset = offset;
> >
> > Why not just return entire GElf_Sym information and let user process
> > it as desired. So basically for each symbol you'll give back its name,
> > GElf_Sym info, and I'd return symbol index as well. That will keep
> > this very generic for future uses.
>
> ok, so you have other users of this iterator in mind already?

well, there is linker.c that also iterates ELF symbols, though that
one is assuming Elf64_Sym, so I wouldn't go updating it. So it's more
of a general feeling that "ELF symbol iterator" shouldn't assume
functions and func_offset translation, it should just return symbols.

>
> >
> > > +       return ret;
> >
> > I'd structure this a bit different. If we got out of loop, just return
> > NULL. Then inside the for loop, when we found the symbol, fill out ret
> > and return from inside the for loop. I think it's more
> > straightforward.
>
> ok, will change
>
> thanks,
> jirka
>
> >
> > > +}
> > > +
> > >  /* Find offset of function name in the provided ELF object. "binary_path" is
> > >   * the path to the ELF binary represented by "elf", and only used for error
> > >   * reporting matters. "name" matches symbol name or name@@LIB for library
> >
> > [...]

Re: [PATCHv2 bpf-next 06/24] libbpf: Add elf symbol iterator

[Andrii Nakryiko]

On Fri, Jun 23, 2023 at 1:19 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Jun 22, 2023 at 05:31:58PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jun 20, 2023 at 1:36 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > >
> > > Adding elf symbol iterator object (and some functions) that follow
> > > open-coded iterator pattern and some functions to ease up iterating
> > > elf object symbols.
> > >
> > > The idea is to iterate single symbol section with:
> > >
> > >   struct elf_symbol_iter iter;
> > >   struct elf_symbol *sym;
> > >
> > >   if (elf_symbol_iter_new(&iter, elf, binary_path, SHT_DYNSYM))
> > >         goto error;
> > >
> > >   while ((sym = elf_symbol_iter_next(&iter))) {
> > >         ...
> > >   }
> > >
> > > I considered opening the elf inside the iterator and iterate all symbol
> > > sections, but then it gets more complicated wrt user checks for when
> > > the next section is processed.
> > >
> > > Plus side is the we don't need 'exit' function, because caller/user is
> > > in charge of that.
> > >
> > > The returned iterated symbol object from elf_symbol_iter_next function
> > > is placed inside the struct elf_symbol_iter, so no extra allocation or
> > > argument is needed.
> > >
> > > Suggested-by: Andrii Nakryiko <andrii@kernel.org>
> > > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > > ---
> > >  tools/lib/bpf/libbpf.c | 179 ++++++++++++++++++++++++++---------------
> > >  1 file changed, 114 insertions(+), 65 deletions(-)
> > >
> >
> > This is great. Left a few nits below. I'm thinkin maybe we should add
> > a separate elf.c file for all these ELF-related helpers and start
> > offloading code from libbpf.c, which got pretty big already. WDYT?
>
> yes, I thought doing the move after this is merged might be better,
> because it's quite big already

true (and btw, please give me a bit more time to review the rest of
patches before posting a new version), but I'm ok with just going
straight to elf.c move. Either way it's a lot of +++ and ---, whether
it's in the same file or not doesn't matter all that much.


>
> >
> >
> > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > index af52188daa80..cdac368c7ce1 100644
> > > --- a/tools/lib/bpf/libbpf.c
> > > +++ b/tools/lib/bpf/libbpf.c
> > > @@ -10824,6 +10824,109 @@ static Elf_Scn *elf_find_next_scn_by_type(Elf *elf, int sh_type, Elf_Scn *scn)
> > >         return NULL;
> > >  }
> > >
> > > +struct elf_symbol {
> > > +       const char *name;
> > > +       unsigned long offset;
> > > +       int bind;
> > > +};
> > > +
> > > +struct elf_symbol_iter {
> >

[...]

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Alexei Starovoitov]

On Fri, Jun 23, 2023 at 9:24 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> > > > +
> > > > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > > > +                          unsigned long entry_ip,
> > > > +                          struct pt_regs *regs)
> > > > +{
> > > > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > > > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > > > +               .entry_ip = entry_ip,
> > > > +       };
> > > > +       struct bpf_prog *prog = link->link.prog;
> > > > +       struct bpf_run_ctx *old_run_ctx;
> > > > +       int err = 0;
> > > > +
> > > > +       might_fault();
> > > > +
> > > > +       rcu_read_lock_trace();
> > >
> > > we don't need this if uprobe is not sleepable, right? why unconditional then?
> >
> > I won't pretend I understand what rcu_read_lock_trace does ;-)
> >
> > I tried to follow bpf_prog_run_array_sleepable where it's called
> > unconditionally for both sleepable and non-sleepable progs
> >
> > there are conditional rcu_read_un/lock calls later on
> >
> > I will check
>
> hm... Alexei can chime in here, but given here we actually are trying
> to run one BPF program (not entire array of them), we do know whether
> it's going to be sleepable or not. So we can avoid unnecessary
> rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used
> when there is going to be sleepable BPF program executed to protect
> BPF maps and other resources from being freed too soon. But if we know
> that we don't need sleepable, we can avoid that.

We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(),
but it will likely be slower. These calls are very fast.
Simpler and faster to do it unconditionally even when the array doesn't
have sleepable progs.
rcu_read_lock() we have to do conditionally, because it won't be ok
if sleepable progs are in the array.

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Andrii Nakryiko]

On Fri, Jun 23, 2023 at 9:39 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Jun 23, 2023 at 9:24 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > > > > +
> > > > > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > > > > +                          unsigned long entry_ip,
> > > > > +                          struct pt_regs *regs)
> > > > > +{
> > > > > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > > > > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > > > > +               .entry_ip = entry_ip,
> > > > > +       };
> > > > > +       struct bpf_prog *prog = link->link.prog;
> > > > > +       struct bpf_run_ctx *old_run_ctx;
> > > > > +       int err = 0;
> > > > > +
> > > > > +       might_fault();
> > > > > +
> > > > > +       rcu_read_lock_trace();
> > > >
> > > > we don't need this if uprobe is not sleepable, right? why unconditional then?
> > >
> > > I won't pretend I understand what rcu_read_lock_trace does ;-)
> > >
> > > I tried to follow bpf_prog_run_array_sleepable where it's called
> > > unconditionally for both sleepable and non-sleepable progs
> > >
> > > there are conditional rcu_read_un/lock calls later on
> > >
> > > I will check
> >
> > hm... Alexei can chime in here, but given here we actually are trying
> > to run one BPF program (not entire array of them), we do know whether
> > it's going to be sleepable or not. So we can avoid unnecessary
> > rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used
> > when there is going to be sleepable BPF program executed to protect
> > BPF maps and other resources from being freed too soon. But if we know
> > that we don't need sleepable, we can avoid that.
>
> We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(),
> but it will likely be slower. These calls are very fast.

that's ok then. But seeing how we do

rcu_read_lock_trace();
if (!sleepable)
    rcu_read_lock();

it felt like we might as well just do

if (sleepable)
    rcu_read_lock_trace();
else
    rcu_read_lock();


As I mentioned, in this case we have a single bpf_prog, not a
bpf_prog_array, so that changes things a bit.

But ultimately, the context switch required for uprobe dwarfs overhead
of any of this, presumably, so it's a minor concern.

> Simpler and faster to do it unconditionally even when the array doesn't
> have sleepable progs.
> rcu_read_lock() we have to do conditionally, because it won't be ok
> if sleepable progs are in the array.

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Alexei Starovoitov]

On Fri, Jun 23, 2023 at 10:11 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Fri, Jun 23, 2023 at 9:39 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Jun 23, 2023 at 9:24 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > > > > +
> > > > > > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > > > > > +                          unsigned long entry_ip,
> > > > > > +                          struct pt_regs *regs)
> > > > > > +{
> > > > > > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > > > > > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > > > > > +               .entry_ip = entry_ip,
> > > > > > +       };
> > > > > > +       struct bpf_prog *prog = link->link.prog;
> > > > > > +       struct bpf_run_ctx *old_run_ctx;
> > > > > > +       int err = 0;
> > > > > > +
> > > > > > +       might_fault();
> > > > > > +
> > > > > > +       rcu_read_lock_trace();
> > > > >
> > > > > we don't need this if uprobe is not sleepable, right? why unconditional then?
> > > >
> > > > I won't pretend I understand what rcu_read_lock_trace does ;-)
> > > >
> > > > I tried to follow bpf_prog_run_array_sleepable where it's called
> > > > unconditionally for both sleepable and non-sleepable progs
> > > >
> > > > there are conditional rcu_read_un/lock calls later on
> > > >
> > > > I will check
> > >
> > > hm... Alexei can chime in here, but given here we actually are trying
> > > to run one BPF program (not entire array of them), we do know whether
> > > it's going to be sleepable or not. So we can avoid unnecessary
> > > rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used
> > > when there is going to be sleepable BPF program executed to protect
> > > BPF maps and other resources from being freed too soon. But if we know
> > > that we don't need sleepable, we can avoid that.
> >
> > We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(),
> > but it will likely be slower. These calls are very fast.
>
> that's ok then. But seeing how we do
>
> rcu_read_lock_trace();
> if (!sleepable)
>     rcu_read_lock();
>
> it felt like we might as well just do
>
> if (sleepable)
>     rcu_read_lock_trace();
> else
>     rcu_read_lock();
>
>
> As I mentioned, in this case we have a single bpf_prog, not a
> bpf_prog_array, so that changes things a bit.

Ahh. It's only one prog. I missed that. Above makes sense then.
But why is it not an array? We can attach multiple uprobes to the same
location. Anyway that can be dealt with later.

Re: [PATCHv2 bpf-next 08/24] libbpf: Add elf_find_multi_func_offset function

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding elf_find_multi_func_offset function that looks up
> offsets for symbols specified in syms array argument.
>
> Offsets are returned in allocated array with the 'cnt' size,
> that needs to be released by the caller.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
>  tools/lib/bpf/libbpf_internal.h |   2 +
>  2 files changed, 114 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 30d9e3b69114..1c310b718961 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
>         return ret;
>  }
>
> +struct elf_symbol_offset {
> +       const char *name;
> +       unsigned long offset;
> +       int bind;
> +       int idx;
> +};
> +
> +static int cmp_func_offset(const void *_a, const void *_b)
> +{
> +       const struct elf_symbol_offset *a = _a;
> +       const struct elf_symbol_offset *b = _b;
> +
> +       return strcmp(a->name, b->name);
> +}
> +
> +static int
> +__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
> +                            const char **syms, unsigned long **poffsets)
> +{
> +       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
> +       struct elf_symbol_offset *func_offs;
> +       int err = 0, i, idx, cnt_done = 0;
> +       unsigned long *offsets = NULL;
> +
> +       func_offs = calloc(cnt, sizeof(*func_offs));
> +       if (!func_offs)
> +               return -ENOMEM;
> +
> +       for (i = 0; i < cnt; i++) {
> +               func_offs[i].name = syms[i];
> +               func_offs[i].idx = i;
> +       }
> +
> +       qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
> +
> +       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
> +               struct elf_symbol_iter iter;
> +               struct elf_symbol *sym;
> +
> +               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))

a bit lax handling of initialization errors here, let's be a bit more
strict here?

> +                       continue;
> +
> +               while ((sym = elf_symbol_iter_next(&iter))) {
> +                       struct elf_symbol_offset *fo, tmp = {
> +                               .name = sym->name,
> +                       };
> +
> +                       fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
> +                                    cmp_func_offset);
> +                       if (!fo)
> +                               continue;
> +
> +                       if (fo->offset > 0) {
> +                               /* same offset, no problem */
> +                               if (fo->offset == sym->offset)
> +                                       continue;
> +                               /* handle multiple matches */
> +                               if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
> +                                       /* Only accept one non-weak bind. */
> +                                       pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
> +                                               sym->name, fo->name, binary_path);
> +                                       err = -LIBBPF_ERRNO__FORMAT;
> +                                       goto out;
> +                               } else if (sym->bind == STB_WEAK) {
> +                                       /* already have a non-weak bind, and
> +                                        * this is a weak bind, so ignore.
> +                                        */
> +                                       continue;
> +                               }
> +                       }
> +                       if (!fo->offset)
> +                               cnt_done++;
> +                       fo->offset = sym->offset;
> +                       fo->bind = sym->bind;
> +               }
> +       }
> +
> +       if (cnt != cnt_done) {
> +               err = -ENOENT;
> +               goto out;
> +       }
> +       offsets = calloc(cnt, sizeof(*offsets));

you can allocate it at the very beginning and fill it out based on
fo->idx, there is no need to store offset in elf_symbol_offset

> +       if (!offsets) {
> +               err = -ENOMEM;
> +               goto out;
> +       }
> +       for (i = 0; i < cnt; i++) {
> +               idx = func_offs[i].idx;
> +               offsets[idx] = func_offs[i].offset;
> +       }
> +
> +out:
> +       *poffsets = offsets;
> +       free(func_offs);
> +       return err;
> +}
> +
> +int elf_find_multi_func_offset(const char *binary_path, int cnt,
> +                              const char **syms, unsigned long **poffsets)
> +{
> +       struct elf_fd elf_fd = {};

do you need to initialize this struct?

> +       long ret = -ENOENT;

same here, you always override ret, so no need to init it?

> +
> +       ret = open_elf(binary_path, &elf_fd);
> +       if (ret)
> +               return ret;
> +
> +       ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);

is there a point of having elf_find_multi_func_offset and
__elf_find_multi_func_offset separately? can you please combine?


> +       close_elf(&elf_fd);
> +       return ret;
> +}
> +
>  /* Find offset of function name in ELF object specified by path. "name" matches
>   * symbol name or name@@LIB for library functions.
>   */
> diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> index e4d05662a96c..13d5c12fbd0b 100644
> --- a/tools/lib/bpf/libbpf_internal.h
> +++ b/tools/lib/bpf/libbpf_internal.h
> @@ -577,4 +577,6 @@ static inline bool is_pow_of_2(size_t x)
>  #define PROG_LOAD_ATTEMPTS 5
>  int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
>
> +int elf_find_multi_func_offset(const char *binary_path, int cnt,
> +                              const char **syms, unsigned long **poffsets);
>  #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 09/24] libbpf: Add elf_find_pattern_func_offset function

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding elf_find_pattern_func_offset function that looks up
> offsets for symbols specified by pattern argument.
>
> The 'pattern' argument allows wildcards (*?' supported).
>
> Offsets are returned in allocated array together with its
> size and needs to be released by the caller.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
>  tools/lib/bpf/libbpf_internal.h |  3 ++
>  2 files changed, 81 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 1c310b718961..3e5c88caf5d5 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
>         return ret;
>  }
>
> +static int
> +__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
> +                              const char ***pnames, unsigned long **poffsets, size_t *pcnt)
> +{
> +       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
> +       struct elf_symbol_offset *func_offs = NULL;
> +       unsigned long *offsets = NULL;
> +       const char **names = NULL;
> +       size_t func_offs_cnt = 0;
> +       size_t func_offs_cap = 0;
> +       int err = 0, i;
> +
> +       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
> +               struct elf_symbol_iter iter;
> +               struct elf_symbol *sym;
> +
> +               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
> +                       continue;

same as in the previous patch, error handling?

> +
> +               while ((sym = elf_symbol_iter_next(&iter))) {
> +                       if (!glob_match(sym->name, pattern))
> +                               continue;
> +
> +                       err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
> +                                               sizeof(*func_offs), func_offs_cnt + 1);
> +                       if (err)
> +                               goto out;
> +
> +                       func_offs[func_offs_cnt].offset = sym->offset;
> +                       func_offs[func_offs_cnt].name = strdup(sym->name);

check for NULL?

and I'm actually unsure why you need to reuse elf_symbol_offset struct
here? You just need names and offsets in two separate array, so just
do that?..

> +                       func_offs_cnt++;
> +               }
> +
> +               /* If we found anything in the first symbol section,
> +                * do not search others to avoid duplicates.
> +                */
> +               if (func_offs_cnt)
> +                       break;
> +       }
> +
> +       offsets = calloc(func_offs_cnt, sizeof(*offsets));
> +       names = calloc(func_offs_cnt, sizeof(*names));
> +       if (!offsets || !names) {
> +               free(offsets);
> +               free(names);
> +               err = -ENOMEM;
> +               goto out;
> +       }
> +
> +       for (i = 0; i < func_offs_cnt; i++) {
> +               offsets[i] = func_offs[i].offset;
> +               names[i] = func_offs[i].name;

see above, why not fill these out right away during elf symbols iteration?

> +       }
> +
> +       *pnames = names;
> +       *poffsets = offsets;
> +       *pcnt = func_offs_cnt;
> +out:
> +       free(func_offs);
> +       return err;
> +}
> +
> +int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
> +                                const char ***pnames, unsigned long **poffsets,
> +                                size_t *pcnt)
> +{
> +       struct elf_fd elf_fd = {};
> +       long ret = -ENOENT;
> +
> +       ret = open_elf(binary_path, &elf_fd);
> +       if (ret)
> +               return ret;
> +
> +       ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);

I don't really like these underscored functions,
elf_find_pattern_func_offset() already has to do goto out for clean
ups, this close_elf() thing is just another resource to clean up
there, I don't see much reason to separate open_elf/close_elf in this
case


> +       close_elf(&elf_fd);
> +       return ret;
> +}
> +
>  /* Find offset of function name in ELF object specified by path. "name" matches
>   * symbol name or name@@LIB for library functions.
>   */
> diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> index 13d5c12fbd0b..22b0834e7fe1 100644
> --- a/tools/lib/bpf/libbpf_internal.h
> +++ b/tools/lib/bpf/libbpf_internal.h
> @@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
>
>  int elf_find_multi_func_offset(const char *binary_path, int cnt,
>                                const char **syms, unsigned long **poffsets);
> +int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
> +                                const char ***pnames, unsigned long **poffsets,
> +                                size_t *pcnt);
>  #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 10/24] libbpf: Add bpf_link_create support for multi uprobes

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding new uprobe_multi struct to bpf_link_create_opts object
> to pass multiple uprobe data to link_create attr uapi.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/bpf.c | 11 +++++++++++
>  tools/lib/bpf/bpf.h | 11 ++++++++++-
>  2 files changed, 21 insertions(+), 1 deletion(-)
>

LGTM

Acked-by: Andrii Nakryiko <andrii@kernel.org>


> diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c
> index ed86b37d8024..0fd35c91f50c 100644
> --- a/tools/lib/bpf/bpf.c
> +++ b/tools/lib/bpf/bpf.c
> @@ -733,6 +733,17 @@ int bpf_link_create(int prog_fd, int target_fd,
>                 if (!OPTS_ZEROED(opts, kprobe_multi))
>                         return libbpf_err(-EINVAL);
>                 break;
> +       case BPF_TRACE_UPROBE_MULTI:
> +               attr.link_create.uprobe_multi.flags = OPTS_GET(opts, uprobe_multi.flags, 0);
> +               attr.link_create.uprobe_multi.cnt = OPTS_GET(opts, uprobe_multi.cnt, 0);
> +               attr.link_create.uprobe_multi.path = ptr_to_u64(OPTS_GET(opts, uprobe_multi.path, 0));
> +               attr.link_create.uprobe_multi.offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.offsets, 0));
> +               attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0));
> +               attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0));
> +               attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0);
> +               if (!OPTS_ZEROED(opts, uprobe_multi))
> +                       return libbpf_err(-EINVAL);
> +               break;
>         case BPF_TRACE_FENTRY:
>         case BPF_TRACE_FEXIT:
>         case BPF_MODIFY_RETURN:
> diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h
> index 9aa0ee473754..82979b4f2769 100644
> --- a/tools/lib/bpf/bpf.h
> +++ b/tools/lib/bpf/bpf.h
> @@ -346,13 +346,22 @@ struct bpf_link_create_opts {
>                         const unsigned long *addrs;
>                         const __u64 *cookies;
>                 } kprobe_multi;
> +               struct {
> +                       __u32 flags;
> +                       __u32 cnt;
> +                       const char *path;
> +                       const unsigned long *offsets;
> +                       const unsigned long *ref_ctr_offsets;
> +                       const __u64 *cookies;
> +                       __u32 pid;
> +               } uprobe_multi;
>                 struct {
>                         __u64 cookie;
>                 } tracing;
>         };
>         size_t :0;
>  };
> -#define bpf_link_create_opts__last_field kprobe_multi.cookies
> +#define bpf_link_create_opts__last_field uprobe_multi.pid
>
>  LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
>                                enum bpf_attach_type attach_type,
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 11/24] libbpf: Add bpf_program__attach_uprobe_multi_opts function

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding bpf_program__attach_uprobe_multi_opts function that
> allows to attach multiple uprobes with uprobe_multi link.
>
> The user can specify uprobes with direct arguments:
>
>   binary_path/func_pattern
>
> or with struct bpf_uprobe_multi_opts opts argument fields:
>
>   const char *path;
>   const char **syms;
>   const unsigned long *offsets;
>   const unsigned long *ref_ctr_offsets;
>
> User can specify 3 mutually exclusive set of incputs:

typo: inputs

>
>  1) use only binary_path/func_pattern aruments

typo: arguments

>
>  2) use only opts argument with allowed combinations of:
>     path/offsets/ref_ctr_offsets/cookies/cnt
>
>  3) use binary_path with allowed combinations of:
>     syms/offsets/ref_ctr_offsets/cookies/cnt
>

why do we need this duplication of binary_path and opts.path? same for
pid and opts.pid?

> Any other usage results in error.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
>  tools/lib/bpf/libbpf.h   |  31 +++++++++
>  tools/lib/bpf/libbpf.map |   1 +
>  3 files changed, 163 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 3e5c88caf5d5..d972cea4c658 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
>         return -ENOENT;
>  }
>
> +struct bpf_link *
> +bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,

let's call it just bpf_program__attach_uprobe_multi()

we should have done that with bpf_program__attach_kprobe_multi_opts()
as well. Generally, if the only variant of API is the one with opts,
then there is no point in adding "_opts" suffix to the API name, IMO


> +                                     pid_t pid,
> +                                     const char *binary_path,
> +                                     const char *func_pattern,
> +                                     const struct bpf_uprobe_multi_opts *opts)
> +{
> +       const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
> +       LIBBPF_OPTS(bpf_link_create_opts, lopts);
> +       unsigned long *resolved_offsets = NULL;
> +       const char **resolved_symbols = NULL;
> +       int err = 0, link_fd, prog_fd;
> +       struct bpf_link *link = NULL;
> +       char errmsg[STRERR_BUFSIZE];
> +       const char *path, **syms;
> +       char full_path[PATH_MAX];
> +       const __u64 *cookies;
> +       bool has_pattern;
> +       bool retprobe;
> +       size_t cnt;
> +
> +       if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
> +               return libbpf_err_ptr(-EINVAL);
> +
> +       path = OPTS_GET(opts, path, NULL);
> +       syms = OPTS_GET(opts, syms, NULL);
> +       offsets = OPTS_GET(opts, offsets, NULL);
> +       ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
> +       cookies = OPTS_GET(opts, cookies, NULL);
> +       cnt = OPTS_GET(opts, cnt, 0);
> +
> +       /*
> +        * User can specify 3 mutually exclusive set of incputs:
> +        *
> +        * 1) use only binary_path/func_pattern aruments

same typos

> +        *
> +        * 2) use only opts argument with allowed combinations of:
> +        *    path/offsets/ref_ctr_offsets/cookies/cnt
> +        *
> +        * 3) use binary_path with allowed combinations of:
> +        *    syms/offsets/ref_ctr_offsets/cookies/cnt
> +        *
> +        * Any other usage results in error.
> +        */
> +


Looking at this part (sorry, I already trimmed reply before realizing
I want to comment on this):


+               err = elf_find_pattern_func_offset(binary_path, func_pattern,
+                                                  &resolved_symbols,
&resolved_offsets,
+                                                  &cnt);
+               if (err < 0)
+                       return libbpf_err_ptr(err);
+               offsets = resolved_offsets;
+       } else if (syms) {
+               err = elf_find_multi_func_offset(binary_path, cnt,
syms, &resolved_offsets);

Few thoughts.

First, we are not using resolved_symbols returned from
elf_find_pattern_func_offset(), do we? If not, let's drop it.

Second, I'm not a big fan of chosen naming. Maybe something like
elf_resolve_{pattern,syms}_offsets?


[...]

> +error:
> +       free(resolved_offsets);
> +       free(resolved_symbols);
> +       free(link);
> +       return libbpf_err_ptr(err);
> +}
> +
>  LIBBPF_API struct bpf_link *
>  bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
>                                 const char *binary_path, size_t func_offset,
> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
> index 754da73c643b..b6ff7d69a1d7 100644
> --- a/tools/lib/bpf/libbpf.h
> +++ b/tools/lib/bpf/libbpf.h
> @@ -529,6 +529,37 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
>                                       const char *pattern,
>                                       const struct bpf_kprobe_multi_opts *opts);
>
> +struct bpf_uprobe_multi_opts {
> +       /* size of this struct, for forward/backward compatibility */
> +       size_t sz;
> +       /* path to attach */
> +       const char *path;

how is this different from binary_path? not clear why we need to paths

> +       /* array of function symbols to attach */
> +       const char **syms;
> +       /* array of function addresses to attach */
> +       const unsigned long *offsets;
> +       /* array of refctr offsets to attach */
> +       const unsigned long *ref_ctr_offsets;
> +       /* array of user-provided values fetchable through bpf_get_attach_cookie */
> +       const __u64 *cookies;
> +       /* number of elements in syms/addrs/cookies arrays */
> +       size_t cnt;
> +       /* create return uprobes */
> +       bool retprobe;
> +       /* pid filter */
> +       int pid;

another duplicate, we already pass pid argument


> +       size_t :0;
> +};
> +
> +#define bpf_uprobe_multi_opts__last_field pid
> +
> +LIBBPF_API struct bpf_link *
> +bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
> +                                     pid_t pid,
> +                                     const char *binary_path,
> +                                     const char *func_pattern,
> +                                     const struct bpf_uprobe_multi_opts *opts);
> +
>  struct bpf_ksyscall_opts {
>         /* size of this struct, for forward/backward compatibility */
>         size_t sz;
> diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> index 7521a2fb7626..81558ef1bc38 100644
> --- a/tools/lib/bpf/libbpf.map
> +++ b/tools/lib/bpf/libbpf.map
> @@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
>                 bpf_link_get_info_by_fd;
>                 bpf_map_get_info_by_fd;
>                 bpf_prog_get_info_by_fd;
> +               bpf_program__attach_uprobe_multi_opts;

should be in 1.3.0?

>  } LIBBPF_1.1.0;
>
>  LIBBPF_1.3.0 {


> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 12/24] libbpf: Add support for u[ret]probe.multi[.s] program sections

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding support for several uprobe_multi program sections
> to allow auto attach of multi_uprobe programs.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 40 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index d972cea4c658..e42080258ec7 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
>  static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
>  static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
>  static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> +static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
>  static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
>  static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
>
> @@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
>         SEC_DEF("uretprobe.s+",         KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
>         SEC_DEF("kprobe.multi+",        KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
>         SEC_DEF("kretprobe.multi+",     KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
> +       SEC_DEF("uprobe.multi+",        KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
> +       SEC_DEF("uretprobe.multi+",     KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
> +       SEC_DEF("uprobe.multi.s+",      KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
> +       SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
>         SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
>         SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
>         SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
> @@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
>         return libbpf_get_error(*link);
>  }
>
> +static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
> +{
> +       char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
> +       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
> +       int n, ret = -EINVAL;
> +
> +       *link = NULL;
> +
> +       n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
> +                  &probe_type, &binary_path, &func_name);
> +       switch (n) {
> +       case 1:
> +               /* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
> +               ret = 0;
> +               break;
> +       case 2:
> +               pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",

message copy/pasted? We don't support +offset part, and it's not
"function", but "pattern" or something along those lines?

> +                       prog->name, prog->sec_name);
> +               break;
> +       case 3:
> +               opts.retprobe = strcmp(probe_type, "uretprobe.multi");

== 0


> +               *link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
> +               ret = libbpf_get_error(*link);
> +               break;
> +       default:
> +               pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
> +                       prog->sec_name);
> +               break;
> +       }
> +       free(probe_type);
> +       free(binary_path);
> +       free(func_name);
> +       return ret;
> +}
> +
>  static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
>                                          const char *binary_path, uint64_t offset)
>  {
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:38 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding uprobe-multi link detection. It will be used later in
> bpf_program__attach_usdt function to check and use uprobe_multi
> link over standard uprobe links.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
>  tools/lib/bpf/libbpf_internal.h |  2 ++
>  2 files changed, 31 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index e42080258ec7..3d570898459e 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
>         return link_fd < 0 && err == -EBADF;
>  }
>
> +static int probe_uprobe_multi_link(void)
> +{
> +       struct bpf_insn insns[] = {
> +               BPF_MOV64_IMM(BPF_REG_0, 0),
> +               BPF_EXIT_INSN(),
> +       };
> +       int prog_fd, link_fd, err;
> +
> +       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
> +                               insns, ARRAY_SIZE(insns), NULL);

I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?

> +       if (prog_fd < 0)
> +               return -errno;
> +
> +       /* No need to specify attach function. If the link is not supported
> +        * we will get -EOPNOTSUPP error before any other check is performed.

what will actually return this -EOPNOTSUPP? I couldn't find this in
the code quickly, can you please point me where?

> +        */
> +       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
> +       err = -errno; /* close() can clobber errno */
> +
> +       if (link_fd >= 0)
> +               close(link_fd);
> +       close(prog_fd);
> +
> +       return link_fd < 0 && err != -EOPNOTSUPP;
> +}
> +
>  static int probe_kern_bpf_cookie(void)
>  {
>         struct bpf_insn insns[] = {
> @@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
>         [FEAT_SYSCALL_WRAPPER] = {
>                 "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
>         },
> +       [FEAT_UPROBE_LINK] = {
> +               "BPF uprobe multi link support", probe_uprobe_multi_link,
> +       },
>  };
>
>  bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
> diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> index 22b0834e7fe1..a257eb81af25 100644
> --- a/tools/lib/bpf/libbpf_internal.h
> +++ b/tools/lib/bpf/libbpf_internal.h
> @@ -354,6 +354,8 @@ enum kern_feature_id {
>         FEAT_BTF_ENUM64,
>         /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
>         FEAT_SYSCALL_WRAPPER,
> +       /* BPF uprobe_multi link support */
> +       FEAT_UPROBE_LINK,

UPROBE_MULTI_LINK, we might have non-multi link in the future as well

>         __FEAT_CNT,
>  };


>
> --
> 2.41.0
>

Re: [PATCHv2 bpf-next 14/24] libbpf: Add uprobe multi link support to bpf_program__attach_usdt

[Andrii Nakryiko]

On Tue, Jun 20, 2023 at 1:38 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding support for usdt_manager_attach_usdt to use uprobe_multi
> link to attach to usdt probes.
>
> The uprobe_multi support is detected before the usdt program is
> loaded and its expected_attach_type is set accordingly.
>
> If uprobe_multi support is detected the usdt_manager_attach_usdt
> gathers uprobes info and calls bpf_program__attach_uprobe_opts to
> create all needed uprobes.
>
> If uprobe_multi support is not detected the old behaviour stays.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/lib/bpf/libbpf.c |  12 ++++-
>  tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
>  2 files changed, 99 insertions(+), 33 deletions(-)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 3d570898459e..9c7a67c5cbe8 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -363,6 +363,8 @@ enum sec_def_flags {
>         SEC_SLEEPABLE = 8,
>         /* BPF program support non-linear XDP buffer */
>         SEC_XDP_FRAGS = 16,
> +       /* Setup proper attach type for usdt probes. */
> +       SEC_USDT = 32,
>  };
>
>  struct bpf_sec_def {
> @@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
>         if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
>                 opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
>
> +       /* special check for usdt to use uprobe_multi link */
> +       if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
> +               prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;

this is quite ugly. I think KPROBE programs do not have enforcement
for expected_attach_type during BPF_PROG_LOAD, so we can set
BPF_TRACE_UPROBE_MULTI unconditionally, right?

> +
>         if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
>                 int btf_obj_fd = 0, btf_type_id = 0, err;
>                 const char *attach_name;
> @@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
>         if (!insns || !insns_cnt)
>                 return -EINVAL;
>
> -       load_attr.expected_attach_type = prog->expected_attach_type;
>         if (kernel_supports(obj, FEAT_PROG_NAME))
>                 prog_name = prog->name;
>         load_attr.attach_prog_fd = prog->attach_prog_fd;
> @@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
>                 insns_cnt = prog->insns_cnt;
>         }
>
> +       /* allow prog_prepare_load_fn to change expected_attach_type */
> +       load_attr.expected_attach_type = prog->expected_attach_type;
> +
>         if (obj->gen_loader) {
>                 bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
>                                    license, insns, insns_cnt, &load_attr,
> @@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
>         SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
>         SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
>         SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
> -       SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
> +       SEC_DEF("usdt+",                KPROBE, 0, SEC_USDT, attach_usdt),

btw, given you are touching USDT stuff, can you please also add
sleepable USDT (usdt.s+) ?


>         SEC_DEF("tc",                   SCHED_CLS, 0, SEC_NONE),
>         SEC_DEF("classifier",           SCHED_CLS, 0, SEC_NONE),
>         SEC_DEF("action",               SCHED_ACT, 0, SEC_NONE),
> diff --git a/tools/lib/bpf/usdt.c b/tools/lib/bpf/usdt.c
> index f1a141555f08..33f0a2b4cc1c 100644
> --- a/tools/lib/bpf/usdt.c
> +++ b/tools/lib/bpf/usdt.c
> @@ -808,6 +808,16 @@ struct bpf_link_usdt {
>                 long abs_ip;
>                 struct bpf_link *link;
>         } *uprobes;
> +
> +       bool has_uprobe_multi;
> +
> +       struct {
> +               char *path;
> +               unsigned long *offsets;
> +               unsigned long *ref_ctr_offsets;
> +               __u64 *cookies;

you shouldn't need to persist this, this can be allocated and freed
inside usdt_manager_attach_usdt(), you only need the link pointer


> +               struct bpf_link *link;
> +       } uprobe_multi;
>  };
>
>  static int bpf_link_usdt_detach(struct bpf_link *link)
> @@ -816,19 +826,23 @@ static int bpf_link_usdt_detach(struct bpf_link *link)
>         struct usdt_manager *man = usdt_link->usdt_man;
>         int i;
>
> -       for (i = 0; i < usdt_link->uprobe_cnt; i++) {
> -               /* detach underlying uprobe link */
> -               bpf_link__destroy(usdt_link->uprobes[i].link);
> -               /* there is no need to update specs map because it will be
> -                * unconditionally overwritten on subsequent USDT attaches,
> -                * but if BPF cookies are not used we need to remove entry
> -                * from ip_to_spec_id map, otherwise we'll run into false
> -                * conflicting IP errors
> -                */
> -               if (!man->has_bpf_cookie) {
> -                       /* not much we can do about errors here */
> -                       (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
> -                                                 &usdt_link->uprobes[i].abs_ip);
> +       if (usdt_link->has_uprobe_multi) {
> +               bpf_link__destroy(usdt_link->uprobe_multi.link);
> +       } else {
> +               for (i = 0; i < usdt_link->uprobe_cnt; i++) {
> +                       /* detach underlying uprobe link */
> +                       bpf_link__destroy(usdt_link->uprobes[i].link);
> +                       /* there is no need to update specs map because it will be
> +                        * unconditionally overwritten on subsequent USDT attaches,
> +                        * but if BPF cookies are not used we need to remove entry
> +                        * from ip_to_spec_id map, otherwise we'll run into false
> +                        * conflicting IP errors
> +                        */
> +                       if (!man->has_bpf_cookie) {
> +                               /* not much we can do about errors here */
> +                               (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
> +                                                         &usdt_link->uprobes[i].abs_ip);
> +                       }
>                 }

you can avoid shifting all this by keeping uprobe_cnt to zero

bpf_link__destory(usdt_link->uprobe_multi.link) will work fine for NULL

so just do both clean ups sequentially, knowing that only one of them
will actually do anything

>         }
>
> @@ -868,9 +882,15 @@ static void bpf_link_usdt_dealloc(struct bpf_link *link)
>  {
>         struct bpf_link_usdt *usdt_link = container_of(link, struct bpf_link_usdt, link);
>
> -       free(usdt_link->spec_ids);
> -       free(usdt_link->uprobes);
> -       free(usdt_link);
> +       if (usdt_link->has_uprobe_multi) {
> +               free(usdt_link->uprobe_multi.offsets);
> +               free(usdt_link->uprobe_multi.ref_ctr_offsets);
> +               free(usdt_link->uprobe_multi.cookies);
> +       } else {
> +               free(usdt_link->spec_ids);
> +               free(usdt_link->uprobes);
> +               free(usdt_link);
> +       }

similar to the above, just do *all* the clean up unconditionally and
rely on free() handling NULLs just fine

>  }
>
>  static size_t specs_hash_fn(long key, void *ctx)
> @@ -943,11 +963,13 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
>                                           const char *usdt_provider, const char *usdt_name,
>                                           __u64 usdt_cookie)
>  {
> +       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts_multi);
>         int i, fd, err, spec_map_fd, ip_map_fd;
>         LIBBPF_OPTS(bpf_uprobe_opts, opts);
>         struct hashmap *specs_hash = NULL;
>         struct bpf_link_usdt *link = NULL;
>         struct usdt_target *targets = NULL;
> +       struct bpf_link *uprobe_link;
>         size_t target_cnt;
>         Elf *elf;
>
> @@ -1003,16 +1025,29 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
>         link->usdt_man = man;
>         link->link.detach = &bpf_link_usdt_detach;
>         link->link.dealloc = &bpf_link_usdt_dealloc;
> +       link->has_uprobe_multi = bpf_program__expected_attach_type(prog) == BPF_TRACE_UPROBE_MULTI;

just use kernel_supports(), it's cleaner (and result is cached, so
it's not less efficient)

>
> -       link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
> -       if (!link->uprobes) {
> -               err = -ENOMEM;
> -               goto err_out;
> +       if (link->has_uprobe_multi) {
> +               link->uprobe_multi.offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.offsets));
> +               link->uprobe_multi.ref_ctr_offsets = calloc(target_cnt, sizeof(*link->uprobe_multi.ref_ctr_offsets));
> +               link->uprobe_multi.cookies = calloc(target_cnt, sizeof(*link->uprobe_multi.cookies));
> +
> +               if (!link->uprobe_multi.offsets ||
> +                   !link->uprobe_multi.ref_ctr_offsets ||
> +                   !link->uprobe_multi.cookies) {
> +                       err = -ENOMEM;
> +                       goto err_out;
> +               }
> +       } else {
> +               link->uprobes = calloc(target_cnt, sizeof(*link->uprobes));
> +               if (!link->uprobes) {
> +                       err = -ENOMEM;
> +                       goto err_out;
> +               }
>         }
>

[...]

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:40:20PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:38 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding uprobe-multi link detection. It will be used later in
> > bpf_program__attach_usdt function to check and use uprobe_multi
> > link over standard uprobe links.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
> >  tools/lib/bpf/libbpf_internal.h |  2 ++
> >  2 files changed, 31 insertions(+)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index e42080258ec7..3d570898459e 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
> >         return link_fd < 0 && err == -EBADF;
> >  }
> >
> > +static int probe_uprobe_multi_link(void)
> > +{
> > +       struct bpf_insn insns[] = {
> > +               BPF_MOV64_IMM(BPF_REG_0, 0),
> > +               BPF_EXIT_INSN(),
> > +       };
> > +       int prog_fd, link_fd, err;
> > +
> > +       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
> > +                               insns, ARRAY_SIZE(insns), NULL);
> 
> I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?

hm it should.. I guess it worked because of the KPROBE/UPROBE
typo you found in the patch earlier, will check

> 
> > +       if (prog_fd < 0)
> > +               return -errno;
> > +
> > +       /* No need to specify attach function. If the link is not supported
> > +        * we will get -EOPNOTSUPP error before any other check is performed.
> 
> what will actually return this -EOPNOTSUPP? I couldn't find this in
> the code quickly, can you please point me where?

	#else /* !CONFIG_UPROBES */
	int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
	{
		return -EOPNOTSUPP;
	}

> 
> > +        */
> > +       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
> > +       err = -errno; /* close() can clobber errno */
> > +
> > +       if (link_fd >= 0)
> > +               close(link_fd);
> > +       close(prog_fd);
> > +
> > +       return link_fd < 0 && err != -EOPNOTSUPP;
> > +}
> > +
> >  static int probe_kern_bpf_cookie(void)
> >  {
> >         struct bpf_insn insns[] = {
> > @@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
> >         [FEAT_SYSCALL_WRAPPER] = {
> >                 "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
> >         },
> > +       [FEAT_UPROBE_LINK] = {
> > +               "BPF uprobe multi link support", probe_uprobe_multi_link,
> > +       },
> >  };
> >
> >  bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
> > diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> > index 22b0834e7fe1..a257eb81af25 100644
> > --- a/tools/lib/bpf/libbpf_internal.h
> > +++ b/tools/lib/bpf/libbpf_internal.h
> > @@ -354,6 +354,8 @@ enum kern_feature_id {
> >         FEAT_BTF_ENUM64,
> >         /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
> >         FEAT_SYSCALL_WRAPPER,
> > +       /* BPF uprobe_multi link support */
> > +       FEAT_UPROBE_LINK,
> 
> UPROBE_MULTI_LINK, we might have non-multi link in the future as well

ok

thanks,
jirka

Re: [PATCHv2 bpf-next 14/24] libbpf: Add uprobe multi link support to bpf_program__attach_usdt

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:40:27PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:38 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding support for usdt_manager_attach_usdt to use uprobe_multi
> > link to attach to usdt probes.
> >
> > The uprobe_multi support is detected before the usdt program is
> > loaded and its expected_attach_type is set accordingly.
> >
> > If uprobe_multi support is detected the usdt_manager_attach_usdt
> > gathers uprobes info and calls bpf_program__attach_uprobe_opts to
> > create all needed uprobes.
> >
> > If uprobe_multi support is not detected the old behaviour stays.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c |  12 ++++-
> >  tools/lib/bpf/usdt.c   | 120 ++++++++++++++++++++++++++++++-----------
> >  2 files changed, 99 insertions(+), 33 deletions(-)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index 3d570898459e..9c7a67c5cbe8 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -363,6 +363,8 @@ enum sec_def_flags {
> >         SEC_SLEEPABLE = 8,
> >         /* BPF program support non-linear XDP buffer */
> >         SEC_XDP_FRAGS = 16,
> > +       /* Setup proper attach type for usdt probes. */
> > +       SEC_USDT = 32,
> >  };
> >
> >  struct bpf_sec_def {
> > @@ -6799,6 +6801,10 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog,
> >         if (prog->type == BPF_PROG_TYPE_XDP && (def & SEC_XDP_FRAGS))
> >                 opts->prog_flags |= BPF_F_XDP_HAS_FRAGS;
> >
> > +       /* special check for usdt to use uprobe_multi link */
> > +       if ((def & SEC_USDT) && kernel_supports(NULL, FEAT_UPROBE_LINK))
> > +               prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;
> 
> this is quite ugly. I think KPROBE programs do not have enforcement
> for expected_attach_type during BPF_PROG_LOAD, so we can set
> BPF_TRACE_UPROBE_MULTI unconditionally, right?

t doesn't but we are adding one earlier to bpf_prog_attach_check_attach_type
(the one with U/K typo):

	+       case BPF_PROG_TYPE_KPROBE:
	+               if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI &&
	+                   attach_type != BPF_TRACE_UPROBE_MULTI)
	+                       return -EINVAL;



for same reasons we did the check for kprobe_multi in:

  db8eae6bc5c7 bpf: Force kprobe multi expected_attach_type for kprobe_multi link

> 
> > +
> >         if ((def & SEC_ATTACH_BTF) && !prog->attach_btf_id) {
> >                 int btf_obj_fd = 0, btf_type_id = 0, err;
> >                 const char *attach_name;
> > @@ -6867,7 +6873,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
> >         if (!insns || !insns_cnt)
> >                 return -EINVAL;
> >
> > -       load_attr.expected_attach_type = prog->expected_attach_type;
> >         if (kernel_supports(obj, FEAT_PROG_NAME))
> >                 prog_name = prog->name;
> >         load_attr.attach_prog_fd = prog->attach_prog_fd;
> > @@ -6903,6 +6908,9 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
> >                 insns_cnt = prog->insns_cnt;
> >         }
> >
> > +       /* allow prog_prepare_load_fn to change expected_attach_type */
> > +       load_attr.expected_attach_type = prog->expected_attach_type;
> > +
> >         if (obj->gen_loader) {
> >                 bpf_gen__prog_load(obj->gen_loader, prog->type, prog->name,
> >                                    license, insns, insns_cnt, &load_attr,
> > @@ -8703,7 +8711,7 @@ static const struct bpf_sec_def section_defs[] = {
> >         SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
> >         SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
> >         SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
> > -       SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
> > +       SEC_DEF("usdt+",                KPROBE, 0, SEC_USDT, attach_usdt),
> 
> btw, given you are touching USDT stuff, can you please also add
> sleepable USDT (usdt.s+) ?

ok

> 
> 
> >         SEC_DEF("tc",                   SCHED_CLS, 0, SEC_NONE),
> >         SEC_DEF("classifier",           SCHED_CLS, 0, SEC_NONE),
> >         SEC_DEF("action",               SCHED_ACT, 0, SEC_NONE),
> > diff --git a/tools/lib/bpf/usdt.c b/tools/lib/bpf/usdt.c
> > index f1a141555f08..33f0a2b4cc1c 100644
> > --- a/tools/lib/bpf/usdt.c
> > +++ b/tools/lib/bpf/usdt.c
> > @@ -808,6 +808,16 @@ struct bpf_link_usdt {
> >                 long abs_ip;
> >                 struct bpf_link *link;
> >         } *uprobes;
> > +
> > +       bool has_uprobe_multi;
> > +
> > +       struct {
> > +               char *path;
> > +               unsigned long *offsets;
> > +               unsigned long *ref_ctr_offsets;
> > +               __u64 *cookies;
> 
> you shouldn't need to persist this, this can be allocated and freed
> inside usdt_manager_attach_usdt(), you only need the link pointer

aah right, that can go away, nice

> 
> 
> > +               struct bpf_link *link;
> > +       } uprobe_multi;
> >  };
> >
> >  static int bpf_link_usdt_detach(struct bpf_link *link)
> > @@ -816,19 +826,23 @@ static int bpf_link_usdt_detach(struct bpf_link *link)
> >         struct usdt_manager *man = usdt_link->usdt_man;
> >         int i;
> >
> > -       for (i = 0; i < usdt_link->uprobe_cnt; i++) {
> > -               /* detach underlying uprobe link */
> > -               bpf_link__destroy(usdt_link->uprobes[i].link);
> > -               /* there is no need to update specs map because it will be
> > -                * unconditionally overwritten on subsequent USDT attaches,
> > -                * but if BPF cookies are not used we need to remove entry
> > -                * from ip_to_spec_id map, otherwise we'll run into false
> > -                * conflicting IP errors
> > -                */
> > -               if (!man->has_bpf_cookie) {
> > -                       /* not much we can do about errors here */
> > -                       (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
> > -                                                 &usdt_link->uprobes[i].abs_ip);
> > +       if (usdt_link->has_uprobe_multi) {
> > +               bpf_link__destroy(usdt_link->uprobe_multi.link);
> > +       } else {
> > +               for (i = 0; i < usdt_link->uprobe_cnt; i++) {
> > +                       /* detach underlying uprobe link */
> > +                       bpf_link__destroy(usdt_link->uprobes[i].link);
> > +                       /* there is no need to update specs map because it will be
> > +                        * unconditionally overwritten on subsequent USDT attaches,
> > +                        * but if BPF cookies are not used we need to remove entry
> > +                        * from ip_to_spec_id map, otherwise we'll run into false
> > +                        * conflicting IP errors
> > +                        */
> > +                       if (!man->has_bpf_cookie) {
> > +                               /* not much we can do about errors here */
> > +                               (void)bpf_map_delete_elem(bpf_map__fd(man->ip_to_spec_id_map),
> > +                                                         &usdt_link->uprobes[i].abs_ip);
> > +                       }
> >                 }
> 
> you can avoid shifting all this by keeping uprobe_cnt to zero
> 
> bpf_link__destory(usdt_link->uprobe_multi.link) will work fine for NULL
> 
> so just do both clean ups sequentially, knowing that only one of them
> will actually do anything

ok, good idea

> 
> >         }
> >
> > @@ -868,9 +882,15 @@ static void bpf_link_usdt_dealloc(struct bpf_link *link)
> >  {
> >         struct bpf_link_usdt *usdt_link = container_of(link, struct bpf_link_usdt, link);
> >
> > -       free(usdt_link->spec_ids);
> > -       free(usdt_link->uprobes);
> > -       free(usdt_link);
> > +       if (usdt_link->has_uprobe_multi) {
> > +               free(usdt_link->uprobe_multi.offsets);
> > +               free(usdt_link->uprobe_multi.ref_ctr_offsets);
> > +               free(usdt_link->uprobe_multi.cookies);
> > +       } else {
> > +               free(usdt_link->spec_ids);
> > +               free(usdt_link->uprobes);
> > +               free(usdt_link);
> > +       }
> 
> similar to the above, just do *all* the clean up unconditionally and
> rely on free() handling NULLs just fine

ok

> 
> >  }
> >
> >  static size_t specs_hash_fn(long key, void *ctx)
> > @@ -943,11 +963,13 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
> >                                           const char *usdt_provider, const char *usdt_name,
> >                                           __u64 usdt_cookie)
> >  {
> > +       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts_multi);
> >         int i, fd, err, spec_map_fd, ip_map_fd;
> >         LIBBPF_OPTS(bpf_uprobe_opts, opts);
> >         struct hashmap *specs_hash = NULL;
> >         struct bpf_link_usdt *link = NULL;
> >         struct usdt_target *targets = NULL;
> > +       struct bpf_link *uprobe_link;
> >         size_t target_cnt;
> >         Elf *elf;
> >
> > @@ -1003,16 +1025,29 @@ struct bpf_link *usdt_manager_attach_usdt(struct usdt_manager *man, const struct
> >         link->usdt_man = man;
> >         link->link.detach = &bpf_link_usdt_detach;
> >         link->link.dealloc = &bpf_link_usdt_dealloc;
> > +       link->has_uprobe_multi = bpf_program__expected_attach_type(prog) == BPF_TRACE_UPROBE_MULTI;
> 
> just use kernel_supports(), it's cleaner (and result is cached, so
> it's not less efficient)

ok

thanks,
jirka

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

On Fri, Jun 23, 2023 at 09:24:22AM -0700, Andrii Nakryiko wrote:

SNIP

> > > > +
> > > > +       if (!uprobes || !ref_ctr_offsets || !link)
> > > > +               goto error_free;
> > > > +
> > > > +       for (i = 0; i < cnt; i++) {
> > > > +               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
> > > > +                       err = -EFAULT;
> > > > +                       goto error_free;
> > > > +               }
> > > > +               if (__get_user(offset, uoffsets + i)) {
> > > > +                       err = -EFAULT;
> > > > +                       goto error_free;
> > > > +               }
> > > > +
> > > > +               uprobes[i].offset = offset;
> > > > +               uprobes[i].link = link;
> > > > +
> > > > +               if (flags & BPF_F_UPROBE_MULTI_RETURN)
> > > > +                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
> > > > +               else
> > > > +                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
> > > > +
> > > > +               ref_ctr_offsets[i] = ref_ctr_offset;
> > > > +       }
> > > > +
> > > > +       link->cnt = cnt;
> > > > +       link->uprobes = uprobes;
> > > > +       link->path = path;
> > > > +
> > > > +       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
> > > > +                     &bpf_uprobe_multi_link_lops, prog);
> > > > +
> > > > +       err = bpf_link_prime(&link->link, &link_primer);
> > > > +       if (err)
> > > > +               goto error_free;
> > > > +
> > > > +       for (i = 0; i < cnt; i++) {
> > > > +               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
> > > > +                                            uprobes[i].offset, ref_ctr_offsets[i],
> > > > +                                            &uprobes[i].consumer);
> > > > +               if (err) {
> > > > +                       bpf_uprobe_unregister(&path, uprobes, i);
> > >
> > > bpf_link_cleanup() will do this through
> > > bpf_uprobe_multi_link_release(), no? So you are double unregistering?
> > > Either drop cnt to zero, or just don't do this here? Latter is better,
> > > IMO.
> >
> > bpf_link_cleanup path won't call release callback so we have to do that
> 
> bpf_link_cleanup() does fput(primer->file); which eventually calls
> release callback, no? I'd add printk and simulate failure just to be
> sure

I recall we had similar discussion for kprobe_multi link ;-)

I'll double check that but I think bpf_link_cleanup calls just
dealloc callback not release

jirka

> 
> >
> > I think I can add simple selftest to have this path covered
> >
> > thanks,
> > jirka

SNIP

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

On Fri, Jun 23, 2023 at 10:20:26AM -0700, Alexei Starovoitov wrote:
> On Fri, Jun 23, 2023 at 10:11 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Fri, Jun 23, 2023 at 9:39 AM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Fri, Jun 23, 2023 at 9:24 AM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > > > > +
> > > > > > > +static int uprobe_prog_run(struct bpf_uprobe *uprobe,
> > > > > > > +                          unsigned long entry_ip,
> > > > > > > +                          struct pt_regs *regs)
> > > > > > > +{
> > > > > > > +       struct bpf_uprobe_multi_link *link = uprobe->link;
> > > > > > > +       struct bpf_uprobe_multi_run_ctx run_ctx = {
> > > > > > > +               .entry_ip = entry_ip,
> > > > > > > +       };
> > > > > > > +       struct bpf_prog *prog = link->link.prog;
> > > > > > > +       struct bpf_run_ctx *old_run_ctx;
> > > > > > > +       int err = 0;
> > > > > > > +
> > > > > > > +       might_fault();
> > > > > > > +
> > > > > > > +       rcu_read_lock_trace();
> > > > > >
> > > > > > we don't need this if uprobe is not sleepable, right? why unconditional then?
> > > > >
> > > > > I won't pretend I understand what rcu_read_lock_trace does ;-)
> > > > >
> > > > > I tried to follow bpf_prog_run_array_sleepable where it's called
> > > > > unconditionally for both sleepable and non-sleepable progs
> > > > >
> > > > > there are conditional rcu_read_un/lock calls later on
> > > > >
> > > > > I will check
> > > >
> > > > hm... Alexei can chime in here, but given here we actually are trying
> > > > to run one BPF program (not entire array of them), we do know whether
> > > > it's going to be sleepable or not. So we can avoid unnecessary
> > > > rcu_read_{lock,unlock}_trace() calls. rcu_read_lock_trace() is used
> > > > when there is going to be sleepable BPF program executed to protect
> > > > BPF maps and other resources from being freed too soon. But if we know
> > > > that we don't need sleepable, we can avoid that.
> > >
> > > We can add more checks and bool flags to avoid rcu_read_{lock,unlock}_trace(),
> > > but it will likely be slower. These calls are very fast.
> >
> > that's ok then. But seeing how we do
> >
> > rcu_read_lock_trace();
> > if (!sleepable)
> >     rcu_read_lock();
> >
> > it felt like we might as well just do
> >
> > if (sleepable)
> >     rcu_read_lock_trace();
> > else
> >     rcu_read_lock();

ok

> >
> >
> > As I mentioned, in this case we have a single bpf_prog, not a
> > bpf_prog_array, so that changes things a bit.
> 
> Ahh. It's only one prog. I missed that. Above makes sense then.
> But why is it not an array? We can attach multiple uprobes to the same
> location. Anyway that can be dealt with later.

I think we could add support for this later if it's needed

jirka

Re: [PATCHv2 bpf-next 08/24] libbpf: Add elf_find_multi_func_offset function

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:39:54PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding elf_find_multi_func_offset function that looks up
> > offsets for symbols specified in syms array argument.
> >
> > Offsets are returned in allocated array with the 'cnt' size,
> > that needs to be released by the caller.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c          | 112 ++++++++++++++++++++++++++++++++
> >  tools/lib/bpf/libbpf_internal.h |   2 +
> >  2 files changed, 114 insertions(+)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index 30d9e3b69114..1c310b718961 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -11053,6 +11053,118 @@ static long elf_find_func_offset(Elf *elf, const char *binary_path, const char *
> >         return ret;
> >  }
> >
> > +struct elf_symbol_offset {
> > +       const char *name;
> > +       unsigned long offset;
> > +       int bind;
> > +       int idx;
> > +};
> > +
> > +static int cmp_func_offset(const void *_a, const void *_b)
> > +{
> > +       const struct elf_symbol_offset *a = _a;
> > +       const struct elf_symbol_offset *b = _b;
> > +
> > +       return strcmp(a->name, b->name);
> > +}
> > +
> > +static int
> > +__elf_find_multi_func_offset(Elf *elf, const char *binary_path, int cnt,
> > +                            const char **syms, unsigned long **poffsets)
> > +{
> > +       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
> > +       struct elf_symbol_offset *func_offs;
> > +       int err = 0, i, idx, cnt_done = 0;
> > +       unsigned long *offsets = NULL;
> > +
> > +       func_offs = calloc(cnt, sizeof(*func_offs));
> > +       if (!func_offs)
> > +               return -ENOMEM;
> > +
> > +       for (i = 0; i < cnt; i++) {
> > +               func_offs[i].name = syms[i];
> > +               func_offs[i].idx = i;
> > +       }
> > +
> > +       qsort(func_offs, cnt, sizeof(*func_offs), cmp_func_offset);
> > +
> > +       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
> > +               struct elf_symbol_iter iter;
> > +               struct elf_symbol *sym;
> > +
> > +               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
> 
> a bit lax handling of initialization errors here, let's be a bit more
> strict here?

right, I'll have elf_symbol_iter_new return -ENOENT in case there's no
'sh_types[i]' section type (and we want to try next section type) and
fail otherwise

> 
> > +                       continue;
> > +
> > +               while ((sym = elf_symbol_iter_next(&iter))) {
> > +                       struct elf_symbol_offset *fo, tmp = {
> > +                               .name = sym->name,
> > +                       };
> > +
> > +                       fo = bsearch(&tmp, func_offs, cnt, sizeof(*func_offs),
> > +                                    cmp_func_offset);
> > +                       if (!fo)
> > +                               continue;
> > +
> > +                       if (fo->offset > 0) {
> > +                               /* same offset, no problem */
> > +                               if (fo->offset == sym->offset)
> > +                                       continue;
> > +                               /* handle multiple matches */
> > +                               if (fo->bind != STB_WEAK && sym->bind != STB_WEAK) {
> > +                                       /* Only accept one non-weak bind. */
> > +                                       pr_warn("elf: ambiguous match for '%s', '%s' in '%s'\n",
> > +                                               sym->name, fo->name, binary_path);
> > +                                       err = -LIBBPF_ERRNO__FORMAT;
> > +                                       goto out;
> > +                               } else if (sym->bind == STB_WEAK) {
> > +                                       /* already have a non-weak bind, and
> > +                                        * this is a weak bind, so ignore.
> > +                                        */
> > +                                       continue;
> > +                               }
> > +                       }
> > +                       if (!fo->offset)
> > +                               cnt_done++;
> > +                       fo->offset = sym->offset;
> > +                       fo->bind = sym->bind;
> > +               }
> > +       }
> > +
> > +       if (cnt != cnt_done) {
> > +               err = -ENOENT;
> > +               goto out;
> > +       }
> > +       offsets = calloc(cnt, sizeof(*offsets));
> 
> you can allocate it at the very beginning and fill it out based on
> fo->idx, there is no need to store offset in elf_symbol_offset

true, will fix

> 
> > +       if (!offsets) {
> > +               err = -ENOMEM;
> > +               goto out;
> > +       }
> > +       for (i = 0; i < cnt; i++) {
> > +               idx = func_offs[i].idx;
> > +               offsets[idx] = func_offs[i].offset;
> > +       }
> > +
> > +out:
> > +       *poffsets = offsets;
> > +       free(func_offs);
> > +       return err;
> > +}
> > +
> > +int elf_find_multi_func_offset(const char *binary_path, int cnt,
> > +                              const char **syms, unsigned long **poffsets)
> > +{
> > +       struct elf_fd elf_fd = {};
> 
> do you need to initialize this struct?

I recall getting gcc warn/error, which I can't see anymore ;-)
will remove

> 
> > +       long ret = -ENOENT;
> 
> same here, you always override ret, so no need to init it?

yes

> 
> > +
> > +       ret = open_elf(binary_path, &elf_fd);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = __elf_find_multi_func_offset(elf_fd.elf, binary_path, cnt, syms, poffsets);
> 
> is there a point of having elf_find_multi_func_offset and
> __elf_find_multi_func_offset separately? can you please combine?

ok

thanks,
jirka

Re: [PATCHv2 bpf-next 09/24] libbpf: Add elf_find_pattern_func_offset function

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:39:58PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding elf_find_pattern_func_offset function that looks up
> > offsets for symbols specified by pattern argument.
> >
> > The 'pattern' argument allows wildcards (*?' supported).
> >
> > Offsets are returned in allocated array together with its
> > size and needs to be released by the caller.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c          | 78 +++++++++++++++++++++++++++++++++
> >  tools/lib/bpf/libbpf_internal.h |  3 ++
> >  2 files changed, 81 insertions(+)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index 1c310b718961..3e5c88caf5d5 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -11165,6 +11165,84 @@ int elf_find_multi_func_offset(const char *binary_path, int cnt,
> >         return ret;
> >  }
> >
> > +static int
> > +__elf_find_pattern_func_offset(Elf *elf, const char *binary_path, const char *pattern,
> > +                              const char ***pnames, unsigned long **poffsets, size_t *pcnt)
> > +{
> > +       int sh_types[2] = { SHT_DYNSYM, SHT_SYMTAB };
> > +       struct elf_symbol_offset *func_offs = NULL;
> > +       unsigned long *offsets = NULL;
> > +       const char **names = NULL;
> > +       size_t func_offs_cnt = 0;
> > +       size_t func_offs_cap = 0;
> > +       int err = 0, i;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(sh_types); i++) {
> > +               struct elf_symbol_iter iter;
> > +               struct elf_symbol *sym;
> > +
> > +               if (elf_symbol_iter_new(&iter, elf, binary_path, sh_types[i]))
> > +                       continue;
> 
> same as in the previous patch, error handling?

ok, same solution as in my previous answer

> 
> > +
> > +               while ((sym = elf_symbol_iter_next(&iter))) {
> > +                       if (!glob_match(sym->name, pattern))
> > +                               continue;
> > +
> > +                       err = libbpf_ensure_mem((void **) &func_offs, &func_offs_cap,
> > +                                               sizeof(*func_offs), func_offs_cnt + 1);
> > +                       if (err)
> > +                               goto out;
> > +
> > +                       func_offs[func_offs_cnt].offset = sym->offset;
> > +                       func_offs[func_offs_cnt].name = strdup(sym->name);
> 
> check for NULL?
> 
> and I'm actually unsure why you need to reuse elf_symbol_offset struct
> here? You just need names and offsets in two separate array, so just
> do that?..

I guess it seemed easier to do just single libbpf_ensure_mem call,
then doing two and maintain cap/cnt for both arrays.. but right,
I guess there will be good solution without the extra array

> 
> > +                       func_offs_cnt++;
> > +               }
> > +
> > +               /* If we found anything in the first symbol section,
> > +                * do not search others to avoid duplicates.
> > +                */
> > +               if (func_offs_cnt)
> > +                       break;
> > +       }
> > +
> > +       offsets = calloc(func_offs_cnt, sizeof(*offsets));
> > +       names = calloc(func_offs_cnt, sizeof(*names));
> > +       if (!offsets || !names) {
> > +               free(offsets);
> > +               free(names);
> > +               err = -ENOMEM;
> > +               goto out;
> > +       }
> > +
> > +       for (i = 0; i < func_offs_cnt; i++) {
> > +               offsets[i] = func_offs[i].offset;
> > +               names[i] = func_offs[i].name;
> 
> see above, why not fill these out right away during elf symbols iteration?

ok

> 
> > +       }
> > +
> > +       *pnames = names;
> > +       *poffsets = offsets;
> > +       *pcnt = func_offs_cnt;
> > +out:
> > +       free(func_offs);
> > +       return err;
> > +}
> > +
> > +int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
> > +                                const char ***pnames, unsigned long **poffsets,
> > +                                size_t *pcnt)
> > +{
> > +       struct elf_fd elf_fd = {};
> > +       long ret = -ENOENT;
> > +
> > +       ret = open_elf(binary_path, &elf_fd);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = __elf_find_pattern_func_offset(elf_fd.elf, binary_path, pattern, pnames, poffsets, pcnt);
> 
> I don't really like these underscored functions,
> elf_find_pattern_func_offset() already has to do goto out for clean
> ups, this close_elf() thing is just another resource to clean up
> there, I don't see much reason to separate open_elf/close_elf in this
> case

I think that happened because I originally did the 'struct elf_fd' cleanup
as last patch and only then moved it up the git log, so I did not realize
elf_find_pattern_func_offset could be single function, will change

thanks,
jirka

> 
> 
> > +       close_elf(&elf_fd);
> > +       return ret;
> > +}
> > +
> >  /* Find offset of function name in ELF object specified by path. "name" matches
> >   * symbol name or name@@LIB for library functions.
> >   */
> > diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> > index 13d5c12fbd0b..22b0834e7fe1 100644
> > --- a/tools/lib/bpf/libbpf_internal.h
> > +++ b/tools/lib/bpf/libbpf_internal.h
> > @@ -579,4 +579,7 @@ int sys_bpf_prog_load(union bpf_attr *attr, unsigned int size, int attempts);
> >
> >  int elf_find_multi_func_offset(const char *binary_path, int cnt,
> >                                const char **syms, unsigned long **poffsets);
> > +int elf_find_pattern_func_offset(const char *binary_path, const char *pattern,
> > +                                const char ***pnames, unsigned long **poffsets,
> > +                                size_t *pcnt);
> >  #endif /* __LIBBPF_LIBBPF_INTERNAL_H */
> > --
> > 2.41.0
> >

Re: [PATCHv2 bpf-next 11/24] libbpf: Add bpf_program__attach_uprobe_multi_opts function

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:40:08PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding bpf_program__attach_uprobe_multi_opts function that
> > allows to attach multiple uprobes with uprobe_multi link.
> >
> > The user can specify uprobes with direct arguments:
> >
> >   binary_path/func_pattern
> >
> > or with struct bpf_uprobe_multi_opts opts argument fields:
> >
> >   const char *path;
> >   const char **syms;
> >   const unsigned long *offsets;
> >   const unsigned long *ref_ctr_offsets;
> >
> > User can specify 3 mutually exclusive set of incputs:
> 
> typo: inputs
> 
> >
> >  1) use only binary_path/func_pattern aruments
> 
> typo: arguments

ok

> 
> >
> >  2) use only opts argument with allowed combinations of:
> >     path/offsets/ref_ctr_offsets/cookies/cnt
> >
> >  3) use binary_path with allowed combinations of:
> >     syms/offsets/ref_ctr_offsets/cookies/cnt
> >
> 
> why do we need this duplication of binary_path and opts.path? same for
> pid and opts.pid?

right, will remove opts.path|pid

> 
> > Any other usage results in error.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c   | 131 +++++++++++++++++++++++++++++++++++++++
> >  tools/lib/bpf/libbpf.h   |  31 +++++++++
> >  tools/lib/bpf/libbpf.map |   1 +
> >  3 files changed, 163 insertions(+)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index 3e5c88caf5d5..d972cea4c658 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -11402,6 +11402,137 @@ static int resolve_full_path(const char *file, char *result, size_t result_sz)
> >         return -ENOENT;
> >  }
> >
> > +struct bpf_link *
> > +bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
> 
> let's call it just bpf_program__attach_uprobe_multi()
> 
> we should have done that with bpf_program__attach_kprobe_multi_opts()
> as well. Generally, if the only variant of API is the one with opts,
> then there is no point in adding "_opts" suffix to the API name, IMO

ok

> 
> 
> > +                                     pid_t pid,
> > +                                     const char *binary_path,
> > +                                     const char *func_pattern,
> > +                                     const struct bpf_uprobe_multi_opts *opts)
> > +{
> > +       const unsigned long *ref_ctr_offsets = NULL, *offsets = NULL;
> > +       LIBBPF_OPTS(bpf_link_create_opts, lopts);
> > +       unsigned long *resolved_offsets = NULL;
> > +       const char **resolved_symbols = NULL;
> > +       int err = 0, link_fd, prog_fd;
> > +       struct bpf_link *link = NULL;
> > +       char errmsg[STRERR_BUFSIZE];
> > +       const char *path, **syms;
> > +       char full_path[PATH_MAX];
> > +       const __u64 *cookies;
> > +       bool has_pattern;
> > +       bool retprobe;
> > +       size_t cnt;
> > +
> > +       if (!OPTS_VALID(opts, bpf_uprobe_multi_opts))
> > +               return libbpf_err_ptr(-EINVAL);
> > +
> > +       path = OPTS_GET(opts, path, NULL);
> > +       syms = OPTS_GET(opts, syms, NULL);
> > +       offsets = OPTS_GET(opts, offsets, NULL);
> > +       ref_ctr_offsets = OPTS_GET(opts, ref_ctr_offsets, NULL);
> > +       cookies = OPTS_GET(opts, cookies, NULL);
> > +       cnt = OPTS_GET(opts, cnt, 0);
> > +
> > +       /*
> > +        * User can specify 3 mutually exclusive set of incputs:
> > +        *
> > +        * 1) use only binary_path/func_pattern aruments
> 
> same typos
> 
> > +        *
> > +        * 2) use only opts argument with allowed combinations of:
> > +        *    path/offsets/ref_ctr_offsets/cookies/cnt
> > +        *
> > +        * 3) use binary_path with allowed combinations of:
> > +        *    syms/offsets/ref_ctr_offsets/cookies/cnt
> > +        *
> > +        * Any other usage results in error.
> > +        */
> > +
> 
> 
> Looking at this part (sorry, I already trimmed reply before realizing
> I want to comment on this):
> 
> 
> +               err = elf_find_pattern_func_offset(binary_path, func_pattern,
> +                                                  &resolved_symbols,
> &resolved_offsets,
> +                                                  &cnt);
> +               if (err < 0)
> +                       return libbpf_err_ptr(err);
> +               offsets = resolved_offsets;
> +       } else if (syms) {
> +               err = elf_find_multi_func_offset(binary_path, cnt,
> syms, &resolved_offsets);
> 
> Few thoughts.
> 
> First, we are not using resolved_symbols returned from
> elf_find_pattern_func_offset(), do we? If not, let's drop it.

I think I used that in bpftrace with the RFC version,
where I wanted to display found symbols.. but there's
no use for it in libbpf for now.. will remove it

> 
> Second, I'm not a big fan of chosen naming. Maybe something like
> elf_resolve_{pattern,syms}_offsets?

sure

SNIP

> > +LIBBPF_API struct bpf_link *
> > +bpf_program__attach_uprobe_multi_opts(const struct bpf_program *prog,
> > +                                     pid_t pid,
> > +                                     const char *binary_path,
> > +                                     const char *func_pattern,
> > +                                     const struct bpf_uprobe_multi_opts *opts);
> > +
> >  struct bpf_ksyscall_opts {
> >         /* size of this struct, for forward/backward compatibility */
> >         size_t sz;
> > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> > index 7521a2fb7626..81558ef1bc38 100644
> > --- a/tools/lib/bpf/libbpf.map
> > +++ b/tools/lib/bpf/libbpf.map
> > @@ -390,6 +390,7 @@ LIBBPF_1.2.0 {
> >                 bpf_link_get_info_by_fd;
> >                 bpf_map_get_info_by_fd;
> >                 bpf_prog_get_info_by_fd;
> > +               bpf_program__attach_uprobe_multi_opts;
> 
> should be in 1.3.0?

yes, will fix

thanks,
jirka

Re: [PATCHv2 bpf-next 12/24] libbpf: Add support for u[ret]probe.multi[.s] program sections

[Jiri Olsa]

On Fri, Jun 23, 2023 at 01:40:13PM -0700, Andrii Nakryiko wrote:
> On Tue, Jun 20, 2023 at 1:37 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding support for several uprobe_multi program sections
> > to allow auto attach of multi_uprobe programs.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  tools/lib/bpf/libbpf.c | 40 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 40 insertions(+)
> >
> > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > index d972cea4c658..e42080258ec7 100644
> > --- a/tools/lib/bpf/libbpf.c
> > +++ b/tools/lib/bpf/libbpf.c
> > @@ -8652,6 +8652,7 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
> >  static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >  static int attach_trace(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >  static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> > +static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >  static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >  static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link);
> >
> > @@ -8667,6 +8668,10 @@ static const struct bpf_sec_def section_defs[] = {
> >         SEC_DEF("uretprobe.s+",         KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
> >         SEC_DEF("kprobe.multi+",        KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
> >         SEC_DEF("kretprobe.multi+",     KPROBE, BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
> > +       SEC_DEF("uprobe.multi+",        KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
> > +       SEC_DEF("uretprobe.multi+",     KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_NONE, attach_uprobe_multi),
> > +       SEC_DEF("uprobe.multi.s+",      KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
> > +       SEC_DEF("uretprobe.multi.s+",   KPROBE, BPF_TRACE_UPROBE_MULTI, SEC_SLEEPABLE, attach_uprobe_multi),
> >         SEC_DEF("ksyscall+",            KPROBE, 0, SEC_NONE, attach_ksyscall),
> >         SEC_DEF("kretsyscall+",         KPROBE, 0, SEC_NONE, attach_ksyscall),
> >         SEC_DEF("usdt+",                KPROBE, 0, SEC_NONE, attach_usdt),
> > @@ -10728,6 +10733,41 @@ static int attach_kprobe_multi(const struct bpf_program *prog, long cookie, stru
> >         return libbpf_get_error(*link);
> >  }
> >
> > +static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link)
> > +{
> > +       char *probe_type = NULL, *binary_path = NULL, *func_name = NULL;
> > +       LIBBPF_OPTS(bpf_uprobe_multi_opts, opts);
> > +       int n, ret = -EINVAL;
> > +
> > +       *link = NULL;
> > +
> > +       n = sscanf(prog->sec_name, "%m[^/]/%m[^:]:%ms",
> > +                  &probe_type, &binary_path, &func_name);
> > +       switch (n) {
> > +       case 1:
> > +               /* handle SEC("u[ret]probe") - format is valid, but auto-attach is impossible. */
> > +               ret = 0;
> > +               break;
> > +       case 2:
> > +               pr_warn("prog '%s': section '%s' missing ':function[+offset]' specification\n",
> 
> message copy/pasted? We don't support +offset part, and it's not
> "function", but "pattern" or something along those lines?

yes, will fix

> 
> > +                       prog->name, prog->sec_name);
> > +               break;
> > +       case 3:
> > +               opts.retprobe = strcmp(probe_type, "uretprobe.multi");
> 
> == 0

ok, thanks

jirka

> 
> 
> > +               *link = bpf_program__attach_uprobe_multi_opts(prog, -1, binary_path, func_name, &opts);
> > +               ret = libbpf_get_error(*link);
> > +               break;
> > +       default:
> > +               pr_warn("prog '%s': invalid format of section definition '%s'\n", prog->name,
> > +                       prog->sec_name);
> > +               break;
> > +       }
> > +       free(probe_type);
> > +       free(binary_path);
> > +       free(func_name);
> > +       return ret;
> > +}
> > +
> >  static void gen_uprobe_legacy_event_name(char *buf, size_t buf_sz,
> >                                          const char *binary_path, uint64_t offset)
> >  {
> > --
> > 2.41.0
> >

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

[Andrii Nakryiko]

On Sat, Jun 24, 2023 at 6:18 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Fri, Jun 23, 2023 at 01:40:20PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jun 20, 2023 at 1:38 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > >
> > > Adding uprobe-multi link detection. It will be used later in
> > > bpf_program__attach_usdt function to check and use uprobe_multi
> > > link over standard uprobe links.
> > >
> > > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > > ---
> > >  tools/lib/bpf/libbpf.c          | 29 +++++++++++++++++++++++++++++
> > >  tools/lib/bpf/libbpf_internal.h |  2 ++
> > >  2 files changed, 31 insertions(+)
> > >
> > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > > index e42080258ec7..3d570898459e 100644
> > > --- a/tools/lib/bpf/libbpf.c
> > > +++ b/tools/lib/bpf/libbpf.c
> > > @@ -4815,6 +4815,32 @@ static int probe_perf_link(void)
> > >         return link_fd < 0 && err == -EBADF;
> > >  }
> > >
> > > +static int probe_uprobe_multi_link(void)
> > > +{
> > > +       struct bpf_insn insns[] = {
> > > +               BPF_MOV64_IMM(BPF_REG_0, 0),
> > > +               BPF_EXIT_INSN(),
> > > +       };
> > > +       int prog_fd, link_fd, err;
> > > +
> > > +       prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
> > > +                               insns, ARRAY_SIZE(insns), NULL);
> >
> > I thought we needed to specify expected_attach_type (BPF_TRACE_UPROBE_MULTI)?
>
> hm it should.. I guess it worked because of the KPROBE/UPROBE
> typo you found in the patch earlier, will check
>
> >
> > > +       if (prog_fd < 0)
> > > +               return -errno;
> > > +
> > > +       /* No need to specify attach function. If the link is not supported
> > > +        * we will get -EOPNOTSUPP error before any other check is performed.
> >
> > what will actually return this -EOPNOTSUPP? I couldn't find this in
> > the code quickly, can you please point me where?
>
>         #else /* !CONFIG_UPROBES */
>         int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
>         {
>                 return -EOPNOTSUPP;
>         }

that's a new code in new kernel that doesn't have CONFIG_UPROBES. What
about old kernels? They will return -EINVAL, no? Which we will assume
means "yep, multi-uprobe BPF link is supported", which would be wrong.
Or am I missing something?

>
> >
> > > +        */
> > > +       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
> > > +       err = -errno; /* close() can clobber errno */
> > > +
> > > +       if (link_fd >= 0)
> > > +               close(link_fd);
> > > +       close(prog_fd);
> > > +
> > > +       return link_fd < 0 && err != -EOPNOTSUPP;
> > > +}
> > > +
> > >  static int probe_kern_bpf_cookie(void)
> > >  {
> > >         struct bpf_insn insns[] = {
> > > @@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
> > >         [FEAT_SYSCALL_WRAPPER] = {
> > >                 "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
> > >         },
> > > +       [FEAT_UPROBE_LINK] = {
> > > +               "BPF uprobe multi link support", probe_uprobe_multi_link,
> > > +       },
> > >  };
> > >
> > >  bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
> > > diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> > > index 22b0834e7fe1..a257eb81af25 100644
> > > --- a/tools/lib/bpf/libbpf_internal.h
> > > +++ b/tools/lib/bpf/libbpf_internal.h
> > > @@ -354,6 +354,8 @@ enum kern_feature_id {
> > >         FEAT_BTF_ENUM64,
> > >         /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
> > >         FEAT_SYSCALL_WRAPPER,
> > > +       /* BPF uprobe_multi link support */
> > > +       FEAT_UPROBE_LINK,
> >
> > UPROBE_MULTI_LINK, we might have non-multi link in the future as well
>
> ok
>
> thanks,
> jirka

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Andrii Nakryiko]

On Sat, Jun 24, 2023 at 6:19 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Fri, Jun 23, 2023 at 09:24:22AM -0700, Andrii Nakryiko wrote:
>
> SNIP
>
> > > > > +
> > > > > +       if (!uprobes || !ref_ctr_offsets || !link)
> > > > > +               goto error_free;
> > > > > +
> > > > > +       for (i = 0; i < cnt; i++) {
> > > > > +               if (uref_ctr_offsets && __get_user(ref_ctr_offset, uref_ctr_offsets + i)) {
> > > > > +                       err = -EFAULT;
> > > > > +                       goto error_free;
> > > > > +               }
> > > > > +               if (__get_user(offset, uoffsets + i)) {
> > > > > +                       err = -EFAULT;
> > > > > +                       goto error_free;
> > > > > +               }
> > > > > +
> > > > > +               uprobes[i].offset = offset;
> > > > > +               uprobes[i].link = link;
> > > > > +
> > > > > +               if (flags & BPF_F_UPROBE_MULTI_RETURN)
> > > > > +                       uprobes[i].consumer.ret_handler = uprobe_multi_link_ret_handler;
> > > > > +               else
> > > > > +                       uprobes[i].consumer.handler = uprobe_multi_link_handler;
> > > > > +
> > > > > +               ref_ctr_offsets[i] = ref_ctr_offset;
> > > > > +       }
> > > > > +
> > > > > +       link->cnt = cnt;
> > > > > +       link->uprobes = uprobes;
> > > > > +       link->path = path;
> > > > > +
> > > > > +       bpf_link_init(&link->link, BPF_LINK_TYPE_UPROBE_MULTI,
> > > > > +                     &bpf_uprobe_multi_link_lops, prog);
> > > > > +
> > > > > +       err = bpf_link_prime(&link->link, &link_primer);
> > > > > +       if (err)
> > > > > +               goto error_free;
> > > > > +
> > > > > +       for (i = 0; i < cnt; i++) {
> > > > > +               err = uprobe_register_refctr(d_real_inode(link->path.dentry),
> > > > > +                                            uprobes[i].offset, ref_ctr_offsets[i],
> > > > > +                                            &uprobes[i].consumer);
> > > > > +               if (err) {
> > > > > +                       bpf_uprobe_unregister(&path, uprobes, i);
> > > >
> > > > bpf_link_cleanup() will do this through
> > > > bpf_uprobe_multi_link_release(), no? So you are double unregistering?
> > > > Either drop cnt to zero, or just don't do this here? Latter is better,
> > > > IMO.
> > >
> > > bpf_link_cleanup path won't call release callback so we have to do that
> >
> > bpf_link_cleanup() does fput(primer->file); which eventually calls
> > release callback, no? I'd add printk and simulate failure just to be
> > sure
>
> I recall we had similar discussion for kprobe_multi link ;-)
>
> I'll double check that but I think bpf_link_cleanup calls just
> dealloc callback not release

Let's document this in comments for bpf_link_cleanup() so we don't
have to discuss this again :)

I think you are right, btw. I see that bpf_link_cleanup() sets
link->prog to NULL, and bpf_link_free() won't call
link->ops->release() if link->prog is NULL.

Tricky, I keep forgetting this. Let's explicitly explain this in a comment.

>
> jirka
>
> >
> > >
> > > I think I can add simple selftest to have this path covered
> > >
> > > thanks,
> > > jirka
>
> SNIP

Re: [PATCHv2 bpf-next 13/24] libbpf: Add uprobe multi link detection

[Jiri Olsa]

On Mon, Jun 26, 2023 at 11:21:40AM -0700, Andrii Nakryiko wrote:

SNIP

> > >
> > > > +       if (prog_fd < 0)
> > > > +               return -errno;
> > > > +
> > > > +       /* No need to specify attach function. If the link is not supported
> > > > +        * we will get -EOPNOTSUPP error before any other check is performed.
> > >
> > > what will actually return this -EOPNOTSUPP? I couldn't find this in
> > > the code quickly, can you please point me where?
> >
> >         #else /* !CONFIG_UPROBES */
> >         int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
> >         {
> >                 return -EOPNOTSUPP;
> >         }
> 
> that's a new code in new kernel that doesn't have CONFIG_UPROBES. What
> about old kernels? They will return -EINVAL, no? Which we will assume
> means "yep, multi-uprobe BPF link is supported", which would be wrong.
> Or am I missing something?

ah nope, I'm missing old kernel case.. will check and fix that

jirka

> 
> >
> > >
> > > > +        */
> > > > +       link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_UPROBE_MULTI, NULL);
> > > > +       err = -errno; /* close() can clobber errno */
> > > > +
> > > > +       if (link_fd >= 0)
> > > > +               close(link_fd);
> > > > +       close(prog_fd);
> > > > +
> > > > +       return link_fd < 0 && err != -EOPNOTSUPP;
> > > > +}
> > > > +
> > > >  static int probe_kern_bpf_cookie(void)
> > > >  {
> > > >         struct bpf_insn insns[] = {
> > > > @@ -4911,6 +4937,9 @@ static struct kern_feature_desc {
> > > >         [FEAT_SYSCALL_WRAPPER] = {
> > > >                 "Kernel using syscall wrapper", probe_kern_syscall_wrapper,
> > > >         },
> > > > +       [FEAT_UPROBE_LINK] = {
> > > > +               "BPF uprobe multi link support", probe_uprobe_multi_link,
> > > > +       },
> > > >  };
> > > >
> > > >  bool kernel_supports(const struct bpf_object *obj, enum kern_feature_id feat_id)
> > > > diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
> > > > index 22b0834e7fe1..a257eb81af25 100644
> > > > --- a/tools/lib/bpf/libbpf_internal.h
> > > > +++ b/tools/lib/bpf/libbpf_internal.h
> > > > @@ -354,6 +354,8 @@ enum kern_feature_id {
> > > >         FEAT_BTF_ENUM64,
> > > >         /* Kernel uses syscall wrapper (CONFIG_ARCH_HAS_SYSCALL_WRAPPER) */
> > > >         FEAT_SYSCALL_WRAPPER,
> > > > +       /* BPF uprobe_multi link support */
> > > > +       FEAT_UPROBE_LINK,
> > >
> > > UPROBE_MULTI_LINK, we might have non-multi link in the future as well
> >
> > ok
> >
> > thanks,
> > jirka

Re: [PATCHv2 bpf-next 01/24] bpf: Add multi uprobe link

[Jiri Olsa]

On Mon, Jun 26, 2023 at 11:27:25AM -0700, Andrii Nakryiko wrote:

SNIP

> > > > > bpf_link_cleanup() will do this through
> > > > > bpf_uprobe_multi_link_release(), no? So you are double unregistering?
> > > > > Either drop cnt to zero, or just don't do this here? Latter is better,
> > > > > IMO.
> > > >
> > > > bpf_link_cleanup path won't call release callback so we have to do that
> > >
> > > bpf_link_cleanup() does fput(primer->file); which eventually calls
> > > release callback, no? I'd add printk and simulate failure just to be
> > > sure
> >
> > I recall we had similar discussion for kprobe_multi link ;-)
> >
> > I'll double check that but I think bpf_link_cleanup calls just
> > dealloc callback not release
> 
> Let's document this in comments for bpf_link_cleanup() so we don't
> have to discuss this again :)
> 
> I think you are right, btw. I see that bpf_link_cleanup() sets
> link->prog to NULL, and bpf_link_free() won't call
> link->ops->release() if link->prog is NULL.
> 
> Tricky, I keep forgetting this. Let's explicitly explain this in a comment.

ok, will add the comment

jirka

> 
> >
> > jirka
> >
> > >
> > > >
> > > > I think I can add simple selftest to have this path covered
> > > >
> > > > thanks,
> > > > jirka
> >
> > SNIP