[PATCH v3 0/9] stapsdt provider: simple system-wide probing

[PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Alan Maguire]

This series adds wildcard support to stapsdt probes to allow
tracing system-wide; this has caveats due to the way the kernel
implements probe addition.  In essence, probes are added on a
per-inode basis (actually in the VMA associated with the inode)
so it is necessary to identify the file where probes are found.
Probes will fire for existing and new processes (the RFC incorrectly
said they will not work for existing binaries; they in fact do).

Patch 2 describes the approach; to facilitate systemwide tracing
we need to tell DTrace the name of the binary/library via either
using module absolute path (patch 1 updates parsing to support use
of a '/' in a module descriptor to faciliate this support) or an
module name; to expand this we then use [LD_LIBRARY_]PATH
to resolve the full path.  ELF reading of the file and insertion
of probes then proceeds in a similar manner to per-pid tracing.

Patches 3-8 test various aspects of systemwide probes; basic
binary support, library support, listing support and is-enabled
probes support, and use of absolute paths.

Patch 9 updates docs to describe stapsdt wildcard support.

Changes since v2:

- support absolute paths (patch 1, 2)
- add tests for absolute paths (patch 4, 7)
- document systemwide probe support for listing/using absolute
  paths (patch 9)

Changes since RFC:

- update documentation/commit messages to reflect that we also
  catch existing programs/libraries when probes are enabled
- fixup provider name for wildcard probes to be 'provider*'
  rather than using the confusing 'provider-1' since the
  latter is the concatenation of probename and pid (-1 is
  used to connote all pids)
- add test for is-enabled systemwide probes

Alan Maguire (9):
  dt_lex: support '/' in probe descriptors
  stapsdt provider: support systemwide probing
  test: add systemwide stapsdt note test
  test: add systemwide stapsdt note test using absolute path
  test: add systemwide stapsdt note test for library
  stapsdt: add test for listing systemwide probes in object
  stapsdt: add test for listing systemwide probes in absolute path
    object
  stapsdt: add systemwide test for is-enabled probes
  documentation: update stapsdt docs to describe wildcard support

 .../reference/dtrace_providers_stapsdt.md     |  54 +++++-
 libdtrace/dt_lex.l                            |   2 +-
 libdtrace/dt_pid.c                            | 177 ++++++++++++++----
 libdtrace/dt_prov_uprobe.c                    |  17 +-
 .../tst.stapsdt-notes-systemwide-abspath.r    |   2 +
 .../tst.stapsdt-notes-systemwide-abspath.sh   |  51 +++++
 .../tst.stapsdt-notes-systemwide-isenabled.r  |  13 ++
 .../tst.stapsdt-notes-systemwide-isenabled.sh | 177 ++++++++++++++++++
 .../tst.stapsdt-notes-systemwide-l-abspath.sh |  48 +++++
 .../usdt/tst.stapsdt-notes-systemwide-l.sh    |  48 +++++
 .../usdt/tst.stapsdt-notes-systemwide-lib.r   |  14 ++
 .../usdt/tst.stapsdt-notes-systemwide-lib.sh  | 142 ++++++++++++++
 ...tst.stapsdt-notes-systemwide-lv-abspath.sh |  48 +++++
 .../usdt/tst.stapsdt-notes-systemwide-lv.sh   |  48 +++++
 .../usdt/tst.stapsdt-notes-systemwide.r       |   2 +
 .../usdt/tst.stapsdt-notes-systemwide.sh      |  51 +++++
 16 files changed, 845 insertions(+), 49 deletions(-)
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide.sh

-- 
2.43.5

[PATCH v3 1/9] dt_lex: support '/' in probe descriptors

[Alan Maguire]

This will allow us to support paths in module descriptions.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 libdtrace/dt_lex.l | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libdtrace/dt_lex.l b/libdtrace/dt_lex.l
index fd70aa0a..a8268c44 100644
--- a/libdtrace/dt_lex.l
+++ b/libdtrace/dt_lex.l
@@ -48,7 +48,7 @@ static size_t dt_input(char *buf, size_t max_size);
 %s S0 S1 S2 S3 S4 S5 SIDENT
 
 RGX_AGG		"@"[a-zA-Z_][0-9a-zA-Z_]*
-RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`?*\\\[\]!]*
+RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`/?*\\\[\]!]*
 RGX_IDENT	([a-zA-Z_`][0-9a-zA-Z_`]*)|([0-9][0-9a-zA-Z_]*`[0-9a-zA-Z_`]*)
 RGX_INT		([0-9]+|0[xX][0-9A-Fa-f]+)[uU]?[lL]?[lL]?
 RGX_FP		([0-9]+("."?)[0-9]*|"."[0-9]+)((e|E)("+"|-)?[0-9]+)?[fFlL]?
-- 
2.43.5

[PATCH v3 2/9] stapsdt provider: support systemwide probing

[Alan Maguire]

For stapsdt probes we can do systemwide probing by having
the kernel insert traps into the VMAs associated with a file.

The key problem for DTrace is how to specify a file path in
a module in a provider:module:function:probe specification.

Here the approach (also used by libbpf) is to support both
absolute paths and to expand binary/library names into full
paths using [LD_LIBRARY_]PATH from a binary/library name; so
specifying

myprov*:myprog::myprobe

causes us to search /usr/bin, /usr/sbin and PATH directories
to find myprog to instrument it.  If the module contains .so
we check /usr/lib, /usr/lib64 and LD_LIBRARY_PATH.  This is
beneficial as it allows scripts to be interoperable across
distros that use different directories for locating binaries.

The other part that was needed was fixups to offsets that are
optionally provided in .stapsdt.base and the addrs[1] value.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 libdtrace/dt_pid.c         | 177 ++++++++++++++++++++++++++++---------
 libdtrace/dt_prov_uprobe.c |  17 ++--
 2 files changed, 148 insertions(+), 46 deletions(-)

diff --git a/libdtrace/dt_pid.c b/libdtrace/dt_pid.c
index 36a5883b..bd352455 100644
--- a/libdtrace/dt_pid.c
+++ b/libdtrace/dt_pid.c
@@ -7,7 +7,9 @@
 
 #include <sys/ioctl.h>
 #include <sys/types.h>
+#include <sys/stat.h>
 #include <sys/sysmacros.h>
+#include <unistd.h>
 #include <stddef.h>
 #include <assert.h>
 #include <ctype.h>
@@ -34,11 +36,13 @@
 
 #include <dt_impl.h>
 #include <dt_program.h>
+#include <dt_probe.h>
 #include <dt_provider.h>
 #include <dt_pid.h>
 #include <dt_string.h>
 
 #define SEC_STAPSDT_NOTE	".note.stapsdt"
+#define SEC_STAPSDT_BASE	".stapsdt.base"
 #define NAME_STAPSDT_NOTE	"stapsdt"
 
 /*
@@ -1267,9 +1271,10 @@ dt_stapsdt_parse(dtrace_hdl_t *dtp, dt_proc_t *dpr, dtrace_probedesc_t *pdp,
 		 unsigned long addr_start)
 {
 	size_t shstrndx, noff, doff, off, n;
+	Elf_Scn *scn = NULL, *nscn = NULL;
 	const prmap_t *pmp = NULL;
+	unsigned long base = 0;
 	char *mapfile = NULL;
-	Elf_Scn *scn = NULL;
 	Elf *elf = NULL;
 	GElf_Shdr shdr;
 	GElf_Ehdr ehdr;
@@ -1287,11 +1292,16 @@ dt_stapsdt_parse(dtrace_hdl_t *dtp, dt_proc_t *dpr, dtrace_probedesc_t *pdp,
 			     path, strerror(errno));
 		return -1;
 	}
-	mod = strrchr(path, '/');
-	if (mod)
-		mod++;
-	else
-		mod = path;
+
+	if (strlen(pdp->mod) == 0) {
+		mod = strrchr(path, '/');
+		if (mod)
+			mod++;
+		else
+			mod = path;
+	} else {
+		mod = (char *)pdp->mod;
+	}
 
 	elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);   // ELF_C_READ ?
 
@@ -1323,12 +1333,14 @@ dt_stapsdt_parse(dtrace_hdl_t *dtp, dt_proc_t *dpr, dtrace_probedesc_t *pdp,
 		secname = elf_strptr(elf, shstrndx, shdr.sh_name);
 		if (strcmp(secname, SEC_STAPSDT_NOTE) == 0 &&
 		    shdr.sh_type == SHT_NOTE)
-			break;
+			nscn = scn;
+		if (strcmp(secname, SEC_STAPSDT_BASE) == 0)
+			base = shdr.sh_addr;
 	}
 	/* No ELF notes, just bail. */
-	if (scn == NULL)
+	if (nscn == NULL)
 		goto out;
-	data = elf_getdata(scn, 0);
+	data = elf_getdata(nscn, 0);
 	for (off = 0;
 	     (off = gelf_getnote(data, off, &nhdr, &noff, &doff)) > 0;) {
 		char prvname[DTRACE_PROVNAMELEN];
@@ -1385,38 +1397,59 @@ dt_stapsdt_parse(dtrace_hdl_t *dtp, dt_proc_t *dpr, dtrace_probedesc_t *pdp,
 				psp.pps_refcntr_off = addrs[2] - phdr.p_vaddr + phdr.p_offset;
 		}
 
+		/* readjust based on optional .stapsdt.base, note base addr. */
+		if (base && addrs[1])
+			psp.pps_off += base - addrs[1];
+
 		if (!psp.pps_off)
 			continue;
 		psp.pps_nameoff = 0;
 
-		if (!pmp)
-			pmp = Paddr_to_map(dpr->dpr_proc, addr_start + addrs[0]);
-		if (!pmp) {
-			dt_dprintf("%i: cannot determine 0x%lx's mapping\n",
-				   Pgetpid(dpr->dpr_proc), psp.pps_off);
-			continue;
-		}
-		if (!mapfile)
-			mapfile = Pmap_mapfile_name(dpr->dpr_proc, pmp);
+		if (dpr) {
+			if (!pmp)
+				pmp = Paddr_to_map(dpr->dpr_proc, addr_start + addrs[0]);
+			if (!pmp) {
+				dt_dprintf("%i: cannot determine 0x%lx's mapping\n",
+						   Pgetpid(dpr->dpr_proc), psp.pps_off);
+				continue;
+			}
+			if (!mapfile)
+				mapfile = Pmap_mapfile_name(dpr->dpr_proc, pmp);
 
-		if (!mapfile) {
-			dt_pid_error(dtp, pcb, dpr, D_PROC_USDT,
-				     "Cannot get name of mapping containing probe %s for pid %d\n",
-				     psp.pps_prb, dpr->dpr_pid);
-			err = -1;
-			break;
-		}
-		psp.pps_fn = mapfile;
-		if (dt_Plookup_by_addr(dtp, dpr->dpr_pid, addr_start + addrs[0],
-				       &fun, &sym) == 0)
-			psp.pps_fun = (char *)fun;
-		else
-			psp.pps_fun = no_fun;
-		psp.pps_dev = pmp->pr_dev;
-		psp.pps_inum = pmp->pr_inum;
-		psp.pps_pid = dpr->dpr_pid;
-		psp.pps_nameoff = 0;
+			if (!mapfile) {
+				dt_pid_error(dtp, pcb, dpr, D_PROC_USDT,
+					     "Cannot get name of mapping containing probe %s for pid %d\n",
+					     psp.pps_prb, dpr->dpr_pid);
+				err = -1;
+				break;
+			}
+			psp.pps_fn = mapfile;
+			if (dt_Plookup_by_addr(dtp, dpr->dpr_pid, addr_start + addrs[0],
+					       &fun, &sym) == 0)
+				psp.pps_fun = (char *)fun;
+			else
+				psp.pps_fun = no_fun;
+			psp.pps_dev = pmp->pr_dev;
+			psp.pps_inum = pmp->pr_inum;
+			psp.pps_pid = dpr->dpr_pid;
+			psp.pps_nameoff = 0;
+		} else {
+			struct stat stats = {};
 
+			if (stat(path, &stats)) {
+				dt_pid_error(dtp, pcb, dpr, D_PROC_USDT,
+					     "failed to stat() %s", path);
+				dtrace_errmsg(dtp, dtrace_errno(dtp));
+				err = -1;
+				break;
+			}
+			psp.pps_mod = mod;
+			psp.pps_dev = stats.st_dev;
+			psp.pps_inum = stats.st_ino;
+			psp.pps_fn = path;
+			psp.pps_fun = no_fun;
+			psp.pps_pid = -1;
+		}
 		if (pvp->impl->provide_probe(dtp, &psp) < 0) {
 			dt_pid_error(dtp, pcb, dpr, D_PROC_USDT,
 				     "failed to instantiate probe %s for pid %d: %s",
@@ -1506,6 +1539,52 @@ dt_pid_create_stapsdt_probes_proc(dtrace_probedesc_t *pdp, dtrace_hdl_t *dtp,
 	fclose(fp);
 }
 
+static int expand_modpath(const char *mod, char *path, size_t pathsz)
+{
+	const char *searches[2] = {};
+	int perm, i;
+
+	if (mod[0] == '/') {
+		strlcpy(path, mod, pathsz);
+		return 0;
+	}
+	if (strstr(mod, ".so")) {
+		searches[0] = getenv("LD_LIBRARY_PATH");
+		searches[1] = "/usr/lib64:/usr/lib";
+		perm = R_OK;
+	} else {
+		searches[0] = getenv("PATH");
+		searches[1] = "/usr/bin/:/usr/sbin";
+		perm = R_OK | X_OK;
+	}
+
+	for (i = 0; i < sizeof(searches)/sizeof(const char *); i++) {
+		const char *s, *n;
+
+		if (!searches[i])
+			continue;
+
+		for (s = searches[i]; s != NULL; s = n) {
+			int len;
+
+			if (*s == ':')
+				s++;
+			n = strchr(s, ':');
+			if (n)
+				len = n - s;
+			else
+				len = strlen(s);
+			snprintf(path, pathsz, "%.*s/%s", len, s, mod);
+			/* make sure accessible */
+			if (faccessat(AT_FDCWD, path, perm, AT_EACCESS) < 0)
+				continue;
+			dt_dprintf("%s: found full path '%s'\n", mod, path);
+			return 0;
+		}
+	}
+	return -ENOENT;
+}
+
 static int
 dt_pid_create_stapsdt_probes(dtrace_probedesc_t *pdp, dtrace_hdl_t *dtp, dt_pcb_t *pcb)
 {
@@ -1522,14 +1601,25 @@ dt_pid_create_stapsdt_probes(dtrace_probedesc_t *pdp, dtrace_hdl_t *dtp, dt_pcb_
 
 	pidstr = &pdp->prv[len];
 
-	while (isdigit(*(pidstr - 1)))
-		pidstr--;
-	if (strlen(pidstr) == 0)
-		return 0;
-
 	pvp = dt_provider_lookup(dtp, "stapsdt");
 	assert(pvp != NULL);
 
+	while (isdigit(*(pidstr - 1)))
+		pidstr--;
+	if (strlen(pidstr) == 0) {
+		char m[PATH_MAX];
+
+		/* only full pid wildcards are supported. */
+		if (*(pidstr - 1) != '*')
+			return 0;
+		if (isdigit(*(pidstr - 2)))
+			return 0;
+		if (dt_probe_lookup(dtp, pdp) != NULL)
+			return 0;
+		if (expand_modpath(pdp->mod, m, sizeof(m)))
+			return 0;
+		return dt_stapsdt_parse(dtp, NULL, pdp, pcb, pvp, m, 0);
+	}
 	pid = atoll(pidstr);
 	if (pid <= 0)
 		return 0;
@@ -1612,8 +1702,13 @@ dt_pid_create_usdt_probes(dtrace_probedesc_t *pdp, dtrace_hdl_t *dtp, dt_pcb_t *
 	free(globpat);
 	globfree(&globbuf);
 
-	if (err == 0)
+	if (err == 0) {
 		err = dt_pid_create_stapsdt_probes(pdp, dtp, pcb);
+		if (err != 0) {
+			dt_dprintf("stapsdt probe creation %s:%s:%s:%s failed: %d\n",
+				   pdp->prv, pdp->mod, pdp->fun, pdp->prb, err);
+		}
+	}
 
 	/* If no errors, report success. */
 	if (err == 0)
diff --git a/libdtrace/dt_prov_uprobe.c b/libdtrace/dt_prov_uprobe.c
index e94827f2..7b41270a 100644
--- a/libdtrace/dt_prov_uprobe.c
+++ b/libdtrace/dt_prov_uprobe.c
@@ -507,7 +507,7 @@ clean_usdt_probes(dtrace_hdl_t *dtp)
 			list_probe_t		*pup = prp->prv_data;
 			dt_uprobe_t		*upp = pup->probe->prv_data;
 
-			if (Pexists(upp->pid))
+			if (upp->pid == -1 || Pexists(upp->pid))
 				continue;
 		}
 
@@ -629,7 +629,7 @@ static int add_probe_usdt(dtrace_hdl_t *dtp, dt_probe_t *prp)
 		assert(0);   // FIXME do something here
 
 	/* Even though we just enabled this, check it's still live. */
-	if (!Pexists(pid)) {
+	if (pid != -1 && !Pexists(pid)) {
 		probe_disable(dtp, prp);
 		dt_bpf_map_delete(fd, &pdp->id);
 
@@ -919,7 +919,10 @@ static int provide_probe(dtrace_hdl_t *dtp, const pid_probespec_t *psp,
 	dt_probe_t		*prp, *uprp;
 	list_probe_t		*pop, *pup;
 
-	snprintf(prv, sizeof(prv), "%s%d", psp->pps_prv, psp->pps_pid);
+	if (psp->pps_pid == -1)
+		snprintf(prv, sizeof(prv), "%s*", psp->pps_prv);
+	else
+		snprintf(prv, sizeof(prv), "%s%d", psp->pps_prv, psp->pps_pid);
 
 	pd.id = DTRACE_IDNONE;
 	pd.prv = prv;
@@ -944,6 +947,7 @@ static int provide_probe(dtrace_hdl_t *dtp, const pid_probespec_t *psp,
 		return -1;
 
 	upp = uprp->prv_data;
+	upp->pid = psp->pps_pid;
 	upp->flags |= flags;
 
 	/* Look up the overlying probe. */
@@ -1552,7 +1556,7 @@ static int uprobe_create(dtrace_hdl_t *dtp, const dt_uprobe_t *upp,
 	attr.uprobe_path = (uint64_t)upp->fn;
 	attr.probe_offset = upp->off;
 
-	return dt_perf_event_open(&attr, upp->pid, -1, -1, 0);
+	return dt_perf_event_open(&attr, upp->pid, upp->pid == -1 ? 0 : -1, -1, 0);
 }
 
 static int attach(dtrace_hdl_t *dtp, const dt_probe_t *uprp, int bpf_fd)
@@ -1563,7 +1567,10 @@ static int attach(dtrace_hdl_t *dtp, const dt_probe_t *uprp, int bpf_fd)
 	assert(upp->fn != NULL);
 
 	upp->fd = uprobe_create(dtp, upp, upp->refcntr_off);
-
+	if (upp->fd < 0) {
+		dt_dprintf("uprobe_create failed: %d\n", upp->fd);
+		return upp->fd;
+	}
 	/* attach BPF program to the probe */
 	if (ioctl(upp->fd, PERF_EVENT_IOC_SET_BPF, bpf_fd) < 0)
 		return -errno;
-- 
2.43.5

[PATCH v3 3/9] test: add systemwide stapsdt note test

[Alan Maguire]

Add test verifying that systemwide stapsdt probe firing
is detected and we retrieve arguments correctly.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../usdt/tst.stapsdt-notes-systemwide.r       |  2 +
 .../usdt/tst.stapsdt-notes-systemwide.sh      | 51 +++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide.r b/test/unittest/usdt/tst.stapsdt-notes-systemwide.r
new file mode 100644
index 00000000..48ee5472
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide.r
@@ -0,0 +1,2 @@
+test:args:2:./test:val:18
+
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide.sh
new file mode 100755
index 00000000..d4de6d2b
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing various argument forms (constant, register, deref etc).
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+export PATH=.:$PATH
+
+$dtrace -c './test arg1val' -qs /dev/stdin <<EOF
+test_prov*:test::args
+{
+	printf("%s:%s:%li:%s:%s:%li\n", probemod, probename,
+	       arg0, copyinstr(arg1), copyinstr(arg2), arg3);
+}
+
+EOF
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 4/9] test: add systemwide stapsdt note test using absolute path

[Alan Maguire]

Add test verifying that systemwide stapsdt probe firing
is detected and we retrieve arguments correctly where
the specified module path is an absolute path.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../tst.stapsdt-notes-systemwide-abspath.r    |  2 +
 .../tst.stapsdt-notes-systemwide-abspath.sh   | 51 +++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r b/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
new file mode 100644
index 00000000..48ee5472
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
@@ -0,0 +1,2 @@
+test:args:2:./test:val:18
+
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
new file mode 100755
index 00000000..b2dc61ed
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing various argument forms (constant, register, deref etc).
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+export TESTPROG="$(pwd)/test"
+
+$dtrace -c './test arg1val' -qs /dev/stdin <<EOF
+test_prov*:$TESTPROG::args
+{
+	printf("%s:%s:%li:%s:%s:%li\n", basename(probemod), probename,
+	       arg0, copyinstr(arg1), copyinstr(arg2), arg3);
+}
+
+EOF
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 5/9] test: add systemwide stapsdt note test for library

[Alan Maguire]

Add test verifying that systemwide stapsdt probe firing
is detected and we retrieve arguments correctly for
a shared library.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../usdt/tst.stapsdt-notes-systemwide-lib.r   |  14 ++
 .../usdt/tst.stapsdt-notes-systemwide-lib.sh  | 142 ++++++++++++++++++
 2 files changed, 156 insertions(+)
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
new file mode 100644
index 00000000..c85121c4
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
@@ -0,0 +1,14 @@
+libstapsdttest.so:zero
+libstapsdttest.so:one:1
+libstapsdttest.so:two:2:3
+libstapsdttest.so:three:4:5:7
+libstapsdttest.so:four:7:8:9:10
+libstapsdttest.so:five:11:12:13:14:15
+libstapsdttest.so:six:16:17:18:19:20:21
+libstapsdttest.so:seven:22:23:24:25:26:27:28
+libstapsdttest.so:eight:29:30:31:32:33:34:35:36
+libstapsdttest.so:nine:37:38:39:40:41:42:43:44:45
+libstapsdttest.so:ten:46:47:48:49:50:51:52:53:54:55
+libstapsdttest.so:eleven:56:57:58:59:60:61:62:63:64:65
+libstapsdttest.so:twelve:67:68:69:70:71:72:73:74:75:76
+
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
new file mode 100755
index 00000000..1199cfac
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
@@ -0,0 +1,142 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers all stapsdt probes fired by the STAP_PROBEn macros.
+# Arguments values are checked only for first 10 arguments because
+# there is support for arg0 ... arg9 only at this moment.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > libstapsdttest.c <<EOF
+#include <sdt_notes.h>
+
+void
+libfn(int argc, char **argv)
+{
+	if (argc == 0 && argv == 0)
+		return;
+	STAP_PROBE(test_prov, zero);
+	STAP_PROBE1(test_prov, one, argc);
+	STAP_PROBE2(test_prov, two, 2, 3);
+	STAP_PROBE3(test_prov, three, 4, 5, 7);
+	STAP_PROBE4(test_prov, four, 7, 8, 9, 10);
+	STAP_PROBE5(test_prov, five, 11, 12, 13, 14, 15);
+	STAP_PROBE6(test_prov, six, 16, 17, 18, 19, 20, 21);
+	STAP_PROBE7(test_prov, seven, 22, 23, 24, 25, 26, 27, 28);
+	STAP_PROBE8(test_prov, eight, 29, 30, 31, 32, 33, 34, 35, 36);
+	STAP_PROBE9(test_prov, nine, 37, 38, 39, 40, 41, 42, 43, 44, 45);
+	STAP_PROBE10(test_prov, ten, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55);
+	STAP_PROBE11(test_prov, eleven, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66);
+	STAP_PROBE12(test_prov, twelve, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78);
+}
+EOF
+
+cat > test.c <<EOF
+
+#include <unistd.h>
+
+extern void libfn(int argc, char **argv);
+
+int
+main(int argc, char **argv)
+{
+	libfn(0, 0);
+	sleep(10);
+	libfn(argc, argv);
+	return 0;
+}
+EOF
+
+${CC} ${CFLAGS} -c -fpic libstapsdttest.c
+${CC} -shared -o libstapsdttest.so libstapsdttest.o
+${CC} -L. ${CFLAGS} -o test test.c -lstapsdttest
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH}
+$dtrace -c './test' -qs /dev/stdin <<EOF
+test_prov*:libstapsdttest.so::zero
+{
+	printf("%s:%s\n", probemod, probename);
+}
+
+test_prov*:libstapsdttest.so::one
+{
+	printf("%s:%s:%li\n", probemod, probename, arg0);
+}
+
+test_prov*:libstapsdttest.so::two
+{
+	printf("%s:%s:%li:%li\n", probemod, probename, arg0, arg1);
+}
+
+test_prov*:libstapsdttest.so::three
+{
+	printf("%s:%s:%li:%li:%li\n", probemod, probename, arg0, arg1,
+	       arg2);
+}
+
+test_prov*:libstapsdttest.so::four
+{
+	printf("%s:%s:%li:%li:%li:%li\n", probemod, probename, arg0, arg1,
+	       arg2, arg3);
+}
+
+test_prov*:libstapsdttest.so::five
+{
+	printf("%s:%s:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4);
+}
+
+test_prov*:libstapsdttest.so::six
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5);
+}
+
+test_prov*:libstapsdttest.so::seven
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6);
+}
+
+test_prov*:libstapsdttest.so::eight
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7);
+}
+
+test_prov*:libstapsdttest.so::nine
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8);
+}
+
+test_prov*:libstapsdttest.so::ten,
+test_prov*:libstapsdttest.so::eleven,
+test_prov*:libstapsdttest.so::twelve
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9);
+}
+EOF
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 6/9] stapsdt: add test for listing systemwide probes in object

[Alan Maguire]

Need to use -lm since we need to specify module level to
identify the object.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../usdt/tst.stapsdt-notes-systemwide-l.sh    | 48 +++++++++++++++++++
 .../usdt/tst.stapsdt-notes-systemwide-lv.sh   | 48 +++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
new file mode 100755
index 00000000..fab71973
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing listing probes.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE(test_prov, zero);
+	STAP_PROBE1(test_prov, one, argc);
+	STAP_PROBE2(test_prov, two, 2, 3);
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+export PATH=.:$PATH
+
+$dtrace -c './test arg1val' -lm test_prov\*:test
+status=$?
+
+exit $status
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
new file mode 100755
index 00000000..e77eaf38
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing listing probes.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE(test_prov, zero);
+	STAP_PROBE1(test_prov, one, argc);
+	STAP_PROBE2(test_prov, two, argc, argv[0][0]);
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+export PATH=.:$PATH
+
+$dtrace -c './test arg1val' -lvm test_prov\*:test
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 7/9] stapsdt: add test for listing systemwide probes in absolute path object

[Alan Maguire]

Verify using an absolute path for listing succeeds for -l and -lv.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../tst.stapsdt-notes-systemwide-l-abspath.sh | 48 +++++++++++++++++++
 ...tst.stapsdt-notes-systemwide-lv-abspath.sh | 48 +++++++++++++++++++
 2 files changed, 96 insertions(+)
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
new file mode 100755
index 00000000..878e8502
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing listing probes.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE(test_prov, zero);
+	STAP_PROBE1(test_prov, one, argc);
+	STAP_PROBE2(test_prov, two, 2, 3);
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+TEST_PROG=$(pwd)/test
+
+$dtrace -lm test_prov\*:$TEST_PROG
+status=$?
+
+exit $status
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
new file mode 100755
index 00000000..d0ff3e84
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers stapsdt probes fired by the STAP_PROBEn macros,
+# testing listing probes.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > test.c <<EOF
+#include <sdt_notes.h>
+
+int
+main(int argc, char **argv)
+{
+	STAP_PROBE(test_prov, zero);
+	STAP_PROBE1(test_prov, one, argc);
+	STAP_PROBE2(test_prov, two, argc, argv[0][0]);
+	STAP_PROBE4(test_prov, args, argc, argv[0], argv[1] + 4, 18);
+}
+EOF
+
+${CC} ${CFLAGS} -o test test.c
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+TEST_PROG=$(pwd)/test
+
+$dtrace -lvm test_prov\*:$TEST_PROG
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 8/9] stapsdt: add systemwide test for is-enabled probes

[Alan Maguire]

Add is-enabled probes to library and ensure they fire with expected
arguments when enabled.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../tst.stapsdt-notes-systemwide-isenabled.r  |  13 ++
 .../tst.stapsdt-notes-systemwide-isenabled.sh | 177 ++++++++++++++++++
 2 files changed, 190 insertions(+)
 create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
 create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh

diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r b/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
new file mode 100644
index 00000000..d7824fb5
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
@@ -0,0 +1,13 @@
+libstapsdttest.so:zero
+libstapsdttest.so:one:1
+libstapsdttest.so:two:2:3
+libstapsdttest.so:three:4:5:7
+libstapsdttest.so:four:7:8:9:10
+libstapsdttest.so:five:11:12:13:14:15
+libstapsdttest.so:six:16:17:18:19:20:21
+libstapsdttest.so:seven:22:23:24:25:26:27:28
+libstapsdttest.so:eight:29:30:31:32:33:34:35:36
+libstapsdttest.so:nine:37:38:39:40:41:42:43:44:45
+libstapsdttest.so:eleven:56:57:58:59:60:61:62:63:64:65
+libstapsdttest.so:twelve:67:68:69:70:71:72:73:74:75:76
+
diff --git a/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh b/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
new file mode 100755
index 00000000..dca869a8
--- /dev/null
+++ b/test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
@@ -0,0 +1,177 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2026, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+# This test covers all stapsdt probes fired by the STAP_PROBEn macros.
+# Arguments values are checked only for first 10 arguments because
+# there is support for arg0 ... arg9 only at this moment.
+
+if [ $# != 1 ]; then
+	echo expected one argument: '<'dtrace-path'>'
+	exit 2
+fi
+
+dtrace=$1
+CC=/usr/bin/gcc
+CFLAGS="-I${PWD}/test/unittest/usdt"
+
+DIRNAME="$tmpdir/usdt-notes.$$.$RANDOM"
+mkdir -p $DIRNAME
+cd $DIRNAME
+
+cat > libstapsdttest.c <<EOF
+#include <stdio.h>
+#define _SDT_HAS_SEMAPHORES 1
+#include <sdt_notes.h>
+
+#define SEMA_DEFINE(name)					\
+	unsigned short test_prov_##name##_semaphore = 0
+
+#define SEMA_ENABLED(name)					\
+	test_prov_##name##_semaphore
+
+SEMA_DEFINE(zero);
+SEMA_DEFINE(one);
+SEMA_DEFINE(two);
+SEMA_DEFINE(three);
+SEMA_DEFINE(four);
+SEMA_DEFINE(five);
+SEMA_DEFINE(six);
+SEMA_DEFINE(seven);
+SEMA_DEFINE(eight);
+SEMA_DEFINE(nine);
+SEMA_DEFINE(ten);
+SEMA_DEFINE(eleven);
+SEMA_DEFINE(twelve);
+
+void
+libfn(int argc, char **argv)
+{
+	if (argc == 0 && argv == 0)
+		return;
+	if (SEMA_ENABLED(zero))
+		STAP_PROBE(test_prov, zero);
+	if (SEMA_ENABLED(one))
+		STAP_PROBE1(test_prov, one, argc);
+	if (SEMA_ENABLED(two))
+		STAP_PROBE2(test_prov, two, 2, 3);
+	if (SEMA_ENABLED(three))
+		STAP_PROBE3(test_prov, three, 4, 5, 7);
+	if (SEMA_ENABLED(four))
+		STAP_PROBE4(test_prov, four, 7, 8, 9, 10);
+	if (SEMA_ENABLED(five))
+		STAP_PROBE5(test_prov, five, 11, 12, 13, 14, 15);
+	if (SEMA_ENABLED(six))
+		STAP_PROBE6(test_prov, six, 16, 17, 18, 19, 20, 21);
+	if (SEMA_ENABLED(seven))
+		STAP_PROBE7(test_prov, seven, 22, 23, 24, 25, 26, 27, 28);
+	if (SEMA_ENABLED(eight))
+		STAP_PROBE8(test_prov, eight, 29, 30, 31, 32, 33, 34, 35, 36);
+	if (SEMA_ENABLED(nine))
+		STAP_PROBE9(test_prov, nine, 37, 38, 39, 40, 41, 42, 43, 44, 45);
+	if (SEMA_ENABLED(ten)) {
+		printf("should not see this since ten probe is not enabled!\n");
+		STAP_PROBE10(test_prov, ten, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55);
+		fflush(stdout);
+	}
+	if (SEMA_ENABLED(eleven))
+		STAP_PROBE11(test_prov, eleven, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66);
+	if (SEMA_ENABLED(twelve))
+		STAP_PROBE12(test_prov, twelve, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78);
+}
+EOF
+
+cat > test.c <<EOF
+
+#include <unistd.h>
+
+extern void libfn(int argc, char **argv);
+
+int
+main(int argc, char **argv)
+{
+	libfn(argc, argv);
+	return 0;
+}
+EOF
+
+${CC} ${CFLAGS} -c -fpic libstapsdttest.c
+${CC} -shared -o libstapsdttest.so libstapsdttest.o
+${CC} -L. ${CFLAGS} -o test test.c -lstapsdttest
+if [ $? -ne 0 ]; then
+	echo "failed to compile test.c" >& 2
+	exit 1
+fi
+
+export LD_LIBRARY_PATH=.:${LD_LIBRARY_PATH}
+$dtrace -c './test' -qs /dev/stdin <<EOF
+test_prov*:libstapsdttest.so::zero
+{
+	printf("%s:%s\n", probemod, probename);
+}
+
+test_prov*:libstapsdttest.so::one
+{
+	printf("%s:%s:%li\n", probemod, probename, arg0);
+}
+
+test_prov*:libstapsdttest.so::two
+{
+	printf("%s:%s:%li:%li\n", probemod, probename, arg0, arg1);
+}
+
+test_prov*:libstapsdttest.so::three
+{
+	printf("%s:%s:%li:%li:%li\n", probemod, probename, arg0, arg1,
+	       arg2);
+}
+
+test_prov*:libstapsdttest.so::four
+{
+	printf("%s:%s:%li:%li:%li:%li\n", probemod, probename, arg0, arg1,
+	       arg2, arg3);
+}
+
+test_prov*:libstapsdttest.so::five
+{
+	printf("%s:%s:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4);
+}
+
+test_prov*:libstapsdttest.so::six
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5);
+}
+
+test_prov*:libstapsdttest.so::seven
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6);
+}
+
+test_prov*:libstapsdttest.so::eight
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7);
+}
+
+test_prov*:libstapsdttest.so::nine
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8);
+}
+
+test_prov*:libstapsdttest.so::eleven,
+test_prov*:libstapsdttest.so::twelve
+{
+	printf("%s:%s:%li:%li:%li:%li:%li:%li:%li:%li:%li:%li\n", probemod, probename,
+	       arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, arg9);
+}
+EOF
+status=$?
+
+exit $status
-- 
2.43.5

[PATCH v3 9/9] documentation: update stapsdt docs to describe wildcard support

[Alan Maguire]

We now have a limited form of wildcard support; document it.

Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
---
 .../reference/dtrace_providers_stapsdt.md     | 54 ++++++++++++++++++-
 1 file changed, 52 insertions(+), 2 deletions(-)

diff --git a/doc/userguide/reference/dtrace_providers_stapsdt.md b/doc/userguide/reference/dtrace_providers_stapsdt.md
index cb6ce71f..b91a9d23 100644
--- a/doc/userguide/reference/dtrace_providers_stapsdt.md
+++ b/doc/userguide/reference/dtrace_providers_stapsdt.md
@@ -52,12 +52,25 @@ DTrace will pass semaphore details to the kernel for reference counting.)
 The `stapsdt` provider also supports probes created dynamically via `libstapsdt`.
 See [https://github.com/linux-usdt/libstapsdt](https://github.com/linux-usdt/libstapsdt).
 
-In your D scripts, the provider name must have the pid for your process
+In your D scripts, two forms of provider specification are supported.
+
+In the first, the provider name must have the pid for your process
 appended.  For the provider shown above, you might have `myprovider12345` for
-pid 12345.  No wildcard may be used, but a symbolic value like `$target` may.
+pid 12345.  A symbolic value like `$target` may be used also.
+
+In the second approach, a wildcard may be used, but it must be used
+in combination with the path or name of binary or library.  In the case
+where only a binary or library name is specified, DTrace will use
+PATH or LD_LIBRARY_PATH to find the absolute path and add instrumentation.
+Probes are added on a file basis, so it is vital to specify the correct
+file that contains the probe;  unlike in the per-process case we need to
+know if a probe is delivered in a library or in a program for example.
 
 In your D scripts, the `stapsdt` module name may be `a.out`, just as with
 the [`pid` provider](dtrace_providers_pid.md#dt_ref_pidprobes_prov).
+However as noted above, for a wildcarded pid a filename discoverable
+via PATH or LD_LIBRARY_PATH must be specified so that DTrace can find
+the file to instrument.
 
 In probe names, two consecutive underscores \(`__`\) become a dash \(`-`\)
 just as with USDT probe names.  For example, if a probe is defined in source
@@ -147,6 +160,13 @@ Displaying notes found in: .note.stapsdt
     Arguments: -4@-4(%rbp) 8@%rax -4@$18
 ```
 
+Alternatively we can list the probes in the object via DTrace
+provided we know the provider name:
+
+```
+$ dtrace -lm 'myprovider*:/path2/a.out'
+```
+
 Notice that the instrumented source code includes the header file
 `<sys/sdt.h>`.  On Oracle Linux, this file is installed
 by the RPM package `systemtap-sdt-devel`, but the package also
@@ -154,6 +174,36 @@ installs `/usr/bin/dtrace`.  So be sure to have `/usr/sbin`
 in front of `/usr/bin` in your path, or explicitly specify
 `/usr/sbin/dtrace` whenever you want to use `dtrace`.
 
+We can also specify systemwide probes for tracing; for example when
+Python 3 is built `--with-dtrace` we can do the following to show the top 5
+function-entry calls system-wide:
+
+```
+$ sudo dtrace -qn 'python*:libpython3.6m.so::function-entry {
+	@c[execname,stringof(arg0),stringof(arg1)] = count();
+}
+END {
+	trunc(@c, 5);
+	printf("%10s %35s %20s %10s\n", "PROG", "FILE", "FUNC", "COUNT");
+	printa("%10s %35s %20s %@10d\n", @c);
+}'
+^C
+      PROG                                FILE                 FUNC      COUNT
+     tuned   /usr/lib64/python3.6/threading.py             __exit__          5
+     tuned   /usr/lib64/python3.6/threading.py     _acquire_restore          5
+     tuned   /usr/lib64/python3.6/threading.py            _is_owned          5
+     tuned   /usr/lib64/python3.6/threading.py        _release_save          5
+     tuned   /usr/lib64/python3.6/threading.py                 wait         10
+```
+
+So we see the top 5 functions called were all called by `tuned`.
+
+We specified `libpython.3.6m.so` because that is where the probes are
+defined; in some cases they are in the python3 program itself, and cavets
+mentioned above about identifying whether probes are in a library or a
+program (in this case libypthon3.6.m.so or python3.6) apply.  To determine
+probe location we can use `dtrace -lm` as described above.
+
 ## stapsdt Stability <a id="dt_ref_stapsdtstab_prov">
 
 The `stapsdt` provider uses DTrace's stability mechanism to describe its stabilities.
-- 
2.43.5

Re: [PATCH v3 1/9] dt_lex: support '/' in probe descriptors

[Kris Van Hees]

On Tue, Jan 13, 2026 at 04:51:24PM +0000, Alan Maguire wrote:
> This will allow us to support paths in module descriptions.
> 
> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
> ---
>  libdtrace/dt_lex.l | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libdtrace/dt_lex.l b/libdtrace/dt_lex.l
> index fd70aa0a..a8268c44 100644
> --- a/libdtrace/dt_lex.l
> +++ b/libdtrace/dt_lex.l
> @@ -48,7 +48,7 @@ static size_t dt_input(char *buf, size_t max_size);
>  %s S0 S1 S2 S3 S4 S5 SIDENT
>  
>  RGX_AGG		"@"[a-zA-Z_][0-9a-zA-Z_]*
> -RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`?*\\\[\]!]*
> +RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`/?*\\\[\]!]*

This change makes it that / is allowed anywhere in a probe specification,
except as first character.  That is a much wider change than what you propose
this patch to provide.

It also does not accomplish what you want, because if a probe specification
does not specify the provider name, a module name starting with / would be
reported as a syntax error.

Instead of introducing a change here, would it suffice to have the specified
library name be resolved by means of the /proc/$PID/maps file or something like
that, i.e. look at what the absolute path is of the library that matches the
provided module name?  That avoids needing to change the probe specification
syntax while also avoiding the complexity of not knowing what the library
search path is for the task being traced.

>  RGX_IDENT	([a-zA-Z_`][0-9a-zA-Z_`]*)|([0-9][0-9a-zA-Z_]*`[0-9a-zA-Z_`]*)
>  RGX_INT		([0-9]+|0[xX][0-9A-Fa-f]+)[uU]?[lL]?[lL]?
>  RGX_FP		([0-9]+("."?)[0-9]*|"."[0-9]+)((e|E)("+"|-)?[0-9]+)?[fFlL]?
> -- 
> 2.43.5
> 

Re: [PATCH v3 1/9] dt_lex: support '/' in probe descriptors

[Alan Maguire]

On 15/01/2026 22:48, Kris Van Hees wrote:
> On Tue, Jan 13, 2026 at 04:51:24PM +0000, Alan Maguire wrote:
>> This will allow us to support paths in module descriptions.
>>
>> Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
>> ---
>>  libdtrace/dt_lex.l | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/libdtrace/dt_lex.l b/libdtrace/dt_lex.l
>> index fd70aa0a..a8268c44 100644
>> --- a/libdtrace/dt_lex.l
>> +++ b/libdtrace/dt_lex.l
>> @@ -48,7 +48,7 @@ static size_t dt_input(char *buf, size_t max_size);
>>  %s S0 S1 S2 S3 S4 S5 SIDENT
>>  
>>  RGX_AGG		"@"[a-zA-Z_][0-9a-zA-Z_]*
>> -RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`?*\\\[\]!]*
>> +RGX_PSPEC	[-$:a-zA-Z_.?*\\\[\]!][-$:0-9a-zA-Z_.`/?*\\\[\]!]*
> 
> This change makes it that / is allowed anywhere in a probe specification,
> except as first character.  That is a much wider change than what you propose
> this patch to provide.
> 
> It also does not accomplish what you want, because if a probe specification
> does not specify the provider name, a module name starting with / would be
> reported as a syntax error.
> 
> Instead of introducing a change here, would it suffice to have the specified
> library name be resolved by means of the /proc/$PID/maps file or something like
> that, i.e. look at what the absolute path is of the library that matches the
> provided module name?  That avoids needing to change the probe specification
> syntax while also avoiding the complexity of not knowing what the library
> search path is for the task being traced.
> 

I'm not sure how that would work to be honest. We sometimes want to trace
stapsdt probes before the library or binary is running, and then there's no pid
to work with. And if there's a pid to work with, it raises the question of
why we need systemwide probing at all. In practice the [LD_LIBRARY_]PATH
resolution approach works well, and provides a very flexible means of
tracing userspace, even if we want to avoid full path specification
in modules. So if you'd prefer, we can drop the absolute path support, but
I think the automatic path resolution needs to avoid using anything that
relies on having a process running since it needs to handle both running and
not currently running programs to be effective.

>>  RGX_IDENT	([a-zA-Z_`][0-9a-zA-Z_`]*)|([0-9][0-9a-zA-Z_]*`[0-9a-zA-Z_`]*)
>>  RGX_INT		([0-9]+|0[xX][0-9A-Fa-f]+)[uU]?[lL]?[lL]?
>>  RGX_FP		([0-9]+("."?)[0-9]*|"."[0-9]+)((e|E)("+"|-)?[0-9]+)?[fFlL]?
>> -- 
>> 2.43.5
>>

Re: [PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Kris Van Hees]

I think I am missing something...  IS it your intent that this patch provides
for full system-wide tracing with wildcard support, i.e. where you can use a
probe specification with wildcards that is not satisfied yet by any probes in
userspace processes, and as processes are started, dtrace would attach to the
probes and start tracing them?

I do not see how this patch series accomplishes that.  If anything, the tests
are all using direct invocation of a single program being traced, so the probes
are guaranteed to be available (discoverable) at the time of D script
compilation.  I do not see any test that ensures that a 2nd process with probes
that matches the probe specification would also be traced correctly.  There is
also no test that demonstrates that starting dtrace with -Z (allowing probe
specifications that cannot be satisfied yet) and then starting a process with
matching stap probes in it results in being able to trace those probes.

So, I do think I am misunderstanding what this patch series is aiming to
accomplish, which is probably the reason I am having a heard time reviewing it.

ALl the tests in this series, as far as I can see, do not need wildcards at all
because you can simply write prov$target:... and it will trace the correct
process.

Can you elaborate on what this is intended to provide?  Or if it is meant to
provide system-wide tracing with wildcard support, can you add tests that do
exercise that functionality and show it works?

On Tue, Jan 13, 2026 at 04:51:23PM +0000, Alan Maguire wrote:
> This series adds wildcard support to stapsdt probes to allow
> tracing system-wide; this has caveats due to the way the kernel
> implements probe addition.  In essence, probes are added on a
> per-inode basis (actually in the VMA associated with the inode)
> so it is necessary to identify the file where probes are found.
> Probes will fire for existing and new processes (the RFC incorrectly
> said they will not work for existing binaries; they in fact do).
> 
> Patch 2 describes the approach; to facilitate systemwide tracing
> we need to tell DTrace the name of the binary/library via either
> using module absolute path (patch 1 updates parsing to support use
> of a '/' in a module descriptor to faciliate this support) or an
> module name; to expand this we then use [LD_LIBRARY_]PATH
> to resolve the full path.  ELF reading of the file and insertion
> of probes then proceeds in a similar manner to per-pid tracing.
> 
> Patches 3-8 test various aspects of systemwide probes; basic
> binary support, library support, listing support and is-enabled
> probes support, and use of absolute paths.
> 
> Patch 9 updates docs to describe stapsdt wildcard support.
> 
> Changes since v2:
> 
> - support absolute paths (patch 1, 2)
> - add tests for absolute paths (patch 4, 7)
> - document systemwide probe support for listing/using absolute
>   paths (patch 9)
> 
> Changes since RFC:
> 
> - update documentation/commit messages to reflect that we also
>   catch existing programs/libraries when probes are enabled
> - fixup provider name for wildcard probes to be 'provider*'
>   rather than using the confusing 'provider-1' since the
>   latter is the concatenation of probename and pid (-1 is
>   used to connote all pids)
> - add test for is-enabled systemwide probes
> 
> Alan Maguire (9):
>   dt_lex: support '/' in probe descriptors
>   stapsdt provider: support systemwide probing
>   test: add systemwide stapsdt note test
>   test: add systemwide stapsdt note test using absolute path
>   test: add systemwide stapsdt note test for library
>   stapsdt: add test for listing systemwide probes in object
>   stapsdt: add test for listing systemwide probes in absolute path
>     object
>   stapsdt: add systemwide test for is-enabled probes
>   documentation: update stapsdt docs to describe wildcard support
> 
>  .../reference/dtrace_providers_stapsdt.md     |  54 +++++-
>  libdtrace/dt_lex.l                            |   2 +-
>  libdtrace/dt_pid.c                            | 177 ++++++++++++++----
>  libdtrace/dt_prov_uprobe.c                    |  17 +-
>  .../tst.stapsdt-notes-systemwide-abspath.r    |   2 +
>  .../tst.stapsdt-notes-systemwide-abspath.sh   |  51 +++++
>  .../tst.stapsdt-notes-systemwide-isenabled.r  |  13 ++
>  .../tst.stapsdt-notes-systemwide-isenabled.sh | 177 ++++++++++++++++++
>  .../tst.stapsdt-notes-systemwide-l-abspath.sh |  48 +++++
>  .../usdt/tst.stapsdt-notes-systemwide-l.sh    |  48 +++++
>  .../usdt/tst.stapsdt-notes-systemwide-lib.r   |  14 ++
>  .../usdt/tst.stapsdt-notes-systemwide-lib.sh  | 142 ++++++++++++++
>  ...tst.stapsdt-notes-systemwide-lv-abspath.sh |  48 +++++
>  .../usdt/tst.stapsdt-notes-systemwide-lv.sh   |  48 +++++
>  .../usdt/tst.stapsdt-notes-systemwide.r       |   2 +
>  .../usdt/tst.stapsdt-notes-systemwide.sh      |  51 +++++
>  16 files changed, 845 insertions(+), 49 deletions(-)
>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide.r
>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide.sh
> 
> -- 
> 2.43.5
> 

Re: [PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Alan Maguire]

On 16/01/2026 14:48, Kris Van Hees wrote:
> I think I am missing something...  IS it your intent that this patch provides
> for full system-wide tracing with wildcard support, i.e. where you can use a
> probe specification with wildcards that is not satisfied yet by any probes in
> userspace processes, and as processes are started, dtrace would attach to the
> probes and start tracing them?
>

Yep; this works, along with tracing of existing processes for the traced
object. See the documentation patch for an example of doing
this with python. The example will trace python probes in python processes
that were started either before or after tracing has begun.
 
> I do not see how this patch series accomplishes that.  If anything, the tests
> are all using direct invocation of a single program being traced, so the probes
> are guaranteed to be available (discoverable) at the time of D script
> compilation.  I do not see any test that ensures that a 2nd process with probes
> that matches the probe specification would also be traced correctly.  There is
> also no test that demonstrates that starting dtrace with -Z (allowing probe
> specifications that cannot be satisfied yet) and then starting a process with
> matching stap probes in it results in being able to trace those probes.
> 

That's more an issue with the tests than the functionality itself. I can
expand the tests to more fully demonstrate this. It's just easier in
practice to invoke the associated command with 'dtrace -c'.

> So, I do think I am misunderstanding what this patch series is aiming to
> accomplish, which is probably the reason I am having a heard time reviewing it.
> 
> ALl the tests in this series, as far as I can see, do not need wildcards at all
> because you can simply write prov$target:... and it will trace the correct
> process.
> 
> Can you elaborate on what this is intended to provide?  Or if it is meant to
> provide system-wide tracing with wildcard support, can you add tests that do
> exercise that functionality and show it works?
>

The tests demonstrate the case where we start system-wide tracing
before starting the program we want to trace. In that case 
systemwide tracing adds the value that we don't need to trace
a specific process in advance; any process that has the probes
will trigger firing.  For example, a user wants that functionality to
trace probes in future QEMU processes regardless of pid, so that's 
definitely something folks want.

The documentation patch demonstrates the case where we want to trace
an already running program/library; in that case I started tracing
python3 probes and captured (already running) tuned-related events.

Both are valuable I think, and the combination of both is too; i.e.
regardless of when a process started (whether prior to starting
tracing or afterwards) I want to see events.

 
> On Tue, Jan 13, 2026 at 04:51:23PM +0000, Alan Maguire wrote:
>> This series adds wildcard support to stapsdt probes to allow
>> tracing system-wide; this has caveats due to the way the kernel
>> implements probe addition.  In essence, probes are added on a
>> per-inode basis (actually in the VMA associated with the inode)
>> so it is necessary to identify the file where probes are found.
>> Probes will fire for existing and new processes (the RFC incorrectly
>> said they will not work for existing binaries; they in fact do).
>>
>> Patch 2 describes the approach; to facilitate systemwide tracing
>> we need to tell DTrace the name of the binary/library via either
>> using module absolute path (patch 1 updates parsing to support use
>> of a '/' in a module descriptor to faciliate this support) or an
>> module name; to expand this we then use [LD_LIBRARY_]PATH
>> to resolve the full path.  ELF reading of the file and insertion
>> of probes then proceeds in a similar manner to per-pid tracing.
>>
>> Patches 3-8 test various aspects of systemwide probes; basic
>> binary support, library support, listing support and is-enabled
>> probes support, and use of absolute paths.
>>
>> Patch 9 updates docs to describe stapsdt wildcard support.
>>
>> Changes since v2:
>>
>> - support absolute paths (patch 1, 2)
>> - add tests for absolute paths (patch 4, 7)
>> - document systemwide probe support for listing/using absolute
>>   paths (patch 9)
>>
>> Changes since RFC:
>>
>> - update documentation/commit messages to reflect that we also
>>   catch existing programs/libraries when probes are enabled
>> - fixup provider name for wildcard probes to be 'provider*'
>>   rather than using the confusing 'provider-1' since the
>>   latter is the concatenation of probename and pid (-1 is
>>   used to connote all pids)
>> - add test for is-enabled systemwide probes
>>
>> Alan Maguire (9):
>>   dt_lex: support '/' in probe descriptors
>>   stapsdt provider: support systemwide probing
>>   test: add systemwide stapsdt note test
>>   test: add systemwide stapsdt note test using absolute path
>>   test: add systemwide stapsdt note test for library
>>   stapsdt: add test for listing systemwide probes in object
>>   stapsdt: add test for listing systemwide probes in absolute path
>>     object
>>   stapsdt: add systemwide test for is-enabled probes
>>   documentation: update stapsdt docs to describe wildcard support
>>
>>  .../reference/dtrace_providers_stapsdt.md     |  54 +++++-
>>  libdtrace/dt_lex.l                            |   2 +-
>>  libdtrace/dt_pid.c                            | 177 ++++++++++++++----
>>  libdtrace/dt_prov_uprobe.c                    |  17 +-
>>  .../tst.stapsdt-notes-systemwide-abspath.r    |   2 +
>>  .../tst.stapsdt-notes-systemwide-abspath.sh   |  51 +++++
>>  .../tst.stapsdt-notes-systemwide-isenabled.r  |  13 ++
>>  .../tst.stapsdt-notes-systemwide-isenabled.sh | 177 ++++++++++++++++++
>>  .../tst.stapsdt-notes-systemwide-l-abspath.sh |  48 +++++
>>  .../usdt/tst.stapsdt-notes-systemwide-l.sh    |  48 +++++
>>  .../usdt/tst.stapsdt-notes-systemwide-lib.r   |  14 ++
>>  .../usdt/tst.stapsdt-notes-systemwide-lib.sh  | 142 ++++++++++++++
>>  ...tst.stapsdt-notes-systemwide-lv-abspath.sh |  48 +++++
>>  .../usdt/tst.stapsdt-notes-systemwide-lv.sh   |  48 +++++
>>  .../usdt/tst.stapsdt-notes-systemwide.r       |   2 +
>>  .../usdt/tst.stapsdt-notes-systemwide.sh      |  51 +++++
>>  16 files changed, 845 insertions(+), 49 deletions(-)
>>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
>>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
>>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
>>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide.r
>>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide.sh
>>
>> -- 
>> 2.43.5
>>

Re: [PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Kris Van Hees]

On Fri, Jan 16, 2026 at 03:10:52PM +0000, Alan Maguire wrote:
> On 16/01/2026 14:48, Kris Van Hees wrote:
> > I think I am missing something...  IS it your intent that this patch provides
> > for full system-wide tracing with wildcard support, i.e. where you can use a
> > probe specification with wildcards that is not satisfied yet by any probes in
> > userspace processes, and as processes are started, dtrace would attach to the
> > probes and start tracing them?
> >
> 
> Yep; this works, along with tracing of existing processes for the traced
> object. See the documentation patch for an example of doing
> this with python. The example will trace python probes in python processes
> that were started either before or after tracing has begun.
>  
> > I do not see how this patch series accomplishes that.  If anything, the tests
> > are all using direct invocation of a single program being traced, so the probes
> > are guaranteed to be available (discoverable) at the time of D script
> > compilation.  I do not see any test that ensures that a 2nd process with probes
> > that matches the probe specification would also be traced correctly.  There is
> > also no test that demonstrates that starting dtrace with -Z (allowing probe
> > specifications that cannot be satisfied yet) and then starting a process with
> > matching stap probes in it results in being able to trace those probes.
> > 
> 
> That's more an issue with the tests than the functionality itself. I can
> expand the tests to more fully demonstrate this. It's just easier in
> practice to invoke the associated command with 'dtrace -c'.

OK, but without tests that actually exercise the functionality, there is no
good way to verify that the implementation does the right thing, and that it
will keep doing that (i.e. detect any regressions).  Without tests that
exercise the functionality, we cannot review and merge the patch series.

> > So, I do think I am misunderstanding what this patch series is aiming to
> > accomplish, which is probably the reason I am having a heard time reviewing it.
> > 
> > ALl the tests in this series, as far as I can see, do not need wildcards at all
> > because you can simply write prov$target:... and it will trace the correct
> > process.
> > 
> > Can you elaborate on what this is intended to provide?  Or if it is meant to
> > provide system-wide tracing with wildcard support, can you add tests that do
> > exercise that functionality and show it works?
> >
> 
> The tests demonstrate the case where we start system-wide tracing
> before starting the program we want to trace. In that case 
> systemwide tracing adds the value that we don't need to trace
> a specific process in advance; any process that has the probes
> will trigger firing.  For example, a user wants that functionality to
> trace probes in future QEMU processes regardless of pid, so that's 
> definitely something folks want.

Actually, that is not true.  Since the tests in the series all use dtrace -c,
the process is started prior to resolving the probe specifications as part of
the D script compilation.  So, they do not exercise the case of starting
tracing prior to starting the program at all.

> The documentation patch demonstrates the case where we want to trace
> an already running program/library; in that case I started tracing
> python3 probes and captured (already running) tuned-related events.
> 
> Both are valuable I think, and the combination of both is too; i.e.
> regardless of when a process started (whether prior to starting
> tracing or afterwards) I want to see events.

Oh, I certainly agree that this is the functionality we want.  I am just saying
that the tests in the series do not in any way exercise that functionality, so
there is no testsuite verification that any of this actually works and that if
a regression were to occur, that there would be tests that show that.

So, we definitely need tests in this series that show:

1. show it works for executables and libraries with -c (you already have those)
2. show it works for executables and libraries with a program that was started
   before dtrace
3. show it works for executables and libraries with a program that was started
   before dtrace AND for a second program that was started after dtrace
4. show it works for executables and libraries with two or more programs that
   get started after dtrace

(I think those are all the cases one would expect to work)

> > On Tue, Jan 13, 2026 at 04:51:23PM +0000, Alan Maguire wrote:
> >> This series adds wildcard support to stapsdt probes to allow
> >> tracing system-wide; this has caveats due to the way the kernel
> >> implements probe addition.  In essence, probes are added on a
> >> per-inode basis (actually in the VMA associated with the inode)
> >> so it is necessary to identify the file where probes are found.
> >> Probes will fire for existing and new processes (the RFC incorrectly
> >> said they will not work for existing binaries; they in fact do).
> >>
> >> Patch 2 describes the approach; to facilitate systemwide tracing
> >> we need to tell DTrace the name of the binary/library via either
> >> using module absolute path (patch 1 updates parsing to support use
> >> of a '/' in a module descriptor to faciliate this support) or an
> >> module name; to expand this we then use [LD_LIBRARY_]PATH
> >> to resolve the full path.  ELF reading of the file and insertion
> >> of probes then proceeds in a similar manner to per-pid tracing.
> >>
> >> Patches 3-8 test various aspects of systemwide probes; basic
> >> binary support, library support, listing support and is-enabled
> >> probes support, and use of absolute paths.
> >>
> >> Patch 9 updates docs to describe stapsdt wildcard support.
> >>
> >> Changes since v2:
> >>
> >> - support absolute paths (patch 1, 2)
> >> - add tests for absolute paths (patch 4, 7)
> >> - document systemwide probe support for listing/using absolute
> >>   paths (patch 9)
> >>
> >> Changes since RFC:
> >>
> >> - update documentation/commit messages to reflect that we also
> >>   catch existing programs/libraries when probes are enabled
> >> - fixup provider name for wildcard probes to be 'provider*'
> >>   rather than using the confusing 'provider-1' since the
> >>   latter is the concatenation of probename and pid (-1 is
> >>   used to connote all pids)
> >> - add test for is-enabled systemwide probes
> >>
> >> Alan Maguire (9):
> >>   dt_lex: support '/' in probe descriptors
> >>   stapsdt provider: support systemwide probing
> >>   test: add systemwide stapsdt note test
> >>   test: add systemwide stapsdt note test using absolute path
> >>   test: add systemwide stapsdt note test for library
> >>   stapsdt: add test for listing systemwide probes in object
> >>   stapsdt: add test for listing systemwide probes in absolute path
> >>     object
> >>   stapsdt: add systemwide test for is-enabled probes
> >>   documentation: update stapsdt docs to describe wildcard support
> >>
> >>  .../reference/dtrace_providers_stapsdt.md     |  54 +++++-
> >>  libdtrace/dt_lex.l                            |   2 +-
> >>  libdtrace/dt_pid.c                            | 177 ++++++++++++++----
> >>  libdtrace/dt_prov_uprobe.c                    |  17 +-
> >>  .../tst.stapsdt-notes-systemwide-abspath.r    |   2 +
> >>  .../tst.stapsdt-notes-systemwide-abspath.sh   |  51 +++++
> >>  .../tst.stapsdt-notes-systemwide-isenabled.r  |  13 ++
> >>  .../tst.stapsdt-notes-systemwide-isenabled.sh | 177 ++++++++++++++++++
> >>  .../tst.stapsdt-notes-systemwide-l-abspath.sh |  48 +++++
> >>  .../usdt/tst.stapsdt-notes-systemwide-l.sh    |  48 +++++
> >>  .../usdt/tst.stapsdt-notes-systemwide-lib.r   |  14 ++
> >>  .../usdt/tst.stapsdt-notes-systemwide-lib.sh  | 142 ++++++++++++++
> >>  ...tst.stapsdt-notes-systemwide-lv-abspath.sh |  48 +++++
> >>  .../usdt/tst.stapsdt-notes-systemwide-lv.sh   |  48 +++++
> >>  .../usdt/tst.stapsdt-notes-systemwide.r       |   2 +
> >>  .../usdt/tst.stapsdt-notes-systemwide.sh      |  51 +++++
> >>  16 files changed, 845 insertions(+), 49 deletions(-)
> >>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.r
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-abspath.sh
> >>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.r
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-isenabled.sh
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l-abspath.sh
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-l.sh
> >>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.r
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lib.sh
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv-abspath.sh
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide-lv.sh
> >>  create mode 100644 test/unittest/usdt/tst.stapsdt-notes-systemwide.r
> >>  create mode 100755 test/unittest/usdt/tst.stapsdt-notes-systemwide.sh
> >>
> >> -- 
> >> 2.43.5
> >>
> 

Re: [PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Alan Maguire]

On 16/01/2026 15:33, Kris Van Hees wrote:
> On Fri, Jan 16, 2026 at 03:10:52PM +0000, Alan Maguire wrote:
>> On 16/01/2026 14:48, Kris Van Hees wrote:
>>> I think I am missing something...  IS it your intent that this patch provides
>>> for full system-wide tracing with wildcard support, i.e. where you can use a
>>> probe specification with wildcards that is not satisfied yet by any probes in
>>> userspace processes, and as processes are started, dtrace would attach to the
>>> probes and start tracing them?
>>>
>>
>> Yep; this works, along with tracing of existing processes for the traced
>> object. See the documentation patch for an example of doing
>> this with python. The example will trace python probes in python processes
>> that were started either before or after tracing has begun.
>>  
>>> I do not see how this patch series accomplishes that.  If anything, the tests
>>> are all using direct invocation of a single program being traced, so the probes
>>> are guaranteed to be available (discoverable) at the time of D script
>>> compilation.  I do not see any test that ensures that a 2nd process with probes
>>> that matches the probe specification would also be traced correctly.  There is
>>> also no test that demonstrates that starting dtrace with -Z (allowing probe
>>> specifications that cannot be satisfied yet) and then starting a process with
>>> matching stap probes in it results in being able to trace those probes.
>>>
>>
>> That's more an issue with the tests than the functionality itself. I can
>> expand the tests to more fully demonstrate this. It's just easier in
>> practice to invoke the associated command with 'dtrace -c'.
> 
> OK, but without tests that actually exercise the functionality, there is no
> good way to verify that the implementation does the right thing, and that it
> will keep doing that (i.e. detect any regressions).  Without tests that
> exercise the functionality, we cannot review and merge the patch series.
> 
>>> So, I do think I am misunderstanding what this patch series is aiming to
>>> accomplish, which is probably the reason I am having a heard time reviewing it.
>>>
>>> ALl the tests in this series, as far as I can see, do not need wildcards at all
>>> because you can simply write prov$target:... and it will trace the correct
>>> process.
>>>
>>> Can you elaborate on what this is intended to provide?  Or if it is meant to
>>> provide system-wide tracing with wildcard support, can you add tests that do
>>> exercise that functionality and show it works?
>>>
>>
>> The tests demonstrate the case where we start system-wide tracing
>> before starting the program we want to trace. In that case 
>> systemwide tracing adds the value that we don't need to trace
>> a specific process in advance; any process that has the probes
>> will trigger firing.  For example, a user wants that functionality to
>> trace probes in future QEMU processes regardless of pid, so that's 
>> definitely something folks want.
> 
> Actually, that is not true.  Since the tests in the series all use dtrace -c,
> the process is started prior to resolving the probe specifications as part of
> the D script compilation.  So, they do not exercise the case of starting
> tracing prior to starting the program at all.
> 
>> The documentation patch demonstrates the case where we want to trace
>> an already running program/library; in that case I started tracing
>> python3 probes and captured (already running) tuned-related events.
>>
>> Both are valuable I think, and the combination of both is too; i.e.
>> regardless of when a process started (whether prior to starting
>> tracing or afterwards) I want to see events.
> 
> Oh, I certainly agree that this is the functionality we want.  I am just saying
> that the tests in the series do not in any way exercise that functionality, so
> there is no testsuite verification that any of this actually works and that if
> a regression were to occur, that there would be tests that show that.
> 
> So, we definitely need tests in this series that show:
> 
> 1. show it works for executables and libraries with -c (you already have those)
> 2. show it works for executables and libraries with a program that was started
>    before dtrace
> 3. show it works for executables and libraries with a program that was started
>    before dtrace AND for a second program that was started after dtrace
> 4. show it works for executables and libraries with two or more programs that
>    get started after dtrace
> 
> (I think those are all the cases one would expect to work)
>

yep, that sounds right. I'll work to add tests to cover these cases explicitly.
What do you want to do about the absolute path support - will I remove it?

Re: [PATCH v3 0/9] stapsdt provider: simple system-wide probing

[Kris Van Hees]

On Fri, Jan 16, 2026 at 03:36:01PM +0000, Alan Maguire wrote:

<< trimmed >>

> > Oh, I certainly agree that this is the functionality we want.  I am just saying
> > that the tests in the series do not in any way exercise that functionality, so
> > there is no testsuite verification that any of this actually works and that if
> > a regression were to occur, that there would be tests that show that.
> > 
> > So, we definitely need tests in this series that show:
> > 
> > 1. show it works for executables and libraries with -c (you already have those)
> > 2. show it works for executables and libraries with a program that was started
> >    before dtrace
> > 3. show it works for executables and libraries with a program that was started
> >    before dtrace AND for a second program that was started after dtrace
> > 4. show it works for executables and libraries with two or more programs that
> >    get started after dtrace
> > 
> > (I think those are all the cases one would expect to work)
> >
> 
> yep, that sounds right. I'll work to add tests to cover these cases explicitly.

Thanks.

> What do you want to do about the absolute path support - will I remove it?

Let's leave it in for now - once we have the tests, I can review this more in
detail, and we can evaluate better what the best approach may be.  If adding
support for absolute paths in the module component, then we will have to rework
more in the lexer to support that without side-effects.  If using library path
turns out to be the best way to go after all, we can revive that.  But I won't
know until we have full tests and I can do a better review and make more
informed suggestions.