[PATCH] Set lockmem limit before checking BPF helper functions

[eugene.loh@oracle.com]

From: Eugene Loh <eugene.loh@oracle.com>

In dtrace_init(), we set the locked-memory limit, either to the
user-specified value (if any) or to unlimited (by default).  We also check
to make sure that certain BPF helper functions are available, falling
over to alternatives or indicating they are not available in case of
problems.

It is possible, however, that the limit is too low when dtrace starts,
causing problems with the helper-function tests before dtrace_init()
even has a chance to reset the limit.

Switch the order to set the limit before checking the helper functions.

A test is added.  The underlying problem, however, depends on kernel
version, how locked memory is handled, the behavior of fallback
functions, and so on.  So the test could easily pass on some systems
even if the fix is not employed.

Signed-off-by: Eugene Loh <eugene.loh@oracle.com>
---
 libdtrace/dt_open.c                    | 14 +++++++-------
 test/unittest/misc/tst.lockmem-init.r  |  3 +++
 test/unittest/misc/tst.lockmem-init.sh | 20 ++++++++++++++++++++
 3 files changed, 30 insertions(+), 7 deletions(-)
 create mode 100644 test/unittest/misc/tst.lockmem-init.r
 create mode 100755 test/unittest/misc/tst.lockmem-init.sh

diff --git a/libdtrace/dt_open.c b/libdtrace/dt_open.c
index 17dfbf9a6..54adec02a 100644
--- a/libdtrace/dt_open.c
+++ b/libdtrace/dt_open.c
@@ -1213,13 +1213,6 @@ dtrace_init(dtrace_hdl_t *dtp)
 		return dt_set_errno(dtp, EDT_CTF);
 	}
 
-	/*
-	 * Initialize the BPF library handling.
-	 */
-	dt_bpf_init(dtp);
-	dt_btf_get_module_ids(dtp);
-	dt_dlib_init(dtp);
-
 	/*
 	 * Set the locked-memory limit.
 	 */
@@ -1228,6 +1221,13 @@ dtrace_init(dtrace_hdl_t *dtp)
 	rl.rlim_cur = rl.rlim_max = lockmem;
 	setrlimit(RLIMIT_MEMLOCK, &rl);
 
+	/*
+	 * Initialize the BPF library handling.
+	 */
+	dt_bpf_init(dtp);
+	dt_btf_get_module_ids(dtp);
+	dt_dlib_init(dtp);
+
 	/*
 	 * Initialize consume handling.
 	 */
diff --git a/test/unittest/misc/tst.lockmem-init.r b/test/unittest/misc/tst.lockmem-init.r
new file mode 100644
index 000000000..da8b2a5f9
--- /dev/null
+++ b/test/unittest/misc/tst.lockmem-init.r
@@ -0,0 +1,3 @@
+|Delay in ns needed in delay env|
+-- @@stderr --
+Delay in ns needed in delay env var.
diff --git a/test/unittest/misc/tst.lockmem-init.sh b/test/unittest/misc/tst.lockmem-init.sh
new file mode 100755
index 000000000..1b4dcdeea
--- /dev/null
+++ b/test/unittest/misc/tst.lockmem-init.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+# Oracle Linux DTrace.
+# Copyright (c) 2025, Oracle and/or its affiliates. All rights reserved.
+# Licensed under the Universal Permissive License v 1.0 as shown at
+# http://oss.oracle.com/licenses/upl.
+
+dtrace=$1
+
+# Check that dtrace runs by default even if ulimit -l is very low.
+ulimit -l 1
+
+$dtrace $dt_flags -c test/triggers/delaydie -qn '
+syscall::write:entry
+/pid == $target/
+{
+	printf("|%s|", copyinstr(arg1, 32));
+}'
+
+exit $?
-- 
2.47.3