[Cluster-devel] [PATCH dlm-tool 0/6] dlm: debian patches and python

[Cluster-devel] [PATCH dlm-tool 0/6] dlm: debian patches and python

[Alexander Aring]

Hi,

this patch contains patches from the debian dlm package which I want to
sync to upstream. Also I resend the python bindings for libdlm again and
add an example for make some histogram latency tracing between dlm_lock()
and ast callback by using ebpf. You need the recent dlm tracepoint patches
to use those examples.

- Alex

Alexander Aring (6):
  init: Enable systemctl help dlm to show relevant man pages
  dlm_controld: man: Fix typo: specfic => specific
  stonith_helper: Fix gcc build warnings
  stonith_helper: Don't link dlm_stonith against libxml2
  python: add bindings and test example
  python: add ebpf histogram example

 dlm_controld/dlm.conf.5         |   2 +-
 fence/Makefile                  |   2 -
 fence/stonith_helper.c          |   3 +-
 init/dlm.service                |   1 +
 python/README                   |   7 +
 python/bindings/__init__.py     |   0
 python/bindings/dlm.py          | 244 ++++++++++++++++++++++++++++++++
 python/ebpf/README              |   6 +
 python/ebpf/dlmhist.py          |  78 ++++++++++
 python/tests/dlm.py             |   1 +
 python/tests/recovery_interrupt |  60 ++++++++
 11 files changed, 399 insertions(+), 5 deletions(-)
 create mode 100644 python/README
 create mode 100644 python/bindings/__init__.py
 create mode 100644 python/bindings/dlm.py
 create mode 100644 python/ebpf/README
 create mode 100755 python/ebpf/dlmhist.py
 create mode 120000 python/tests/dlm.py
 create mode 100755 python/tests/recovery_interrupt

-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 1/6] init: Enable systemctl help dlm to show relevant man pages

[Alexander Aring]

Took it from the debian patches. This patch adds entries to the systemd
unit file to reference relevant man pages.

Reported-by: Ferenc W?gner <wferi@debian.org>
---
 init/dlm.service | 1 +
 1 file changed, 1 insertion(+)

diff --git a/init/dlm.service b/init/dlm.service
index 1db59251..6def985b 100644
--- a/init/dlm.service
+++ b/init/dlm.service
@@ -2,6 +2,7 @@
 Description=dlm control daemon
 Requires=corosync.service sys-kernel-config.mount
 After=corosync.service sys-kernel-config.mount
+Documentation=man:dlm_controld man:dlm.conf man:dlm_stonith
 
 [Service]
 OOMScoreAdjust=-1000
-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 2/6] dlm_controld: man: Fix typo: specfic => specific

[Alexander Aring]

Took it from the debian patches. This patch fixes a typo in the dlm.conf
man page.

Reported-by: Ferenc W?gner <wferi@debian.org>
---
 dlm_controld/dlm.conf.5 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/dlm_controld/dlm.conf.5 b/dlm_controld/dlm.conf.5
index 0eb22635..164b5b70 100644
--- a/dlm_controld/dlm.conf.5
+++ b/dlm_controld/dlm.conf.5
@@ -413,7 +413,7 @@ pairs.
 .B mark
 The mark key can be used to set a specific mark value which is then used
 by the in-kernel DLM socket creation. This can be used to match for DLM
-specfic packets for e.g. routing.
+specific packets for e.g. routing.
 
 Example of setting a per socket value for nodeid 1 and a mark value
 of 42:
-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 3/6] stonith_helper: Fix gcc build warnings

[Alexander Aring]

Took it from the debian patches. The return value of getopt() is int not
char. This patch switches the return value which is stored in variable
c to int.

Reported-by: Valentin Vidic <vvidic@debian.org>
---
 fence/stonith_helper.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/fence/stonith_helper.c b/fence/stonith_helper.c
index b1db352a..92fe0e5d 100644
--- a/fence/stonith_helper.c
+++ b/fence/stonith_helper.c
@@ -25,8 +25,7 @@ static int get_options(int argc, char *argv[])
 	char arg[MAX_ARG_LEN];
 	char key[MAX_ARG_LEN];
 	char val[MAX_ARG_LEN];
-	char c;
-	int rv;
+	int rv, c;
 
 	if (argc > 1) {
 		while ((c = getopt(argc, argv, "n:t:")) != -1) {
-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 4/6] stonith_helper: Don't link dlm_stonith against libxml2

[Alexander Aring]

Took it from the debian patches. There is no need to link against
libxml2 for the stonith_helper tool.

Reported-by: Ferenc W?gner <wferi@debian.org>
---
 fence/Makefile | 2 --
 1 file changed, 2 deletions(-)

diff --git a/fence/Makefile b/fence/Makefile
index 2b080468..547f7ba9 100644
--- a/fence/Makefile
+++ b/fence/Makefile
@@ -18,12 +18,10 @@ CFLAGS += -D_GNU_SOURCE -O2 -ggdb \
 	-fstack-clash-protection -Wl,-z,now
 
 CFLAGS += -fPIE -DPIE
-CFLAGS += `xml2-config --cflags`
 CFLAGS += -I../include
 CFLAGS += $(shell pkg-config --cflags pacemaker-fencing)
 
 LDFLAGS += -Wl,-z,relro -Wl,-z,defs -pie
-LDFLAGS += `xml2-config --libs`
 LDFLAGS += -ldl
 
 all: $(BIN_TARGET)
-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 5/6] python: add bindings and test example

[Alexander Aring]

This patch introduces a python directory with some python examples. This
is still in some experimental state to provide some dlm testing
framework by using libdlm.
---
 python/README                   |   7 +
 python/bindings/__init__.py     |   0
 python/bindings/dlm.py          | 244 ++++++++++++++++++++++++++++++++
 python/tests/dlm.py             |   1 +
 python/tests/recovery_interrupt |  60 ++++++++
 5 files changed, 312 insertions(+)
 create mode 100644 python/README
 create mode 100644 python/bindings/__init__.py
 create mode 100644 python/bindings/dlm.py
 create mode 120000 python/tests/dlm.py
 create mode 100755 python/tests/recovery_interrupt

diff --git a/python/README b/python/README
new file mode 100644
index 00000000..b598a53d
--- /dev/null
+++ b/python/README
@@ -0,0 +1,7 @@
+This directory contains python bindings to libdlm and short written tests in python to test dlm functionality.
+NOTE: the bindings are still experimental and only used for testing now.
+
+Future work:
+	- look into pytest
+	- add ebpf examples
+	- look into MPI for barrier()
diff --git a/python/bindings/__init__.py b/python/bindings/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/python/bindings/dlm.py b/python/bindings/dlm.py
new file mode 100644
index 00000000..b77d374c
--- /dev/null
+++ b/python/bindings/dlm.py
@@ -0,0 +1,244 @@
+from enum import IntEnum, IntFlag
+import sys, os, errno, platform
+import ctypes, ctypes.util
+
+#bindings
+
+if platform.system() != "Linux":
+	sys.exit("Not supported")
+
+c_path_libdlm = ctypes.util.find_library("dlm_lt")
+if not c_path_libdlm:
+	print("Unable to find the specified library.")
+	sys.exit()
+
+try:
+	c_libdlm = ctypes.CDLL(c_path_libdlm)
+except OSError:
+	print("Unable to load the system C library")
+	sys.exit()
+
+class C_DLM_LSHANDLE(ctypes.c_void_p):
+	pass
+
+class C_DLM_LKSB(ctypes.Structure):
+	_fields_ = [('sb_status', ctypes.c_int),
+		    ('sb_lkid', ctypes.c_uint32),
+		    ('sb_flags', ctypes.c_char),
+		    ('sb_lvbptr', ctypes.c_char_p)]
+
+C_BAST_CB = ctypes.CFUNCTYPE(None, ctypes.py_object)
+
+#dlm_create_lockspace
+c_dlm_create_lockspace = c_libdlm.dlm_create_lockspace
+c_dlm_create_lockspace.argtypes = [ctypes.c_char_p, #name
+				   ctypes.c_uint, #mode
+				  ]
+c_dlm_create_lockspace.restype = C_DLM_LSHANDLE
+
+#dlm_release_lockspace
+c_dlm_release_lockspace = c_libdlm.dlm_release_lockspace
+c_dlm_release_lockspace.argtypes = [ctypes.c_char_p, #name
+				    C_DLM_LSHANDLE, #ls
+				    ctypes.c_int, #force
+				   ]
+c_dlm_release_lockspace.restype = ctypes.c_int
+
+#dlm_ls_lock_wait
+c_dlm_ls_lock_wait = c_libdlm.dlm_ls_lock_wait
+c_dlm_ls_lock_wait.argtypes = [C_DLM_LSHANDLE, #ls
+			       ctypes.c_uint32, #mode
+			       ctypes.POINTER(C_DLM_LKSB), #lksb
+			       ctypes.c_uint32, #flags
+			       ctypes.c_char_p, #name
+			       ctypes.c_uint, #namelen
+			       ctypes.c_uint32, #parent
+			       ctypes.py_object, #bastarg
+			       C_BAST_CB, #bastaddr
+			       ctypes.c_void_p, #range
+			      ]
+c_dlm_ls_lock_wait.restype = ctypes.c_int
+
+#dlm_ls_unlock_wait
+c_dlm_ls_unlock_wait = c_libdlm.dlm_ls_unlock_wait
+c_dlm_ls_unlock_wait.argtypes = [C_DLM_LSHANDLE, #ls
+				 ctypes.c_uint32, #lkid
+				 ctypes.c_uint32, #flags
+				 ctypes.POINTER(C_DLM_LKSB), #lksb
+				]
+c_dlm_ls_unlock_wait.restype = ctypes.c_int
+
+#classes
+
+class LockMode(IntEnum):
+	IV = -1
+	NL = 0
+	CR = 1
+	CW = 2
+	PR = 3
+	PW = 4
+	EX = 5
+
+class LockFlag(IntFlag):
+	NOQUEUE = 0x00000001
+	CANCEL = 0x00000002
+	CONVERT = 0x00000004
+	VALBLK = 0x00000008
+	QUECVT = 0x00000010
+	IVVALBLK = 0x00000020
+	CONVDEADLK = 0x00000040
+	PERSISTENT = 0x00000080
+	NODLCKWT = 0x00000100
+	NODLCKBLK = 0x00000200
+	EXPEDITE = 0x00000400
+	NOQUEUEBAST = 0x00000800
+	HEADQUE = 0x00001000
+	NOORDER = 0x00002000
+	ORPHAN = 0x00004000
+	ALTPR = 0x00008000
+	ALTCW = 0x00010000
+	FORCEUNLOCK = 0x00020000
+	TIMEOUT = 0x00040000
+
+class LockSBFlag(IntFlag):
+	DEMOTED = 0x01
+	VALNOTVALID = 0x02
+	ALTMODE = 0x04
+
+class DLMError(OSError):
+
+	def __init__(self, errno):
+		if not errno < 0:
+			raise ValueError()
+
+		errno = abs(errno)
+		super().__init__(errno, os.strerror(errno))
+
+DLM_LOCK_TO_STR_FORMAT = \
+"""name: {}
+last_mode: {}
+last_flags: {}
+local_locked: {}
+last_sb: status: {}, lkid: {}, flags: {}, lvb: {}"""
+
+class Lockspace:
+
+	def __init__(self, name="default", mode=0o600):
+		self.__lsname = name
+		self.__ls = c_dlm_create_lockspace(self.__lsname.encode(), mode)
+		if not self.__ls:
+			raise DLMError(-errno.ENOMEM)
+
+	def release(self, force=0):
+		if not self.__ls:
+			return
+
+		rc = c_dlm_release_lockspace(self.__lsname.encode(), self.__ls,
+					     force)
+		if rc:
+			raise DLMError(rc)
+
+		self.__ls = None
+
+	def __del__(self):
+		self.release()
+
+	def __str__(self):
+		return "Lockspace: {}".format(self.__lsname)
+
+	def get_name(self):
+		return self.__lsname
+
+	# lockspace lock factory
+	def create_lock(self, name):
+		class Lock:
+
+			#note name should be 8 byte aligned for now
+			def __init__(self, ls, c_ls, name):
+				self.__local_locked = False
+				self.__last_mode = LockMode.IV
+				self.__last_flags = LockFlag(0)
+
+				self.__lk = C_DLM_LKSB()
+				self.__lk.sb_status = 0
+				self.__lk.sb_lkid = 0
+				self.__lk.sb_flags = LockSBFlag(0)
+				self.__lk.sb_lvbptr = None
+
+				self.__ls = ls
+				self.__c_ls = c_ls
+				self.__name = name
+
+			def __del__(self):
+				if self.__local_locked:
+					self.unlock_wait()
+
+			def __str__(self):
+				sb = self.get_sb()
+				return DLM_LOCK_TO_STR_FORMAT.format(
+						self.__name,
+						str(self.__last_mode),
+						str(self.__last_flags),
+						self.__local_locked,
+						str(sb.status),
+						sb.lkid, str(sb.flags),
+						str(sb.lvb))
+
+			def get_name(self):
+				return self.__name
+
+			def get_ls(self):
+				return self.__ls
+
+			# get a copy of current sb state in high-level python
+			def get_sb(self):
+				class LockSB:
+
+					def __init__(self, status, lkid,
+						     flags, lvb):
+						self.status = status
+						self.lkid = lkid
+						self.flags = LockSBFlag(flags[0])
+						self.lvb = lvb
+
+				return LockSB(self.__lk.sb_status,
+					      self.__lk.sb_lkid,
+					      self.__lk.sb_flags,
+					      self.__lk.sb_lvbptr)
+
+			def lock_wait(self, mode=LockMode.EX,
+				      flags=LockFlag(0), bast=None,
+				      bastarg=None):
+				if bast:
+					bast = C_BAST_CB(bast)
+				else:
+					bast = ctypes.cast(None, C_BAST_CB)
+
+				rc = c_dlm_ls_lock_wait(self.__c_ls, mode,
+							ctypes.byref(self.__lk),
+							flags,
+							self.__name.encode(),
+							len(self.__name), 0,
+							bastarg, bast, None)
+				if rc:
+					raise DLMError(rc)
+
+				self.__last_mode = mode
+				self.__last_flags = flags
+				self.__local_locked = True
+
+			def unlock_wait(self, flags=0):
+				rc = c_dlm_ls_unlock_wait(self.__c_ls,
+							  self.__lk.sb_lkid,
+							  flags,
+							  ctypes.byref(self.__lk))
+				if rc:
+					raise DLMError(rc)
+
+				self.__last_flags = flags
+				self.__local_locked = False
+
+		lock = Lock(self, self.__ls, name)
+		return lock
+
+# vim: tabstop=8 softtabstop=8 shiftwidth=8 noexpandtab
diff --git a/python/tests/dlm.py b/python/tests/dlm.py
new file mode 120000
index 00000000..057a99c5
--- /dev/null
+++ b/python/tests/dlm.py
@@ -0,0 +1 @@
+../bindings/dlm.py
\ No newline@end of file
diff --git a/python/tests/recovery_interrupt b/python/tests/recovery_interrupt
new file mode 100755
index 00000000..de9d2600
--- /dev/null
+++ b/python/tests/recovery_interrupt
@@ -0,0 +1,60 @@
+#!/bin/env python3
+
+from signal import signal, SIGINT
+from dlm import Lockspace
+import argparse
+import logging
+import time
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument('-l', '--lock',
+		    action='store_true',
+		    help='do lock activity between ls start/stop')
+parser.add_argument('-w', '--wait',
+		    help='wait time for contention',
+		    type=int, default=1)
+parser.add_argument("-d", "--debug", default="info",
+		    help=("logging debug level"),
+		    choices=["debug", "info", "warning"])
+
+args = parser.parse_args()
+
+debug_levels = {"debug": logging.DEBUG, "info": logging.INFO, "warning": logging.WARNING}
+debug_level = debug_levels[args.debug.lower()]
+
+logging.basicConfig(level=debug_level, format="%(asctime)s:%(levelname)s: %(message)s")
+
+def handler(signal, frame):
+	global end
+	end = True
+
+signal(SIGINT, handler)
+end = False
+while not end:
+	ls = Lockspace()
+	lsname = ls.get_name()
+	logging.info("lockspace {} created".format(lsname))
+
+	if args.lock:
+		lock = ls.create_lock("fooobaar")
+		lockname = lock.get_name()
+		logging.info("lock {} created".format(lockname))
+
+		lock.lock_wait()
+		logging.info("lock {} lock()".format(lockname))
+
+		#contention
+		logging.info("lock {} wait for {} secs".format(lockname, args.wait))
+		time.sleep(args.wait)
+
+		lock.unlock_wait()
+		logging.info("lock {} unlock()".format(lockname))
+
+		del lock
+		logging.info("lock {} removed".format(lockname))
+
+	del ls
+	logging.info("lockspace {} removed".format(lsname))
+
+# vim: tabstop=8 softtabstop=8 shiftwidth=8 noexpandtab
-- 
2.27.0

[Cluster-devel] [PATCH dlm-tool 6/6] python: add ebpf histogram example

[Alexander Aring]

This patch adds an example how a measurement between a kernel dlm_lock()
and their ast callback can be captured to get at the end a histogram.
This is hopefully a start of more useful dlm tracing collection with use
of ebpf.
---
 python/ebpf/README     |  6 ++++
 python/ebpf/dlmhist.py | 78 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)
 create mode 100644 python/ebpf/README
 create mode 100755 python/ebpf/dlmhist.py

diff --git a/python/ebpf/README b/python/ebpf/README
new file mode 100644
index 00000000..dec59ed0
--- /dev/null
+++ b/python/ebpf/README
@@ -0,0 +1,6 @@
+In this directory are some ebpf tracing examples. It requires bcc toolchain
+(usually known as bcc package name) and the python bcc module (usually known
+as python3-bcc package name).
+
+Also the current kernel headers need to be available or activate
+CONFIG_IKHEADERS in your kernel configuration.
diff --git a/python/ebpf/dlmhist.py b/python/ebpf/dlmhist.py
new file mode 100755
index 00000000..ed1eab4f
--- /dev/null
+++ b/python/ebpf/dlmhist.py
@@ -0,0 +1,78 @@
+#!/usr/bin/python
+#
+# This example shows how to capture latency between a dlm_lock() kernel
+# call for DLM_LOCK_EX requests with flag DLM_LKF_NOQUEUE and the ast
+# response.
+#
+# You will probably see two line peaks, one in case of that the current
+# node is the lock master and another one which requires network
+# communication. There is currently no way to filter them out, so the
+# second peak is interessting to get an idea what time it takes to
+# call dlm_lock() and get a response back.
+
+from bcc import BPF
+
+import threading
+
+b = BPF(text="""
+#include <uapi/linux/ptrace.h>
+#include <uapi/linux/dlm.h>
+
+BPF_HASH(start, u32);
+BPF_HISTOGRAM(dist, u64);
+
+#define DLM_HASH(args) (args->ls_id ^ args->lkb_id)
+
+TRACEPOINT_PROBE(dlm, dlm_lock_start)
+{
+    u64 ts = bpf_ktime_get_ns();
+    u32 hash = DLM_HASH(args);
+
+    if (args->flags & DLM_LKF_NOQUEUE &&
+        args->mode == DLM_LOCK_EX)
+        start.update(&hash, &ts);
+
+    return 0;
+}
+
+TRACEPOINT_PROBE(dlm, dlm_lock_end)
+{
+    u32 hash = DLM_HASH(args);
+
+    if (args->error != 0)
+        start.delete(&hash);
+
+    return 0;
+}
+
+TRACEPOINT_PROBE(dlm, dlm_ast)
+{
+    u32 hash = DLM_HASH(args);
+    u64 *tsp, delta;
+
+    tsp = start.lookup(&hash);
+    if (tsp != 0) {
+        start.delete(&hash);
+        delta = bpf_ktime_get_ns() - *tsp;
+
+        if (args->sb_status != 0)
+            return 0;
+
+        dist.increment(bpf_log2l(delta));
+    }
+
+    return 0;
+}
+""")
+
+print("Tracing... Hit Ctrl-C anytime to end.")
+
+forever = threading.Event()
+try:
+    forever.wait()
+except KeyboardInterrupt:
+    print()
+
+print("log2 histogram")
+print("--------------")
+b["dist"].print_log2_hist("ns")
-- 
2.27.0