[PATCH 0/6] daxctl: use dax kmem atomic whole-device hotplug

[PATCH 0/6] daxctl: use dax kmem atomic whole-device hotplug

[Gregory Price (Meta)]

Kernels with the dax_kmem daxX.Y/state attribute [1] can unplug or
plug-and-online a whole device atomically. Use it in daxctl
online-memory and offline-memory so other agents (udev, auto-online)
can't race daxctl block by block, notably the offline-then-unbind race
on teardown.

Kernels without the attribute keep the per-block path.

Behavior change: offline-memory now always unplugs the device (state
'unplugged'), removing its memory blocks, even if they were all
offline already. Previously that was a no-op.

'daxctl list' (and 'ndctl/cxl list -X') gains 'hotplug_state',
reporting daxX.Y/state. The kernel only updates that when written, so
it goes stale if blocks are toggled individually.

If there is a mismatch between 'dax/state' and the blocks, then
'hotplug_state' reports 'inconsistent' (online state / zones disagree).
That is intentional: it flags agents fighting over the device.
It only appears when the attribute exists.

Compatibility notes:
- Where the unplug fails (e.g. no memory hotremove support),
  online-memory falls back to the per-block path and offline-memory
  of an already offline device remains a no-op.
- Older daxctl/ndctl/cxl binaries don't expect a device with no memory
  blocks.  Basically multiple daxctl versions on the same box can cause
  older versions to fail when in the unplugged state.  Not a huge issue.
- If re-adding fails after an unplug (e.g. region can't be reserved)
  the device is left unplugged.
- For multi-range devices the kernel succeeds if any range is added.
  daxctl can't tell that a range is missing.

[1] https://lore.kernel.org/all/20260712154505.3564379-1-gourry@gourry.net/

Gregory Price (6):
  libdaxctl: add dax kmem atomic hotplug state accessors
  daxctl: online whole-device memory atomically when available
  daxctl: offline whole-device memory via atomic unplug when available
  daxctl: report whole-device hotplug state in 'daxctl list'
  Documentation: describe atomic whole-device hotplug in daxctl man
    pages
  test/daxctl-devices.sh: cycle offline/online to cover unplug and
    replug

 Documentation/daxctl/daxctl-list.txt          |   5 +
 .../daxctl/daxctl-offline-memory.txt          |   5 +
 Documentation/daxctl/daxctl-online-memory.txt |   5 +
 .../daxctl/daxctl-reconfigure-device.txt      |   5 +-
 daxctl/device.c                               | 156 +++++++++++++++++-
 daxctl/json.c                                 |  53 +++++-
 daxctl/lib/libdaxctl-private.h                |   9 +
 daxctl/lib/libdaxctl.c                        |  71 +++++++-
 daxctl/lib/libdaxctl.sym                      |   6 +
 daxctl/libdaxctl.h                            |  18 ++
 test/daxctl-devices.sh                        |  53 ++++++
 11 files changed, 379 insertions(+), 7 deletions(-)

-- 
2.53.0-Meta

[PATCH 1/6] libdaxctl: add dax kmem atomic hotplug state accessors

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

Recent kernels add /sys/bus/dax/devices/daxX.Y/state to dax_kmem
devices for atomic whole-device hotplug. It reads 'unplugged',
'offline', 'online', 'online_kernel' or 'online_movable', and accepts
all but 'offline'.

Add an enum and get/set accessors for it. The getter returns
DAXCTL_KMEM_UNKNOWN when the attribute is absent, so callers can fall
back to per-block operations.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 daxctl/lib/libdaxctl-private.h |  9 ++++++
 daxctl/lib/libdaxctl.c         | 59 ++++++++++++++++++++++++++++++++++
 daxctl/lib/libdaxctl.sym       |  6 ++++
 daxctl/libdaxctl.h             | 18 +++++++++++
 4 files changed, 92 insertions(+)

diff --git a/daxctl/lib/libdaxctl-private.h b/daxctl/lib/libdaxctl-private.h
index b902e3d..00e906c 100644
--- a/daxctl/lib/libdaxctl-private.h
+++ b/daxctl/lib/libdaxctl-private.h
@@ -58,6 +58,15 @@ static const char *state_strings[] = {
 	[MEM_ZONE_MOVABLE] = "online_movable",
 };
 
+/* dax_kmem daxX.Y/state strings */
+static const char *kmem_state_strings[] = {
+	[DAXCTL_KMEM_UNPLUGGED] = "unplugged",
+	[DAXCTL_KMEM_OFFLINE] = "offline",
+	[DAXCTL_KMEM_ONLINE] = "online",
+	[DAXCTL_KMEM_ONLINE_KERNEL] = "online_kernel",
+	[DAXCTL_KMEM_ONLINE_MOVABLE] = "online_movable",
+};
+
 /**
  * struct daxctl_region - container for dax_devices
  */
diff --git a/daxctl/lib/libdaxctl.c b/daxctl/lib/libdaxctl.c
index 01b1915..cbb42e3 100644
--- a/daxctl/lib/libdaxctl.c
+++ b/daxctl/lib/libdaxctl.c
@@ -1794,3 +1794,62 @@ DAXCTL_EXPORT int daxctl_dev_has_online_memory(struct daxctl_dev *dev)
 	else
 		return 0;
 }
+
+DAXCTL_EXPORT enum daxctl_kmem_state daxctl_dev_get_kmem_state(
+		struct daxctl_dev *dev)
+{
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf;
+	int len = dev->buf_len;
+	char buf[SYSFS_ATTR_SIZE];
+	int i;
+
+	if (snprintf(path, len, "%s/state", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n",
+				daxctl_dev_get_devname(dev));
+		return DAXCTL_KMEM_UNKNOWN;
+	}
+
+	/* absent on older kernels */
+	if (sysfs_read_attr(ctx, path, buf) < 0)
+		return DAXCTL_KMEM_UNKNOWN;
+
+	for (i = DAXCTL_KMEM_UNPLUGGED; i <= DAXCTL_KMEM_ONLINE_MOVABLE; i++)
+		if (strcmp(buf, kmem_state_strings[i]) == 0)
+			return i;
+
+	dbg(ctx, "%s: unrecognized kmem state '%s'\n",
+			daxctl_dev_get_devname(dev), buf);
+	return DAXCTL_KMEM_UNKNOWN;
+}
+
+DAXCTL_EXPORT int daxctl_dev_set_kmem_state(struct daxctl_dev *dev,
+		enum daxctl_kmem_state state)
+{
+	struct daxctl_ctx *ctx = daxctl_dev_get_ctx(dev);
+	char *path = dev->dev_buf;
+	int len = dev->buf_len;
+
+	switch (state) {
+	case DAXCTL_KMEM_UNPLUGGED:
+	case DAXCTL_KMEM_ONLINE:
+	case DAXCTL_KMEM_ONLINE_KERNEL:
+	case DAXCTL_KMEM_ONLINE_MOVABLE:
+		break;
+	/* 'offline' is read-only */
+	case DAXCTL_KMEM_OFFLINE:
+	case DAXCTL_KMEM_UNKNOWN:
+	default:
+		dbg(ctx, "%s: invalid kmem state %d\n",
+				daxctl_dev_get_devname(dev), state);
+		return -EINVAL;
+	}
+
+	if (snprintf(path, len, "%s/state", dev->dev_path) >= len) {
+		err(ctx, "%s: buffer too small!\n",
+				daxctl_dev_get_devname(dev));
+		return -ENXIO;
+	}
+
+	return sysfs_write_attr(ctx, path, kmem_state_strings[state]);
+}
diff --git a/daxctl/lib/libdaxctl.sym b/daxctl/lib/libdaxctl.sym
index 43dd60b..d247542 100644
--- a/daxctl/lib/libdaxctl.sym
+++ b/daxctl/lib/libdaxctl.sym
@@ -113,3 +113,9 @@ global:
 	daxctl_dev_get_mode;
 	daxctl_dev_is_system_ram_mode;
 } LIBDAXCTL_10;
+
+LIBDAXCTL_12 {
+global:
+	daxctl_dev_get_kmem_state;
+	daxctl_dev_set_kmem_state;
+} LIBDAXCTL_11;
diff --git a/daxctl/libdaxctl.h b/daxctl/libdaxctl.h
index 7ec159e..e2812cc 100644
--- a/daxctl/libdaxctl.h
+++ b/daxctl/libdaxctl.h
@@ -77,6 +77,24 @@ int daxctl_dev_get_target_node(struct daxctl_dev *dev);
 int daxctl_dev_will_auto_online_memory(struct daxctl_dev *dev);
 int daxctl_dev_has_online_memory(struct daxctl_dev *dev);
 
+/*
+ * dax_kmem whole-device hotplug state (daxX.Y/state). UNKNOWN means the
+ * attribute is absent (older kernel, or device not bound to kmem) or
+ * reports a state this library does not recognize.
+ */
+enum daxctl_kmem_state {
+	DAXCTL_KMEM_UNKNOWN = -1,
+	DAXCTL_KMEM_UNPLUGGED = 0,
+	DAXCTL_KMEM_OFFLINE,
+	DAXCTL_KMEM_ONLINE,
+	DAXCTL_KMEM_ONLINE_KERNEL,
+	DAXCTL_KMEM_ONLINE_MOVABLE,
+};
+
+enum daxctl_kmem_state daxctl_dev_get_kmem_state(struct daxctl_dev *dev);
+int daxctl_dev_set_kmem_state(struct daxctl_dev *dev,
+		enum daxctl_kmem_state state);
+
 struct daxctl_memory;
 
 enum daxctl_dev_mode {
-- 
2.53.0-Meta

[PATCH 2/6] daxctl: online whole-device memory atomically when available

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

When the kernel provides daxX.Y/state, online a fully offline device
by writing the online policy to it rather than onlining each block.
This avoids racing other agents (udev, auto-online) that online blocks
as they appear. The kernel only onlines from 'unplugged', so a device
in 'offline' is unplugged first; if that fails (e.g. no memory
hotremove support), the device is unchanged and is onlined per-block.

Memory onlined by another agent is left alone: the 'offline' case
re-checks that no block is online before unplugging, and losing a
racing online falls back to the per-block path. A small window remains
between that check and the unplug.

The per-block path now fails when the device has no memory blocks,
rather than reporting zero sections already online.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 daxctl/device.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)

diff --git a/daxctl/device.c b/daxctl/device.c
index 47942f1..fd84799 100644
--- a/daxctl/device.c
+++ b/daxctl/device.c
@@ -510,10 +510,99 @@ static const char *parse_device_options(int argc, const char **argv,
 	return device;
 }
 
+static int dev_online_memory_atomic(struct daxctl_dev *dev,
+		struct daxctl_memory *mem, enum daxctl_kmem_state state)
+{
+	const char *devname = daxctl_dev_get_devname(dev);
+	enum daxctl_kmem_state target;
+	int num_sections, num_on, rc;
+
+	if (param.no_movable)
+		target = DAXCTL_KMEM_ONLINE;
+	else
+		target = DAXCTL_KMEM_ONLINE_MOVABLE;
+
+	if (state == DAXCTL_KMEM_OFFLINE) {
+		/*
+		 * Onlining requires 'unplugged'. Don't unplug memory another
+		 * agent onlined meanwhile (a small window remains).
+		 */
+		num_on = daxctl_memory_is_online(mem);
+		if (num_on < 0) {
+			fprintf(stderr, "%s: failed to determine online state: %s\n",
+				devname, strerror(-num_on));
+			return num_on;
+		}
+		if (num_on > 0)
+			return -EAGAIN;
+		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
+		if (rc < 0) {
+			/* unchanged (e.g. no hotremove support); go per-block */
+			if (param.verbose)
+				fprintf(stderr, "%s: unplug failed: %s\n",
+					devname, strerror(-rc));
+			return -EAGAIN;
+		}
+	}
+
+	rc = daxctl_dev_set_kmem_state(dev, target);
+	if (rc == -EBUSY) {
+		/* back off only if another writer onlined it */
+		switch (daxctl_dev_get_kmem_state(dev)) {
+		case DAXCTL_KMEM_ONLINE:
+		case DAXCTL_KMEM_ONLINE_KERNEL:
+		case DAXCTL_KMEM_ONLINE_MOVABLE:
+			return -EAGAIN;
+		default:
+			break;
+		}
+	}
+	if (rc < 0) {
+		fprintf(stderr, "%s: failed to online memory: %s\n",
+			devname, strerror(-rc));
+		if (state == DAXCTL_KMEM_OFFLINE)
+			fprintf(stderr,
+				"%s: device left unplugged; retry to recover\n",
+				devname);
+		return rc;
+	}
+
+	/* all sections should now be online */
+	num_sections = daxctl_memory_num_sections(mem);
+	if (num_sections < 0) {
+		fprintf(stderr, "%s: failed to get number of memory sections\n",
+			devname);
+		return num_sections;
+	}
+	if (num_sections == 0) {
+		fprintf(stderr, "%s: no memory sections present after online\n",
+			devname);
+		return -ENXIO;
+	}
+
+	num_on = daxctl_memory_is_online(mem);
+	if (num_on < 0) {
+		fprintf(stderr, "%s: failed to determine online state: %s\n",
+			devname, strerror(-num_on));
+		return num_on;
+	}
+	if (num_on < num_sections) {
+		fprintf(stderr, "%s: failed to online %d memory sections\n",
+			devname, num_sections - num_on);
+		return -ENXIO;
+	}
+	if (param.verbose)
+		fprintf(stderr, "%s: %d memory section%s onlined\n", devname,
+			num_on, num_on == 1 ? "" : "s");
+
+	return 0;
+}
+
 static int dev_online_memory(struct daxctl_dev *dev)
 {
 	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
 	const char *devname = daxctl_dev_get_devname(dev);
+	enum daxctl_kmem_state state;
 	int num_sections, num_on, rc;
 
 	if (!mem) {
@@ -522,6 +611,18 @@ static int dev_online_memory(struct daxctl_dev *dev)
 		return -ENXIO;
 	}
 
+	/*
+	 * Online a fully offline device atomically where supported.
+	 * -EAGAIN means another agent onlined memory, or the device
+	 * could not be unplugged first; go per-block.
+	 */
+	state = daxctl_dev_get_kmem_state(dev);
+	if (state == DAXCTL_KMEM_UNPLUGGED || state == DAXCTL_KMEM_OFFLINE) {
+		rc = dev_online_memory_atomic(dev, mem, state);
+		if (rc != -EAGAIN)
+			return rc;
+	}
+
 	/* get total number of sections and sections already online */
 	num_sections = daxctl_memory_num_sections(mem);
 	if (num_sections < 0) {
@@ -529,6 +630,10 @@ static int dev_online_memory(struct daxctl_dev *dev)
 			devname);
 		return num_sections;
 	}
+	if (num_sections == 0) {
+		fprintf(stderr, "%s: no memory sections present\n", devname);
+		return -ENXIO;
+	}
 
 	num_on = daxctl_memory_is_online(mem);
 	if (num_on < 0) {
-- 
2.53.0-Meta

[PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

When the kernel provides daxX.Y/state, offline by writing 'unplugged'.
The kernel offlines and removes the whole device atomically, so no
agent can online blocks before a later unbind. On failure the device
is left unchanged.

The memory blocks are removed rather than left offline, including
when they are all offline already; 'daxctl online-memory' restores
them. If they are all offline and the unplug fails (e.g. no memory
hotremove support), this remains a no-op.

Unplugging a node's last memory removes its sysfs directory, so treat
a missing node directory as zero blocks when the device is unplugged.
'daxctl list' likewise omits 'movable' for a device with no blocks.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 daxctl/device.c        | 51 +++++++++++++++++++++++++++++++++++++++---
 daxctl/json.c          |  6 ++++-
 daxctl/lib/libdaxctl.c | 12 ++++++++--
 3 files changed, 63 insertions(+), 6 deletions(-)

diff --git a/daxctl/device.c b/daxctl/device.c
index fd84799..99c0d96 100644
--- a/daxctl/device.c
+++ b/daxctl/device.c
@@ -694,7 +694,8 @@ static int dev_offline_memory(struct daxctl_dev *dev)
 {
 	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
 	const char *devname = daxctl_dev_get_devname(dev);
-	int num_sections, num_on, num_off, rc;
+	int num_sections, num_on, num_off, rc, unplug;
+	enum daxctl_kmem_state state;
 
 	if (!mem) {
 		fprintf(stderr, "%s: failed to get the memory object\n",
@@ -717,17 +718,61 @@ static int dev_offline_memory(struct daxctl_dev *dev)
 		return num_on;
 	}
 
+	/*
+	 * Unplug atomically where supported, even if already offline, so
+	 * no agent can online blocks before a later unbind.
+	 */
+	state = daxctl_dev_get_kmem_state(dev);
+	unplug = state != DAXCTL_KMEM_UNKNOWN && state != DAXCTL_KMEM_UNPLUGGED;
+
 	num_off = num_sections - num_on;
-	if (num_off == num_sections) {
+	if (num_off == num_sections && !unplug) {
 		fprintf(stderr, "%s: all memory sections (%d) already offline\n",
 			devname, num_off);
 		return 1;
 	}
-	if (num_off)
+	if (num_off && num_on)
 		fprintf(stderr, "%s: %d memory section%s already offline\n",
 			devname, num_off,
 			num_off == 1 ? "" : "s");
 
+	if (unplug) {
+		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
+		if (rc < 0 && num_on == 0) {
+			/* already offline, unplug is best effort */
+			if (param.verbose)
+				fprintf(stderr, "%s: unplug failed: %s\n",
+					devname, strerror(-rc));
+			fprintf(stderr, "%s: all memory sections (%d) already offline\n",
+				devname, num_off);
+			return 1;
+		}
+		if (rc < 0) {
+			fprintf(stderr, "%s: failed to offline memory: %s\n",
+				devname, strerror(-rc));
+			return rc;
+		}
+		if (param.verbose)
+			fprintf(stderr, "%s: %d memory section%s unplugged\n",
+				devname, num_sections,
+				num_sections == 1 ? "" : "s");
+
+		/* blocks are removed, none should remain online */
+		num_on = daxctl_memory_is_online(mem);
+		if (num_on < 0) {
+			fprintf(stderr, "%s: failed to determine online state: %s\n",
+				devname, strerror(-num_on));
+			return num_on;
+		}
+		if (num_on) {
+			fprintf(stderr, "%s: failed to offline %d memory sections\n",
+				devname, num_on);
+			return -ENXIO;
+		}
+
+		return 0;
+	}
+
 	/* offline the remaining sections */
 	rc = daxctl_memory_offline(mem);
 	if (rc < 0) {
diff --git a/daxctl/json.c b/daxctl/json.c
index 8da91b1..cf81fe6 100644
--- a/daxctl/json.c
+++ b/daxctl/json.c
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 // Copyright (C) 2015-2020 Intel Corporation. All rights reserved.
+#include <errno.h>
 #include <limits.h>
 #include <string.h>
 #include <util/json.h>
@@ -79,7 +80,10 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 		if (jobj)
 			json_object_object_add(jdev, "total_memblocks", jobj);
 
-		movable = daxctl_memory_is_movable(mem);
+		/* an unplugged device has no zone */
+		movable = -ENXIO;
+		if (num_sections > 0)
+			movable = daxctl_memory_is_movable(mem);
 		if (movable == 1)
 			jobj = json_object_new_boolean(true);
 		else if (movable == 0)
diff --git a/daxctl/lib/libdaxctl.c b/daxctl/lib/libdaxctl.c
index cbb42e3..eedf4dd 100644
--- a/daxctl/lib/libdaxctl.c
+++ b/daxctl/lib/libdaxctl.c
@@ -1644,8 +1644,16 @@ static int daxctl_memory_op(struct daxctl_memory *mem, enum memory_op op)
 	}
 
 	node_dir = opendir(node_path);
-	if (!node_dir)
-		return -errno;
+	if (!node_dir) {
+		rc = -errno;
+		/* unplugging a node's last memory removes its directory */
+		if (rc == -ENOENT &&
+		    daxctl_dev_get_kmem_state(dev) == DAXCTL_KMEM_UNPLUGGED) {
+			dbg(ctx, "%s: unplugged, no blocks to count\n", devname);
+			return 0;
+		}
+		return rc;
+	}
 
 	errno = 0;
 	while ((de = readdir(node_dir)) != NULL) {
-- 
2.53.0-Meta

[PATCH 4/6] daxctl: report whole-device hotplug state in 'daxctl list'

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

Report daxX.Y/state as 'hotplug_state' in 'daxctl list'. The kernel
only updates it when written, so it goes stale if blocks are toggled
individually. Report 'inconsistent' when the blocks' online state or
zone disagrees with it; online_memblocks, total_memblocks and movable
show the actual state.

The attribute only appears when daxX.Y/state exists, so output on
older kernels is unchanged. ('state' already reports 'disabled'.)

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 daxctl/json.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/daxctl/json.c b/daxctl/json.c
index cf81fe6..986b695 100644
--- a/daxctl/json.c
+++ b/daxctl/json.c
@@ -12,6 +12,42 @@
 #include "filter.h"
 #include "json.h"
 
+/*
+ * daxX.Y/state only changes when written, so it goes stale if blocks are
+ * toggled individually. Report "inconsistent" if the blocks disagree.
+ */
+static const char *hotplug_state(enum daxctl_kmem_state state, int online,
+		int total, int movable)
+{
+	int all_online = total > 0 && online == total;
+
+	switch (state) {
+	case DAXCTL_KMEM_UNPLUGGED:
+		if (total == 0)
+			return "unplugged";
+		break;
+	case DAXCTL_KMEM_OFFLINE:
+		if (total > 0 && online == 0)
+			return "offline";
+		break;
+	case DAXCTL_KMEM_ONLINE:
+		if (all_online)
+			return "online";
+		break;
+	case DAXCTL_KMEM_ONLINE_KERNEL:
+		if (all_online && movable == 0)
+			return "online_kernel";
+		break;
+	case DAXCTL_KMEM_ONLINE_MOVABLE:
+		if (all_online && movable == 1)
+			return "online_movable";
+		break;
+	default:
+		break;
+	}
+	return "inconsistent";
+}
+
 struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 		unsigned long flags)
 {
@@ -19,6 +55,7 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 	const char *devname = daxctl_dev_get_devname(dev);
 	struct json_object *jdev, *jobj, *jmappings = NULL;
 	struct daxctl_mapping *mapping = NULL;
+	enum daxctl_kmem_state state;
 	int node, movable, align;
 
 	jdev = json_object_new_object();
@@ -92,6 +129,16 @@ struct json_object *util_daxctl_dev_to_json(struct daxctl_dev *dev,
 			jobj = NULL;
 		if (jobj)
 			json_object_object_add(jdev, "movable", jobj);
+
+		state = daxctl_dev_get_kmem_state(dev);
+		if (state != DAXCTL_KMEM_UNKNOWN && num_online >= 0 &&
+		    num_sections >= 0) {
+			jobj = json_object_new_string(hotplug_state(state,
+					num_online, num_sections, movable));
+			if (jobj)
+				json_object_object_add(jdev, "hotplug_state",
+						       jobj);
+		}
 	}
 
 	if (!daxctl_dev_is_enabled(dev)) {
-- 
2.53.0-Meta

[PATCH 5/6] Documentation: describe atomic whole-device hotplug in daxctl man pages

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

Document that online-memory and offline-memory use daxX.Y/state where
available, that offlining always removes the memory blocks and how
that affects reconfigure-device, and the 'hotplug_state' attribute of
'daxctl list'.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 Documentation/daxctl/daxctl-list.txt               | 5 +++++
 Documentation/daxctl/daxctl-offline-memory.txt     | 5 +++++
 Documentation/daxctl/daxctl-online-memory.txt      | 5 +++++
 Documentation/daxctl/daxctl-reconfigure-device.txt | 5 ++++-
 4 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/Documentation/daxctl/daxctl-list.txt b/Documentation/daxctl/daxctl-list.txt
index 5a6a98e..a2d0ede 100644
--- a/Documentation/daxctl/daxctl-list.txt
+++ b/Documentation/daxctl/daxctl-list.txt
@@ -22,6 +22,11 @@ list' with no options is equivalent to:
 [verse]
 daxctl list --devices
 
+On kernels that support atomic whole-device hotplug, 'hotplug_state'
+reports /sys/bus/dax/devices/daxX.Y/state for system-ram devices, or
+'inconsistent' if the device's memory blocks disagree with it (e.g.
+after blocks were onlined or offlined individually).
+
 EXAMPLE
 -------
 ----
diff --git a/Documentation/daxctl/daxctl-offline-memory.txt b/Documentation/daxctl/daxctl-offline-memory.txt
index ba06287..61a4301 100644
--- a/Documentation/daxctl/daxctl-offline-memory.txt
+++ b/Documentation/daxctl/daxctl-offline-memory.txt
@@ -41,6 +41,11 @@ to the system-ram mode. If one or more blocks are already offline, attempt to
 offline the remaining blocks. If all blocks were already offline, print a
 message and return success without actually doing anything.
 
+If the kernel supports atomic whole-device hotplug
+(/sys/bus/dax/devices/daxX.Y/state), the device is instead unplugged,
+even if its memory is already offline: the memory is offlined and
+removed in one atomic operation. 'daxctl-online-memory' restores it.
+
 This is complementary to the 'daxctl-online-memory' command, and may be used
 when it is wished to offline the memory sections, but not convert the device
 back to 'devdax' mode.
diff --git a/Documentation/daxctl/daxctl-online-memory.txt b/Documentation/daxctl/daxctl-online-memory.txt
index 08b45cc..adc8517 100644
--- a/Documentation/daxctl/daxctl-online-memory.txt
+++ b/Documentation/daxctl/daxctl-online-memory.txt
@@ -48,6 +48,11 @@ Online the memory sections associated with a device that has been converted
 to the system-ram mode. If one or more blocks are already online, print a
 message about them, and attempt to online the remaining blocks.
 
+If the kernel supports atomic whole-device hotplug
+(/sys/bus/dax/devices/daxX.Y/state), a fully offline device is onlined
+as a whole rather than block by block. An 'unplugged' device is re-added
+and onlined atomically; one with blocks present is unplugged first.
+
 This is complementary to the 'daxctl-reconfigure-device' command, when used with
 the '--no-online' option to skip onlining memory sections immediately after the
 reconfigure. In these scenarios, the memory can be onlined at a later time using
diff --git a/Documentation/daxctl/daxctl-reconfigure-device.txt b/Documentation/daxctl/daxctl-reconfigure-device.txt
index 9c3922d..d2b5bbf 100644
--- a/Documentation/daxctl/daxctl-reconfigure-device.txt
+++ b/Documentation/daxctl/daxctl-reconfigure-device.txt
@@ -230,6 +230,8 @@ include::region-option.txt[]
 	By default, memory sections provided by system-ram devices will be
 	brought online automatically and immediately with the 'online_movable'
 	policy. Use this option to disable the automatic onlining behavior.
+	A device unplugged by 'daxctl-offline-memory' has no memory blocks
+	to online; use 'daxctl-online-memory' to restore them.
 
 -C::
 --check-config::
@@ -248,7 +250,8 @@ include::movable-options.txt[]
 	that all the memory sections are first made offline. By default,
 	daxctl won't touch online memory. However with this option, attempt
 	to offline the memory on the NUMA node associated with the dax device
-	before converting it back to "devdax" mode.
+	before converting it back to "devdax" mode. This uses
+	'daxctl-offline-memory', so the device is unplugged where supported.
 
 	- Additionally, if a kernel policy to auto-online blocks is detected,
 	reconfiguration to system-ram fails. With this option, the failure can
-- 
2.53.0-Meta

[PATCH 6/6] test/daxctl-devices.sh: cycle offline/online to cover unplug and replug

[Gregory Price (Meta)]

From: Gregory Price <gourry@gourry.net>

Repeat the online/offline pair to cover onlining from 'unplugged', and
check that offline-memory unplugs an already offline device. After each
step, check block counts and, where present, daxX.Y/state and the
'hotplug_state' reported by 'daxctl list'.

Signed-off-by: Gregory Price <gourry@gourry.net>
---
 test/daxctl-devices.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/test/daxctl-devices.sh b/test/daxctl-devices.sh
index 659428d..cdb13f6 100755
--- a/test/daxctl-devices.sh
+++ b/test/daxctl-devices.sh
@@ -67,6 +67,40 @@ daxctl_get_mode()
 	"$DAXCTL" list -d "$1" | jq -er '.[].mode'
 }
 
+daxctl_get_online_blocks()
+{
+	"$DAXCTL" list -d "$1" | jq -er '.[].online_memblocks'
+}
+
+daxctl_get_total_blocks()
+{
+	"$DAXCTL" list -d "$1" | jq -er '.[].total_memblocks'
+}
+
+check_memory_online()
+{
+	local total
+
+	total=$(daxctl_get_total_blocks "$1")
+	(( total > 0 ))
+	[[ $(daxctl_get_online_blocks "$1") == "$total" ]]
+}
+
+check_memory_offline()
+{
+	[[ $(daxctl_get_online_blocks "$1") == "0" ]]
+}
+
+# skipped on kernels without daxX.Y/state
+check_kmem_state()
+{
+	local path="/sys/bus/dax/devices/$1/state"
+
+	[[ -f "$path" ]] || return 0
+	[[ $(cat "$path") == "$2" ]]
+	[[ $("$DAXCTL" list -d "$1" | jq -er '.[].hotplug_state') == "$2" ]]
+}
+
 set_online_policy()
 {
 	echo "online" > /sys/devices/system/memory/auto_online_blocks
@@ -100,7 +134,26 @@ daxctl_test()
 	"$DAXCTL" reconfigure-device -N -m system-ram "$daxdev"
 	[[ $(daxctl_get_mode "$daxdev") == "system-ram" ]]
 	"$DAXCTL" online-memory "$daxdev"
+	check_memory_online "$daxdev"
+	check_kmem_state "$daxdev" "online_movable"
 	"$DAXCTL" offline-memory "$daxdev"
+	check_memory_offline "$daxdev"
+	check_kmem_state "$daxdev" "unplugged"
+	# again, now from 'unplugged' where supported
+	"$DAXCTL" online-memory "$daxdev"
+	check_memory_online "$daxdev"
+	check_kmem_state "$daxdev" "online_movable"
+	"$DAXCTL" offline-memory "$daxdev"
+	check_memory_offline "$daxdev"
+	check_kmem_state "$daxdev" "unplugged"
+	# an already offline device is unplugged too, where supported
+	if [[ -f /sys/bus/dax/devices/$daxdev/state ]]; then
+		"$DAXCTL" reconfigure-device -m devdax "$daxdev"
+		"$DAXCTL" reconfigure-device -N -m system-ram "$daxdev"
+		check_kmem_state "$daxdev" "offline"
+		"$DAXCTL" offline-memory "$daxdev"
+		check_kmem_state "$daxdev" "unplugged"
+	fi
 	"$DAXCTL" reconfigure-device -m devdax "$daxdev"
 	[[ $(daxctl_get_mode "$daxdev") == "devdax" ]]
 	"$DAXCTL" reconfigure-device -m system-ram "$daxdev"
-- 
2.53.0-Meta

Re: [PATCH 0/6] daxctl: use dax kmem atomic whole-device hotplug

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:42PM -0400, Gregory Price (Meta) wrote:
> Kernels with the dax_kmem daxX.Y/state attribute [1] can unplug or
> plug-and-online a whole device atomically. Use it in daxctl
> online-memory and offline-memory so other agents (udev, auto-online)
> can't race daxctl block by block, notably the offline-then-unbind race
> on teardown.
> 
> Kernels without the attribute keep the per-block path.
> 
> Behavior change: offline-memory now always unplugs the device (state
> 'unplugged'), removing its memory blocks, even if they were all
> offline already. Previously that was a no-op.
> 
> 'daxctl list' (and 'ndctl/cxl list -X') gains 'hotplug_state',
> reporting daxX.Y/state. The kernel only updates that when written, so
> it goes stale if blocks are toggled individually.
> 
> If there is a mismatch between 'dax/state' and the blocks, then
> 'hotplug_state' reports 'inconsistent' (online state / zones disagree).
> That is intentional: it flags agents fighting over the device.
> It only appears when the attribute exists.
> 
> Compatibility notes:
> - Where the unplug fails (e.g. no memory hotremove support),
>   online-memory falls back to the per-block path and offline-memory
>   of an already offline device remains a no-op.
> - Older daxctl/ndctl/cxl binaries don't expect a device with no memory
>   blocks.  Basically multiple daxctl versions on the same box can cause
>   older versions to fail when in the unplugged state.  Not a huge issue.
> - If re-adding fails after an unplug (e.g. region can't be reserved)
>   the device is left unplugged.
> - For multi-range devices the kernel succeeds if any range is added.
>   daxctl can't tell that a range is missing.
> 
> [1] https://lore.kernel.org/all/20260712154505.3564379-1-gourry@gourry.net/

Hi Greg,

Next version, please add the ndctl label, ie [ndctl PATCH v2 0/6]
(https://github.com/pmem/ndctl?tab=contributing-ov-file)

I see this goes with the kernel functionality that arrived in kernel 7.3.
That took me a bit to realized because the linked kernel series shows
crickets at v7.

I'd like to get this into the pending ndctl release, so let's see if we
can close on a few things.

I tested against 7.3, regression and new tests and all is good there.
I tested against 7.2 because daxctl command functionality itself needs to
be backward compatible, and it is!

Some per patch feedback and tags follow.

-- Alison

> 
> Gregory Price (6):
>   libdaxctl: add dax kmem atomic hotplug state accessors
>   daxctl: online whole-device memory atomically when available
>   daxctl: offline whole-device memory via atomic unplug when available
>   daxctl: report whole-device hotplug state in 'daxctl list'
>   Documentation: describe atomic whole-device hotplug in daxctl man
>     pages
>   test/daxctl-devices.sh: cycle offline/online to cover unplug and
>     replug
> 
>  Documentation/daxctl/daxctl-list.txt          |   5 +
>  .../daxctl/daxctl-offline-memory.txt          |   5 +
>  Documentation/daxctl/daxctl-online-memory.txt |   5 +
>  .../daxctl/daxctl-reconfigure-device.txt      |   5 +-
>  daxctl/device.c                               | 156 +++++++++++++++++-
>  daxctl/json.c                                 |  53 +++++-
>  daxctl/lib/libdaxctl-private.h                |   9 +
>  daxctl/lib/libdaxctl.c                        |  71 +++++++-
>  daxctl/lib/libdaxctl.sym                      |   6 +
>  daxctl/libdaxctl.h                            |  18 ++
>  test/daxctl-devices.sh                        |  53 ++++++
>  11 files changed, 379 insertions(+), 7 deletions(-)
> 
> -- 
> 2.53.0-Meta
> 
> 

Re: [PATCH 1/6] libdaxctl: add dax kmem atomic hotplug state accessors

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:43PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> Recent kernels add /sys/bus/dax/devices/daxX.Y/state to dax_kmem
> devices for atomic whole-device hotplug. It reads 'unplugged',
> 'offline', 'online', 'online_kernel' or 'online_movable', and accepts
> all but 'offline'.
> 
> Add an enum and get/set accessors for it. The getter returns
> DAXCTL_KMEM_UNKNOWN when the attribute is absent, so callers can fall
> back to per-block operations.


Reviewed-by: Alison Schofield <alison.schofield@intel.com>

Re: [PATCH 2/6] daxctl: online whole-device memory atomically when available

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:44PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> When the kernel provides daxX.Y/state, online a fully offline device
> by writing the online policy to it rather than onlining each block.
> This avoids racing other agents (udev, auto-online) that online blocks
> as they appear. The kernel only onlines from 'unplugged', so a device
> in 'offline' is unplugged first; if that fails (e.g. no memory
> hotremove support), the device is unchanged and is onlined per-block.
> 
> Memory onlined by another agent is left alone: the 'offline' case
> re-checks that no block is online before unplugging, and losing a
> racing online falls back to the per-block path. A small window remains
> between that check and the unplug.
> 
> The per-block path now fails when the device has no memory blocks,
> rather than reporting zero sections already online.
> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
>  daxctl/device.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 105 insertions(+)
> 
> diff --git a/daxctl/device.c b/daxctl/device.c
> index 47942f1..fd84799 100644
> --- a/daxctl/device.c
> +++ b/daxctl/device.c
> @@ -510,10 +510,99 @@ static const char *parse_device_options(int argc, const char **argv,
>  	return device;
>  }
>  
> +static int dev_online_memory_atomic(struct daxctl_dev *dev,
> +		struct daxctl_memory *mem, enum daxctl_kmem_state state)
> +{
> +	const char *devname = daxctl_dev_get_devname(dev);
> +	enum daxctl_kmem_state target;
> +	int num_sections, num_on, rc;
> +
> +	if (param.no_movable)
> +		target = DAXCTL_KMEM_ONLINE;
> +	else
> +		target = DAXCTL_KMEM_ONLINE_MOVABLE;
> +

Hi Greg, 

A couple of things here...

The atomic --no-movable path uses DAXCTL_KMEM_ONLINE, leaving zone selection
to the kernel. With memory_hotplug.online_policy=auto-movable, IIUC this could
place memory in ZONE_MOVABLE despite --no-movable.

The per-block path checks the resulting zone and fails if it isn't ZONE_NORMAL,
but the atomic path skips that check.

Should this use DAXCTL_KMEM_ONLINE_KERNEL instead, or retain DAXCTL_KMEM_ONLINE
and add the same zone check?


> +	if (state == DAXCTL_KMEM_OFFLINE) {
> +		/*
> +		 * Onlining requires 'unplugged'. Don't unplug memory another
> +		 * agent onlined meanwhile (a small window remains).
> +		 */

With this, a "reconfigure-device -m system-ram" may add the memory twice.
First kmem adds it offline at probe and then daxctl unplugs it and adds it
back online. If that re-add fails, the device is left unplugged rather than
offline.

I get that falling back to per-block onlining would reopen the race this
series is addressing. Could the kernel interface support transitioning
directly from offline to an online state instead? It seems that would avoid
both the extra hotplug work and the left-unplugged failure case.

-- Alison

> +		num_on = daxctl_memory_is_online(mem);
> +		if (num_on < 0) {
> +			fprintf(stderr, "%s: failed to determine online state: %s\n",
> +				devname, strerror(-num_on));
> +			return num_on;
> +		}
> +		if (num_on > 0)
> +			return -EAGAIN;
> +		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
> +		if (rc < 0) {
> +			/* unchanged (e.g. no hotremove support); go per-block */
> +			if (param.verbose)
> +				fprintf(stderr, "%s: unplug failed: %s\n",
> +					devname, strerror(-rc));
> +			return -EAGAIN;
> +		}
> +	}
> +
> +	rc = daxctl_dev_set_kmem_state(dev, target);
> +	if (rc == -EBUSY) {
> +		/* back off only if another writer onlined it */
> +		switch (daxctl_dev_get_kmem_state(dev)) {
> +		case DAXCTL_KMEM_ONLINE:
> +		case DAXCTL_KMEM_ONLINE_KERNEL:
> +		case DAXCTL_KMEM_ONLINE_MOVABLE:
> +			return -EAGAIN;
> +		default:
> +			break;
> +		}
> +	}
> +	if (rc < 0) {
> +		fprintf(stderr, "%s: failed to online memory: %s\n",
> +			devname, strerror(-rc));
> +		if (state == DAXCTL_KMEM_OFFLINE)
> +			fprintf(stderr,
> +				"%s: device left unplugged; retry to recover\n",
> +				devname);
> +		return rc;
> +	}
> +
> +	/* all sections should now be online */
> +	num_sections = daxctl_memory_num_sections(mem);
> +	if (num_sections < 0) {
> +		fprintf(stderr, "%s: failed to get number of memory sections\n",
> +			devname);
> +		return num_sections;
> +	}
> +	if (num_sections == 0) {
> +		fprintf(stderr, "%s: no memory sections present after online\n",
> +			devname);
> +		return -ENXIO;
> +	}
> +
> +	num_on = daxctl_memory_is_online(mem);
> +	if (num_on < 0) {
> +		fprintf(stderr, "%s: failed to determine online state: %s\n",
> +			devname, strerror(-num_on));
> +		return num_on;
> +	}
> +	if (num_on < num_sections) {
> +		fprintf(stderr, "%s: failed to online %d memory sections\n",
> +			devname, num_sections - num_on);
> +		return -ENXIO;
> +	}
> +	if (param.verbose)
> +		fprintf(stderr, "%s: %d memory section%s onlined\n", devname,
> +			num_on, num_on == 1 ? "" : "s");
> +
> +	return 0;
> +}
> +

snip

Re: [PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:45PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> When the kernel provides daxX.Y/state, offline by writing 'unplugged'.
> The kernel offlines and removes the whole device atomically, so no
> agent can online blocks before a later unbind. On failure the device
> is left unchanged.
> 
> The memory blocks are removed rather than left offline, including
> when they are all offline already; 'daxctl online-memory' restores
> them. If they are all offline and the unplug fails (e.g. no memory
> hotremove support), this remains a no-op.
> 
> Unplugging a node's last memory removes its sysfs directory, so treat
> a missing node directory as zero blocks when the device is unplugged.
> 'daxctl list' likewise omits 'movable' for a device with no blocks.
> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
>  daxctl/device.c        | 51 +++++++++++++++++++++++++++++++++++++++---
>  daxctl/json.c          |  6 ++++-
>  daxctl/lib/libdaxctl.c | 12 ++++++++--
>  3 files changed, 63 insertions(+), 6 deletions(-)
> 
> diff --git a/daxctl/device.c b/daxctl/device.c
> index fd84799..99c0d96 100644
> --- a/daxctl/device.c
> +++ b/daxctl/device.c
> @@ -694,7 +694,8 @@ static int dev_offline_memory(struct daxctl_dev *dev)
>  {
>  	struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
>  	const char *devname = daxctl_dev_get_devname(dev);
> -	int num_sections, num_on, num_off, rc;
> +	int num_sections, num_on, num_off, rc, unplug;
> +	enum daxctl_kmem_state state;
>  
>  	if (!mem) {
>  		fprintf(stderr, "%s: failed to get the memory object\n",
> @@ -717,17 +718,61 @@ static int dev_offline_memory(struct daxctl_dev *dev)
>  		return num_on;
>  	}
>  
> +	/*
> +	 * Unplug atomically where supported, even if already offline, so
> +	 * no agent can online blocks before a later unbind.
> +	 */
> +	state = daxctl_dev_get_kmem_state(dev);
> +	unplug = state != DAXCTL_KMEM_UNKNOWN && state != DAXCTL_KMEM_UNPLUGGED;
> +

Hi Greg,

I see you called this out in the cover letter, and that the kernel is
deliberately combining offlining and removing memory to avoid the race
before unbind.

But, this does change the behavior of the command "daxctl offline-memory".
Previously it left memory blocks present but offline and now it removes
them entirely.

Should offline-memory keep its existing behavior of leaving memory blocks
offline, and use atomic unplug only when reconfiguring the device?



>  	num_off = num_sections - num_on;
> -	if (num_off == num_sections) {
> +	if (num_off == num_sections && !unplug) {
>  		fprintf(stderr, "%s: all memory sections (%d) already offline\n",
>  			devname, num_off);
>  		return 1;
>  	}
> -	if (num_off)
> +	if (num_off && num_on)
>  		fprintf(stderr, "%s: %d memory section%s already offline\n",
>  			devname, num_off,
>  			num_off == 1 ? "" : "s");
>  
> +	if (unplug) {
> +		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
> +		if (rc < 0 && num_on == 0) {
> +			/* already offline, unplug is best effort */

When num_on == 0, another agent could online a block between the earlier
check and the unplug and that memory would then be removed.

I see this noted in the commit message. Is that worth documenting for
users?


-- Alison

Re: [PATCH 4/6] daxctl: report whole-device hotplug state in 'daxctl list'

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:46PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> Report daxX.Y/state as 'hotplug_state' in 'daxctl list'. The kernel
> only updates it when written, so it goes stale if blocks are toggled
> individually. Report 'inconsistent' when the blocks' online state or
> zone disagrees with it; online_memblocks, total_memblocks and movable
> show the actual state.
> 
> The attribute only appears when daxX.Y/state exists, so output on
> older kernels is unchanged. ('state' already reports 'disabled'.)


Reviewed-by: Alison Schofield <alison.schofield@intel.com>

Re: [PATCH 5/6] Documentation: describe atomic whole-device hotplug in daxctl man pages

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:47PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> Document that online-memory and offline-memory use daxX.Y/state where
> available, that offlining always removes the memory blocks and how
> that affects reconfigure-device, and the 'hotplug_state' attribute of
> 'daxctl list'.

Hi Greg,

Can you add hotplug_state to sample outputs in daxctl-list.txt and
cxl-list.txt?

-- Alison

Re: [PATCH 6/6] test/daxctl-devices.sh: cycle offline/online to cover unplug and replug

[Alison Schofield]

On Fri, Oct 16, 2026 at 05:42:48PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <gourry@gourry.net>
> 
> Repeat the online/offline pair to cover onlining from 'unplugged', and
> check that offline-memory unplugs an already offline device. After each
> step, check block counts and, where present, daxX.Y/state and the
> 'hotplug_state' reported by 'daxctl list'.

Hi Greg,

Please change how the tests handle the absence of the new sysfs attribute.
If a test expects the new interface, a missing attribute should cause
the test to fail, with a clear error message. We have had problems in
the past with tests silently not executing letting defects go undetected.

Note that we don't require backward compatibility for the unit tests.
The utilities must continue working with older kernels, but tests for new
functionality should fail loudly when that functionality is missing.
(I'm aware that not all tests follow that guideline, and they need fixing.)

Please also leave the existing test cases as they were and run the new
state-dependent cases last.

A few stale state cases would exercise daxctl behavior that the kernel
selftest can't cover - 

  - Offline one memory block out-of-band, verify that daxctl list
    reports hotplug_state "inconsistent", then verify that
    offline-memory still leaves the device unplugged with its blocks
    removed.

  - Offline all blocks out-of-band, then verify that
    "reconfigure-device -m devdax" succeeds and the blocks are removed.

  - With all blocks offline but the device state still reporting
    online, verify that online-memory falls back to the per-block path
    and brings everything online.

For "blocks removed", please check that the device's memory blocks
are gone from /sys/devices/system/memory.  ie verify in sysfs directly
independent of the libdaxctl code being tested.

About multi-range devices-

I found that libdaxctl undercounts memory blocks on multi-range devices.
I'm posting a separate fix and test for that. Nothing in this series needs
to change. I'll cc you for review.

Our ndctl emulation can't exercise multi-range hotplug, but the kernel
selftest can run against a real multi-range device. Could you add coverage
there, including the partial-add case where the kernel reports success
even though not all ranges were added?

-- Alison


> 
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
>  test/daxctl-devices.sh | 53 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 53 insertions(+)
> 
> diff --git a/test/daxctl-devices.sh b/test/daxctl-devices.sh
> index 659428d..cdb13f6 100755
> --- a/test/daxctl-devices.sh
> +++ b/test/daxctl-devices.sh
> @@ -67,6 +67,40 @@ daxctl_get_mode()
>  	"$DAXCTL" list -d "$1" | jq -er '.[].mode'
>  }
>  
> +daxctl_get_online_blocks()
> +{
> +	"$DAXCTL" list -d "$1" | jq -er '.[].online_memblocks'
> +}
> +
> +daxctl_get_total_blocks()
> +{
> +	"$DAXCTL" list -d "$1" | jq -er '.[].total_memblocks'
> +}
> +
> +check_memory_online()
> +{
> +	local total
> +
> +	total=$(daxctl_get_total_blocks "$1")
> +	(( total > 0 ))
> +	[[ $(daxctl_get_online_blocks "$1") == "$total" ]]
> +}
> +
> +check_memory_offline()
> +{
> +	[[ $(daxctl_get_online_blocks "$1") == "0" ]]
> +}
> +
> +# skipped on kernels without daxX.Y/state
> +check_kmem_state()
> +{
> +	local path="/sys/bus/dax/devices/$1/state"
> +
> +	[[ -f "$path" ]] || return 0
> +	[[ $(cat "$path") == "$2" ]]
> +	[[ $("$DAXCTL" list -d "$1" | jq -er '.[].hotplug_state') == "$2" ]]
> +}
> +
>  set_online_policy()
>  {
>  	echo "online" > /sys/devices/system/memory/auto_online_blocks
> @@ -100,7 +134,26 @@ daxctl_test()
>  	"$DAXCTL" reconfigure-device -N -m system-ram "$daxdev"
>  	[[ $(daxctl_get_mode "$daxdev") == "system-ram" ]]
>  	"$DAXCTL" online-memory "$daxdev"
> +	check_memory_online "$daxdev"
> +	check_kmem_state "$daxdev" "online_movable"
>  	"$DAXCTL" offline-memory "$daxdev"
> +	check_memory_offline "$daxdev"
> +	check_kmem_state "$daxdev" "unplugged"
> +	# again, now from 'unplugged' where supported
> +	"$DAXCTL" online-memory "$daxdev"
> +	check_memory_online "$daxdev"
> +	check_kmem_state "$daxdev" "online_movable"
> +	"$DAXCTL" offline-memory "$daxdev"
> +	check_memory_offline "$daxdev"
> +	check_kmem_state "$daxdev" "unplugged"
> +	# an already offline device is unplugged too, where supported
> +	if [[ -f /sys/bus/dax/devices/$daxdev/state ]]; then
> +		"$DAXCTL" reconfigure-device -m devdax "$daxdev"
> +		"$DAXCTL" reconfigure-device -N -m system-ram "$daxdev"
> +		check_kmem_state "$daxdev" "offline"
> +		"$DAXCTL" offline-memory "$daxdev"
> +		check_kmem_state "$daxdev" "unplugged"
> +	fi
>  	"$DAXCTL" reconfigure-device -m devdax "$daxdev"
>  	[[ $(daxctl_get_mode "$daxdev") == "devdax" ]]
>  	"$DAXCTL" reconfigure-device -m system-ram "$daxdev"
> -- 
> 2.53.0-Meta
> 

Re: [PATCH 0/6] daxctl: use dax kmem atomic whole-device hotplug

[Gregory Price (Meta)]

On 2026-10-21 18:59 -0700, Alison Schofield wrote:
> On Fri, Oct 16, 2026 at 05:42:42PM -0400, Gregory Price (Meta) wrote:
> Hi Greg,
> 
> Next version, please add the ndctl label, ie [ndctl PATCH v2 0/6]
> (https://github.com/pmem/ndctl?tab=contributing-ov-file)

yeah sorry, i realized immediately after I sent it.  Noted for the
future.  I did also open a PR on github.

> 
> I see this goes with the kernel functionality that arrived in kernel 7.3.
> That took me a bit to realized because the linked kernel series shows
> crickets at v7.
> 
> I'd like to get this into the pending ndctl release, so let's see if we
> can close on a few things.
> 
> I tested against 7.3, regression and new tests and all is good there.
> I tested against 7.2 because daxctl command functionality itself needs to
> be backward compatible, and it is!
> 

I spent a good amount of time testing that it is :]

> Some per patch feedback and tags follow.
> 

ack

-- 
~Gregory

Re: [PATCH 2/6] daxctl: online whole-device memory atomically when available

[Gregory Price (Meta)]

On 2026-10-21 19:12 -0700, Alison Schofield wrote:
> On Fri, Oct 16, 2026 at 05:42:44PM -0400, Gregory Price (Meta) wrote:
> > From: Gregory Price <gourry@gourry.net>
> > +	if (param.no_movable)
> > +		target = DAXCTL_KMEM_ONLINE;
> > +	else
> > +		target = DAXCTL_KMEM_ONLINE_MOVABLE;
> > +
> 
> Hi Greg, 
> 
> A couple of things here...
> 
> The atomic --no-movable path uses DAXCTL_KMEM_ONLINE, leaving zone selection
> to the kernel. With memory_hotplug.online_policy=auto-movable, IIUC this could
> place memory in ZONE_MOVABLE despite --no-movable.
> 
> The per-block path checks the resulting zone and fails if it isn't ZONE_NORMAL,
> but the atomic path skips that check.
> 
> Should this use DAXCTL_KMEM_ONLINE_KERNEL instead, or retain DAXCTL_KMEM_ONLINE
> and add the same zone check?
>

I think you are right, let me go test.

> 
> > +	if (state == DAXCTL_KMEM_OFFLINE) {
> > +		/*
> > +		 * Onlining requires 'unplugged'. Don't unplug memory another
> > +		 * agent onlined meanwhile (a small window remains).
> > +		 */
> 
> With this, a "reconfigure-device -m system-ram" may add the memory twice.
> First kmem adds it offline at probe and then daxctl unplugs it and adds it
> back online. If that re-add fails, the device is left unplugged rather than
> offline.
>

This was the intended behavior because otherwise a concurrent
block/state change would end up making the blocks unrecoverable.

> I get that falling back to per-block onlining would reopen the race this
> series is addressing. Could the kernel interface support transitioning
> directly from offline to an online state instead? It seems that would avoid
> both the extra hotplug work and the left-unplugged failure case.
> 

This is an unfortunate wart for backward compatibility sake.  It would
have been preferable to create the dax device without auto-hotplugging
the kmem region (hotplug but offline), but that would break userspace.

I can take another look whether i can get offline->online to work
internally, but this may still require the hotplug work you're concerned
about.

-- 
~Gregory

Re: [PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Gregory Price (Meta)]

On 2026-10-21 19:20 -0700, Alison Schofield wrote:
> On Fri, Oct 16, 2026 at 05:42:45PM -0400, Gregory Price (Meta) wrote:
> > +	/*
> > +	 * Unplug atomically where supported, even if already offline, so
> > +	 * no agent can online blocks before a later unbind.
> > +	 */
> > +	state = daxctl_dev_get_kmem_state(dev);
> > +	unplug = state != DAXCTL_KMEM_UNKNOWN && state != DAXCTL_KMEM_UNPLUGGED;
> > +
> 
> Hi Greg,
> 
> I see you called this out in the cover letter, and that the kernel is
> deliberately combining offlining and removing memory to avoid the race
> before unbind.
> 
> But, this does change the behavior of the command "daxctl offline-memory".
> Previously it left memory blocks present but offline and now it removes
> them entirely.
> 
> Should offline-memory keep its existing behavior of leaving memory blocks
> offline, and use atomic unplug only when reconfiguring the device?
>

The kernel explicitly does not allow `echo offline > dax/state` because
this just creates the race we're trying to resolve.

I suppose my pushback is: why does it matter that daxctl offline-memory
changes in this way? If daxctl supports the atomic interface, it all just
works. It's a bit more work (additional hot-unplug) but if the memory is
already offline that's not that expensive.

The only scenario I could come up with is having two versions of daxctl
fight with each other (old - no awareness of dax/state, new - uses
dax/state) but is this a real concern?

> > +	if (unplug) {
> > +		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
> > +		if (rc < 0 && num_on == 0) {
> > +			/* already offline, unplug is best effort */
> 
> When num_on == 0, another agent could online a block between the earlier
> check and the unplug and that memory would then be removed.
> 
> I see this noted in the commit message. Is that worth documenting for
> users?
> 

Yes that's intended, and yeah should probably update the docs.

~Gregory

-- 
~Gregory

Re: [PATCH 5/6] Documentation: describe atomic whole-device hotplug in daxctl man pages

[Gregory Price (Meta)]

On 2026-10-21 19:23 -0700, Alison Schofield wrote:
> On Fri, Oct 16, 2026 at 05:42:47PM -0400, Gregory Price (Meta) wrote:
> > From: Gregory Price <gourry@gourry.net>
> > 
> > Document that online-memory and offline-memory use daxX.Y/state where
> > available, that offlining always removes the memory blocks and how
> > that affects reconfigure-device, and the 'hotplug_state' attribute of
> > 'daxctl list'.
> 
> Hi Greg,
> 
> Can you add hotplug_state to sample outputs in daxctl-list.txt and
> cxl-list.txt?
>

ack

-- 
~Gregory

Re: [PATCH 6/6] test/daxctl-devices.sh: cycle offline/online to cover unplug and replug

[Gregory Price (Meta)]

On 2026-10-21 19:34 -0700, Alison Schofield wrote:
> On Fri, Oct 16, 2026 at 05:42:48PM -0400, Gregory Price (Meta) wrote:
> > From: Gregory Price <gourry@gourry.net>
> > 
> > Repeat the online/offline pair to cover onlining from 'unplugged', and
> > check that offline-memory unplugs an already offline device. After each
> > step, check block counts and, where present, daxX.Y/state and the
> > 'hotplug_state' reported by 'daxctl list'.
> 
> Hi Greg,
> 
> Please change how the tests handle the absence of the new sysfs attribute.
> If a test expects the new interface, a missing attribute should cause
> the test to fail, with a clear error message. We have had problems in
> the past with tests silently not executing letting defects go undetected.
> 

ack

> Note that we don't require backward compatibility for the unit tests.
> The utilities must continue working with older kernels, but tests for new
> functionality should fail loudly when that functionality is missing.
> (I'm aware that not all tests follow that guideline, and they need fixing.)
> 

this is slightly confusing, but noted.  I would have expected the tests
to test backward compatibility.  But ack.

> Please also leave the existing test cases as they were and run the new
> state-dependent cases last.
> 

ack.

> A few stale state cases would exercise daxctl behavior that the kernel
> selftest can't cover - 
> 
>   - Offline one memory block out-of-band, verify that daxctl list
>     reports hotplug_state "inconsistent", then verify that
>     offline-memory still leaves the device unplugged with its blocks
>     removed.
> 
>   - Offline all blocks out-of-band, then verify that
>     "reconfigure-device -m devdax" succeeds and the blocks are removed.
> 
>   - With all blocks offline but the device state still reporting
>     online, verify that online-memory falls back to the per-block path
>     and brings everything online.
> 

good idea.

> For "blocks removed", please check that the device's memory blocks
> are gone from /sys/devices/system/memory.  ie verify in sysfs directly
> independent of the libdaxctl code being tested.
> 

ack

> About multi-range devices-
> 
> I found that libdaxctl undercounts memory blocks on multi-range devices.
> I'm posting a separate fix and test for that. Nothing in this series needs
> to change. I'll cc you for review.
> 

ack

> Our ndctl emulation can't exercise multi-range hotplug, but the kernel
> selftest can run against a real multi-range device. Could you add coverage
> there, including the partial-add case where the kernel reports success
> even though not all ranges were added?
> 

ack.

Thanks Alison!

-- 
~Gregory

Re: [PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Alison Schofield]

On Thu, Oct 22, 2026 at 12:46:40PM -0400, Gregory Price (Meta) wrote:
> On 2026-10-21 19:20 -0700, Alison Schofield wrote:
> > On Fri, Oct 16, 2026 at 05:42:45PM -0400, Gregory Price (Meta) wrote:
> > > +	/*
> > > +	 * Unplug atomically where supported, even if already offline, so
> > > +	 * no agent can online blocks before a later unbind.
> > > +	 */
> > > +	state = daxctl_dev_get_kmem_state(dev);
> > > +	unplug = state != DAXCTL_KMEM_UNKNOWN && state != DAXCTL_KMEM_UNPLUGGED;
> > > +
> > 
> > Hi Greg,
> > 
> > I see you called this out in the cover letter, and that the kernel is
> > deliberately combining offlining and removing memory to avoid the race
> > before unbind.
> > 
> > But, this does change the behavior of the command "daxctl offline-memory".
> > Previously it left memory blocks present but offline and now it removes
> > them entirely.
> > 
> > Should offline-memory keep its existing behavior of leaving memory blocks
> > offline, and use atomic unplug only when reconfiguring the device?
> >
> 
> The kernel explicitly does not allow `echo offline > dax/state` because
> this just creates the race we're trying to resolve.
> 
> I suppose my pushback is: why does it matter that daxctl offline-memory
> changes in this way? If daxctl supports the atomic interface, it all just
> works. It's a bit more work (additional hot-unplug) but if the memory is
> already offline that's not that expensive.
> 
> The only scenario I could come up with is having two versions of daxctl
> fight with each other (old - no awareness of dax/state, new - uses
> dax/state) but is this a real concern?

My concern isn't about 2 version of daxctl fighting. It's about
preserving the documented behavior for "daxctl offline-memory". And just
generally, if there is no need to change it, rather not change it.

From the daxctl offline-memory man page:
If all blocks were already offline, print a message and return success
without actually doing anything.

It seems the benefit of atomic unplug is w "daxctl reconfigure-device"
path where memory needs to be removed before unbinding dax_kmem. Could
we limit atomic unplug to that path?

To be clear, I don't know if that (limiting) is doable and I also don't
know if any current user would care either way. I'm just defending the
documentated behavior (FWIW).

-- Alison


> 
> > > +	if (unplug) {
> > > +		rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
> > > +		if (rc < 0 && num_on == 0) {
> > > +			/* already offline, unplug is best effort */
> > 
> > When num_on == 0, another agent could online a block between the earlier
> > check and the unplug and that memory would then be removed.
> > 
> > I see this noted in the commit message. Is that worth documenting for
> > users?
> > 
> 
> Yes that's intended, and yeah should probably update the docs.
> 
> ~Gregory
> 
> -- 
> ~Gregory
> 

Re: [PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Gregory Price (Meta)]

On 2026-10-22 15:00 -0700, Alison Schofield wrote:
> On Thu, Oct 22, 2026 at 12:46:40PM -0400, Gregory Price (Meta) wrote:
> > On 2026-10-21 19:20 -0700, Alison Schofield wrote:
> > > On Fri, Oct 16, 2026 at 05:42:45PM -0400, Gregory Price (Meta) wrote:
> > The only scenario I could come up with is having two versions of daxctl
> > fight with each other (old - no awareness of dax/state, new - uses
> > dax/state) but is this a real concern?
> 
> My concern isn't about 2 version of daxctl fighting. It's about
> preserving the documented behavior for "daxctl offline-memory". And just
> generally, if there is no need to change it, rather not change it.
> 
> From the daxctl offline-memory man page:
> If all blocks were already offline, print a message and return success
> without actually doing anything.
> 
> It seems the benefit of atomic unplug is w "daxctl reconfigure-device"
> path where memory needs to be removed before unbinding dax_kmem. Could
> we limit atomic unplug to that path?
> 
> To be clear, I don't know if that (limiting) is doable and I also don't
> know if any current user would care either way. I'm just defending the
> documentated behavior (FWIW).
> 

hm, maybe instead:

daxctl offline-memory      - stays exactly as-is
daxctl unplug-memory       - does the unplug
daxctl reconfigure-device  - converts to unplug-memory under the hood

userland that is suffering from these races can update their scripts
from offline-memory to unplug-memory, but if they're getting it from
reconfigure-device, they get it for free.

Unfortunately this is the painful result of `offline` having been the
default hotplug state when dax/kmem gets probed, instead of `unplugged`.
So we're a bit stuck with the `offline -> unplugged -> online` pattern.

The intent here was to try to get drivers creating dax/kmem to actively
set a default and effectively deprecate the global hotplug auto-online
setting (because a single auto-online setting for all devices does not
make much sense).

I intended to follow up the series with a CXL build parameter that lets
a system owner set the default, but omit "offline" and instead make them
choose between unplugged and some form of online.

I think that resolves the daxctl problem entirely more or less?

-- 
~Gregory

Re: [PATCH 3/6] daxctl: offline whole-device memory via atomic unplug when available

[Gregory Price (Meta)]

On 2026-10-22 21:48 -0400, Gregory Price (Meta) wrote:
> The intent here was to try to get drivers creating dax/kmem to actively
> set a default and effectively deprecate the global hotplug auto-online
> setting (because a single auto-online setting for all devices does not
> make much sense).
> 

there may be a path here to allow offline->online to keep userland
clean, let me propose something before i iterate this series further.

-- 
~Gregory