[PATCH v3 00/20] drbd: DRBD 9 rework

[PATCH v3 00/20] drbd: DRBD 9 rework

[Christoph Böhmwalder]

As discussed (context: [0]), here is our DRBD 9 rework series, intended
for linux-next via for-7.x/drbd.

This replays about 10-15 years of active out-of-tree development work
[1], depending on your way of counting. The out-of-tree module has
severely diverged from the in-tree version over the years, which is
what we are aiming to fix now.

Hopefully that somewhat excuses (or at least explains) the massive
diffs -- we've tried to come up with a way to group the changes by
topic, but I realize it's still not exactly trivial to review.

We've been polishing this series for a while now, and we have taken
great care to make it as "upstream-presentable" as possible. That said,
there are still probably imperfections. It's a start -- feedback
welcome!

The netlink interface has been reworked so that userspace is not
broken: the existing "drbd" generic netlink family keeps being served
at version 1, and this series does not touch its generated
serialization or its UAPI header.
Every shipped drbd-utils release therefore keeps working against this
driver. DRBD 9's own utilities speak a new family, "drbd2", written as
a YNL specification under Documentation/netlink/specs/ and following
current genetlink conventions; its generated code is checked in as
usual. The driver registers both families and translates each onto the
same wire-neutral core, so the multi-peer administration model is only
reachable through drbd2, while version-1 clients see the single-peer
semantics they expect.

The RDMA and load-balancing TCP (lb-tcp) transports are not part of
this series; they will be submitted separately once they have had the
same treatment. The rdma_ctrl_rcvbuf_size
and rdma_ctrl_sndbuf_size net options do remain in the UAPI: they are
generic transport options that are interpreted by whichever transport is
in use, so there is no reason to remove and later re-add them.

The base for these patches is for-7.5/block, which has the preparation
patches [2] this series builds on.

As discussed, the goal is to get this applied to a for-7.x/drbd branch
so it can land in linux-next via block-next. After letting it mature
in linux-next for a while, it should go in via the block tree.

Thanks,
Christoph

[0] https://lore.kernel.org/linux-next/899e0337-9642-4ca6-9050-aeab14fa22ef@kernel.dk/
[1] https://github.com/LINBIT/drbd
[2] https://lore.kernel.org/linux-block/20260923140608.1116713-1-christoph.boehmwalder@linbit.com/

v2: https://lore.kernel.org/linux-block/20261006154607.3936501-1-christoph.boehmwalder@linbit.com/

Changes since v2:
- Rebased onto for-7.5/block, now that the preparation patches are
  applied there; v2 no longer applied
- Kept the in-tree DRBD changes that went into v7.4: the explicit
  linux/umh.h include and the conversion to sk_set_nospace()
- Synced with the current out-of-tree state


Christoph Böhmwalder (20):
  drbd: mark as BROKEN during DRBD 9 rework
  drbd: extend wire protocol definitions for DRBD 9
  drbd: introduce DRBD 9 on-disk metadata format
  drbd: add transport layer abstraction
  drbd: add TCP transport implementation
  drbd: add DAX/PMEM support for metadata access
  drbd: add optional compatibility layer for DRBD 8.4
  drbd: add application/resync IO synchronization documentation
  drbd: rework sender for DRBD 9 multi-peer
  drbd: replace per-device state model with multi-peer data structures
  drbd: rewrite state machine for DRBD 9 multi-peer clusters
  drbd: rework activity log and bitmap for multi-peer replication
  drbd: rework request processing for DRBD 9 multi-peer IO
  drbd: rework module core for DRBD 9 transport and multi-peer
  drbd: rework receiver for DRBD 9 transport and multi-peer protocol
  drbd: rework the netlink core for DRBD 9
  drbd: serve the "drbd" genl family at version 1 for existing userspace
  drbd: add the "drbd2" genl family
  drbd: update monitoring interfaces for multi-peer topology
  drbd: remove BROKEN for DRBD

 .../application-resync-synchronization.rst    |   208 +
 .../admin-guide/blockdev/drbd/index.rst       |     1 +
 Documentation/netlink/specs/drbd2.yaml        |  2600 +++
 drivers/block/drbd/Kconfig                    |    43 +
 drivers/block/drbd/Makefile                   |     9 +-
 drivers/block/drbd/drbd2_nl_gen.c             |   810 +
 drivers/block/drbd/drbd2_nl_gen.h             |   113 +
 drivers/block/drbd/drbd_actlog.c              |  1373 +-
 drivers/block/drbd/drbd_bitmap.c              |  2075 +-
 drivers/block/drbd/drbd_config.h              |    45 +-
 drivers/block/drbd/drbd_dax_pmem.c            |   175 +
 drivers/block/drbd/drbd_dax_pmem.h            |    43 +
 drivers/block/drbd/drbd_debugfs.c             |  1758 +-
 drivers/block/drbd/drbd_debugfs.h             |     2 +
 drivers/block/drbd/drbd_int.h                 |  3930 ++--
 drivers/block/drbd/drbd_interval.c            |    37 +-
 drivers/block/drbd/drbd_interval.h            |   130 +-
 drivers/block/drbd/drbd_legacy_84.c           |   641 +
 drivers/block/drbd/drbd_legacy_84.h           |   116 +
 drivers/block/drbd/drbd_main.c                |  6812 ++++--
 drivers/block/drbd/drbd_meta_data.h           |   170 +
 drivers/block/drbd/drbd_nl.c                  | 10232 ++++++---
 drivers/block/drbd/drbd_nl.h                  |   400 +
 drivers/block/drbd/drbd_nl_84.c               |  3622 ++++
 drivers/block/drbd/drbd_nl_defaults.c         |   168 +
 drivers/block/drbd/drbd_nl_drbd2.c            |  2589 +++
 drivers/block/drbd/drbd_nl_types.h            |   402 +
 drivers/block/drbd/drbd_polymorph_printk.h    |    37 +-
 drivers/block/drbd/drbd_proc.c                |   327 +-
 drivers/block/drbd/drbd_protocol.h            |   614 +-
 drivers/block/drbd/drbd_receiver.c            | 17304 ++++++++++++----
 drivers/block/drbd/drbd_req.c                 |  3188 ++-
 drivers/block/drbd/drbd_req.h                 |   332 +-
 drivers/block/drbd/drbd_sender.c              |  4445 ++--
 drivers/block/drbd/drbd_state.c               |  9203 ++++++--
 drivers/block/drbd/drbd_state.h               |   334 +-
 drivers/block/drbd/drbd_state_change.h        |    80 +-
 drivers/block/drbd/drbd_strings.c             |   103 +-
 drivers/block/drbd/drbd_strings.h             |     4 +-
 drivers/block/drbd/drbd_transport.c           |   545 +
 drivers/block/drbd/drbd_transport.h           |   487 +
 drivers/block/drbd/drbd_transport_tcp.c       |  1805 ++
 drivers/block/drbd/drbd_transport_template.c  |   164 +
 include/uapi/linux/drbd.h                     |   305 +-
 include/uapi/linux/drbd2.h                    |  1038 +
 include/uapi/linux/drbd_limits.h              |   138 +-
 46 files changed, 61765 insertions(+), 17192 deletions(-)
 create mode 100644 Documentation/admin-guide/blockdev/drbd/application-resync-synchronization.rst
 create mode 100644 Documentation/netlink/specs/drbd2.yaml
 create mode 100644 drivers/block/drbd/drbd2_nl_gen.c
 create mode 100644 drivers/block/drbd/drbd2_nl_gen.h
 create mode 100644 drivers/block/drbd/drbd_dax_pmem.c
 create mode 100644 drivers/block/drbd/drbd_dax_pmem.h
 create mode 100644 drivers/block/drbd/drbd_legacy_84.c
 create mode 100644 drivers/block/drbd/drbd_legacy_84.h
 create mode 100644 drivers/block/drbd/drbd_meta_data.h
 create mode 100644 drivers/block/drbd/drbd_nl.h
 create mode 100644 drivers/block/drbd/drbd_nl_84.c
 create mode 100644 drivers/block/drbd/drbd_nl_defaults.c
 create mode 100644 drivers/block/drbd/drbd_nl_drbd2.c
 create mode 100644 drivers/block/drbd/drbd_nl_types.h
 create mode 100644 drivers/block/drbd/drbd_transport.c
 create mode 100644 drivers/block/drbd/drbd_transport.h
 create mode 100644 drivers/block/drbd/drbd_transport_tcp.c
 create mode 100644 drivers/block/drbd/drbd_transport_template.c
 create mode 100644 include/uapi/linux/drbd2.h


base-commit: aa7014f8d7565cfea117eac36a5de0a05eba65a4
-- 
2.56.0

[PATCH v3 06/20] drbd: add DAX/PMEM support for metadata access

[Christoph Böhmwalder]

When DRBD's metadata device resides on persistent memory (PMEM/NVDIMM),
accessing it by reading and writing full blocks is unnecessarily
costly.
Add a DAX-based metadata path that directly maps the metadata region,
enabling byte-granular, IRQ-safe access without having to go through
the block layer.

The PMEM path also introduces a more efficient activity log layout:
instead of writing journal transactions, the in-memory LRU-cache hash
table is stored directly in persistent memory and updated in-place.
Similarly, the resync bitmap is accessed directly from PMEM rather than
being loaded into and flushed from DRAM.

This is compiled in only when CONFIG_DEV_DAX_PMEM is enabled.

Co-developed-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Philipp Reisner <philipp.reisner@linbit.com>
Signed-off-by: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>
---
 drivers/block/drbd/Makefile        |   1 +
 drivers/block/drbd/drbd_dax_pmem.c | 175 +++++++++++++++++++++++++++++
 drivers/block/drbd/drbd_dax_pmem.h |  43 +++++++
 3 files changed, 219 insertions(+)
 create mode 100644 drivers/block/drbd/drbd_dax_pmem.c
 create mode 100644 drivers/block/drbd/drbd_dax_pmem.h

diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
index 9f05f74e619c..8ec1ad360449 100644
--- a/drivers/block/drbd/Makefile
+++ b/drivers/block/drbd/Makefile
@@ -5,6 +5,7 @@ drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
 drbd-y += drbd_interval.o drbd_state.o
 drbd-y += drbd_nl_gen.o
 drbd-y += drbd_transport.o
+drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
 drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
 
 obj-$(CONFIG_BLK_DEV_DRBD)     += drbd.o
diff --git a/drivers/block/drbd/drbd_dax_pmem.c b/drivers/block/drbd/drbd_dax_pmem.c
new file mode 100644
index 000000000000..1fb22e57cf5d
--- /dev/null
+++ b/drivers/block/drbd/drbd_dax_pmem.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2017, LINBIT HA-Solutions GmbH.
+ */
+
+/*
+ * In case DRBD's meta-data resides in persistent memory do a few things
+ * different.
+ *
+ * 1 direct access the bitmap in place. Do not load it into DRAM, do not
+ *   write it back from DRAM.
+ * 2 Use a better fitting format for the on-disk activity log. Instead of
+ *   writing transactions, the unmangled LRU-cache hash table is there.
+ */
+
+#include <linux/vmalloc.h>
+#include <linux/slab.h>
+#include <linux/dax.h>
+#include <linux/libnvdimm.h>
+#include <linux/blkdev.h>
+#include <linux/sizes.h>
+#include "drbd_int.h"
+#include "drbd_dax_pmem.h"
+#include "drbd_meta_data.h"
+
+/*
+ * Unit conversions for addressing the metadata area through DAX.
+ * DRBD keeps metadata offsets in sectors, dax_direct_access() works in pages.
+ */
+static pgoff_t md_sector_to_pgoff(sector_t sector)
+{
+	return sector >> (PAGE_SHIFT - SECTOR_SHIFT);
+}
+
+static long md_sectors_to_pages(sector_t nr_sectors)
+{
+	return nr_sectors >> (PAGE_SHIFT - SECTOR_SHIFT);
+}
+
+/* Signed, since the bitmap offset relative to the superblock may be negative. */
+static long md_sector_offset_to_bytes(long sector_offset)
+{
+	return sector_offset * SECTOR_SIZE;
+}
+
+static int map_superblock_for_dax(struct drbd_backing_dev *bdev, struct dax_device *dax_dev)
+{
+	long want = 1;
+	pgoff_t pgoff = md_sector_to_pgoff(bdev->md.md_offset);
+	void *kaddr;
+	long len;
+	int id;
+
+	id = dax_read_lock();
+	len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
+	dax_read_unlock(id);
+
+	if (len < want)
+		return -EIO;
+
+	bdev->md_on_pmem = kaddr;
+
+	return 0;
+}
+
+/**
+ * drbd_dax_open() - Open device for dax and map metadata superblock
+ * @bdev: backing device to be opened
+ */
+int drbd_dax_open(struct drbd_backing_dev *bdev)
+{
+	struct dax_device *dax_dev;
+	int err;
+	u64 part_off;
+
+	dax_dev = fs_dax_get_by_bdev(bdev->md_bdev, &part_off, NULL, NULL);
+	if (!dax_dev)
+		return -ENODEV;
+
+	err = map_superblock_for_dax(bdev, dax_dev);
+	if (!err)
+		bdev->dax_dev = dax_dev;
+	else
+		put_dax(dax_dev);
+
+	return err;
+}
+
+void drbd_dax_close(struct drbd_backing_dev *bdev)
+{
+	put_dax(bdev->dax_dev);
+}
+
+/**
+ * drbd_dax_map() - Map metadata for dax
+ * @bdev: backing device whose metadata is to be mapped
+ */
+int drbd_dax_map(struct drbd_backing_dev *bdev)
+{
+	struct dax_device *dax_dev = bdev->dax_dev;
+	sector_t first_sector = drbd_md_first_sector(bdev);
+	sector_t al_sector = bdev->md.md_offset + bdev->md.al_offset;
+	long want = md_sectors_to_pages(drbd_md_last_sector(bdev) + 1 - first_sector);
+	pgoff_t pgoff = md_sector_to_pgoff(first_sector);
+	long md_offset_byte = md_sector_offset_to_bytes(bdev->md.md_offset - first_sector);
+	long al_offset_byte = md_sector_offset_to_bytes(al_sector - first_sector);
+	void *kaddr;
+	long len;
+	int id;
+
+	id = dax_read_lock();
+	len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
+	dax_read_unlock(id);
+
+	if (len < want)
+		return -EIO;
+
+	bdev->md_on_pmem = kaddr + md_offset_byte;
+	bdev->al_on_pmem = kaddr + al_offset_byte;
+
+	return 0;
+}
+
+void drbd_dax_al_update(struct drbd_device *device, struct lc_element *al_ext)
+{
+	struct al_on_pmem *al_on_pmem = device->ldev->al_on_pmem;
+	__be32 *slot = &al_on_pmem->slots[al_ext->lc_index];
+
+	*slot = cpu_to_be32(al_ext->lc_new_number);
+	arch_wb_cache_pmem(slot, sizeof(*slot));
+}
+
+void drbd_dax_al_begin_io_commit(struct drbd_device *device)
+{
+	struct lc_element *e;
+
+	spin_lock_irq(&device->al_lock);
+
+	list_for_each_entry(e, &device->act_log->to_be_changed, list)
+		drbd_dax_al_update(device, e);
+
+	lc_committed(device->act_log);
+
+	spin_unlock_irq(&device->al_lock);
+}
+
+int drbd_dax_al_initialize(struct drbd_device *device)
+{
+	struct al_on_pmem *al_on_pmem = device->ldev->al_on_pmem;
+	__be32 *slots = al_on_pmem->slots;
+	/* The activity log area holds the magic followed by the slots. */
+	int i, al_slots = device->ldev->md.al_size_4k * (SZ_4K / sizeof(*slots)) - 1;
+
+	al_on_pmem->magic = cpu_to_be32(DRBD_AL_PMEM_MAGIC);
+	/*
+	 * Initialize all slots rather than just the configured number in case
+	 * the configuration is later changed.
+	 */
+	for (i = 0; i < al_slots; i++) {
+		unsigned int extent_nr = i < device->act_log->nr_elements ?
+			lc_element_by_index(device->act_log, i)->lc_number :
+			LC_FREE;
+		slots[i] = cpu_to_be32(extent_nr);
+	}
+
+	return 0;
+}
+
+void *drbd_dax_bitmap(struct drbd_device *device, unsigned long want)
+{
+	struct drbd_backing_dev *bdev = device->ldev;
+	unsigned char *md_on_pmem = (unsigned char *)bdev->md_on_pmem;
+
+	return md_on_pmem + md_sector_offset_to_bytes(bdev->md.bm_offset);
+}
diff --git a/drivers/block/drbd/drbd_dax_pmem.h b/drivers/block/drbd/drbd_dax_pmem.h
new file mode 100644
index 000000000000..52b116300122
--- /dev/null
+++ b/drivers/block/drbd/drbd_dax_pmem.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017, LINBIT HA-Solutions GmbH.
+ */
+#ifndef DRBD_DAX_H
+#define DRBD_DAX_H
+
+#include <linux/kconfig.h>
+
+#if IS_ENABLED(CONFIG_DEV_DAX_PMEM)
+
+int drbd_dax_open(struct drbd_backing_dev *bdev);
+void drbd_dax_close(struct drbd_backing_dev *bdev);
+int drbd_dax_map(struct drbd_backing_dev *bdev);
+void drbd_dax_al_update(struct drbd_device *device, struct lc_element *al_ext);
+void drbd_dax_al_begin_io_commit(struct drbd_device *device);
+int drbd_dax_al_initialize(struct drbd_device *device);
+void *drbd_dax_bitmap(struct drbd_device *device, unsigned long want);
+
+static inline bool drbd_md_dax_active(struct drbd_backing_dev *bdev)
+{
+	return bdev->dax_dev != NULL;
+}
+static inline struct meta_data_on_disk_9 *drbd_dax_md_addr(struct drbd_backing_dev *bdev)
+{
+	return bdev->md_on_pmem;
+}
+#else
+
+#define drbd_dax_open(B) do { } while (0)
+#define drbd_dax_close(B) do { } while (0)
+#define drbd_dax_map(B) (-ENOTSUPP)
+#define drbd_dax_al_begin_io_commit(D) do { } while (0)
+#define drbd_dax_al_initialize(D) (-EIO)
+#define drbd_dax_bitmap(D, L) (NULL)
+#define drbd_md_dax_active(B) (false)
+#define drbd_dax_md_addr(B) (NULL)
+
+#define arch_wb_cache_pmem(A, L) do { } while (0)
+
+#endif /* IS_ENABLED(CONFIG_DEV_DAX_PMEM) */
+
+#endif /* DRBD_DAX_H */
-- 
2.56.0

Re: [PATCH v3 06/20] drbd: add DAX/PMEM support for metadata access

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 7 potential
issues to consider:
- [High] drbd: partition offset ignored in drbd_dax_open() and
  drbd_dax_map()
- [High] drbd: sub-page offset and page-count truncation when PAGE_SIZE
  > 4096
- [High] drbd: missing cache writeback and ordering in
  drbd_dax_al_initialize()
- [High] drbd: __LC_DIRTY never cleared in drbd_dax_al_begin_io_commit()
- [High] drbd: link failure and wrong Kconfig dependency on
  CONFIG_DEV_DAX_PMEM
- [Medium] drbd: signed 32-bit overflow of al_slots in
  drbd_dax_al_initialize()
- [Medium] drbd: uncommitted and unlocked act_log read in
  drbd_dax_al_initialize()
--

commit 78fd73a53d10bb7aa94cbf5a0d1278098930c4df
Author: Christoph Böhmwalder <christoph.boehmwalder@linbit.com>

drbd: add DAX/PMEM support for metadata access

This commit adds DAX/PMEM support for DRBD metadata access, allowing the
superblock, activity log, and resync bitmap to be mapped and updated
directly in persistent memory.

> diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile
> index 9f05f74e619c1..8ec1ad3604499 100644
> --- a/drivers/block/drbd/Makefile
> +++ b/drivers/block/drbd/Makefile
> @@ -5,6 +5,7 @@ drbd-y += drbd_main.o drbd_strings.o drbd_nl.o
>  drbd-y += drbd_interval.o drbd_state.o
>  drbd-y += drbd_nl_gen.o
>  drbd-y += drbd_transport.o
> +drbd-$(CONFIG_DEV_DAX_PMEM) += drbd_dax_pmem.o
>  drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o
>  
>  obj-$(CONFIG_BLK_DEV_DRBD)     += drbd.o

[Severity: High]
Does the build fail to link when CONFIG_BLK_DEV_DRBD=y and
CONFIG_DEV_DAX_PMEM=m?  In that configuration, this line expands to
drbd-m += drbd_dax_pmem.o, which Kbuild ignores when linking built-in
drbd.o, while IS_ENABLED(CONFIG_DEV_DAX_PMEM) in drbd_dax_pmem.h still
evaluates to true and declares drbd_dax_*() as external functions instead
of stubs.

In addition, drbd_dax_open() looks up the DAX device via
fs_dax_get_by_bdev(), which is gated on CONFIG_FS_DAX rather than
CONFIG_DEV_DAX_PMEM (the /dev/daxX.Y character device driver), so
drbd_dax_open() always returns -ENODEV if CONFIG_FS_DAX is disabled.

> diff --git a/drivers/block/drbd/drbd_dax_pmem.c b/drivers/block/drbd/drbd_dax_pmem.c
> new file mode 100644
> index 0000000000000..1fb22e57cf5d0
> --- /dev/null
> +++ b/drivers/block/drbd/drbd_dax_pmem.c
[ ... ]
> +static pgoff_t md_sector_to_pgoff(sector_t sector)
> +{
> +	return sector >> (PAGE_SHIFT - SECTOR_SHIFT);
> +}
> +
> +static long md_sectors_to_pages(sector_t nr_sectors)
> +{
> +	return nr_sectors >> (PAGE_SHIFT - SECTOR_SHIFT);
> +}
> +
> +/* Signed, since the bitmap offset relative to the superblock may be negative. */
> +static long md_sector_offset_to_bytes(long sector_offset)
> +{
> +	return sector_offset * SECTOR_SIZE;
> +}
> +
> +static int map_superblock_for_dax(struct drbd_backing_dev *bdev, struct dax_device *dax_dev)
> +{
> +	long want = 1;
> +	pgoff_t pgoff = md_sector_to_pgoff(bdev->md.md_offset);
> +	void *kaddr;
> +	long len;
> +	int id;
> +
> +	id = dax_read_lock();
> +	len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
> +	dax_read_unlock(id);
> +
> +	if (len < want)
> +		return -EIO;
> +
> +	bdev->md_on_pmem = kaddr;
> +
> +	return 0;
> +}
> +
> +/**
> + * drbd_dax_open() - Open device for dax and map metadata superblock
> + * @bdev: backing device to be opened
> + */
> +int drbd_dax_open(struct drbd_backing_dev *bdev)
> +{
> +	struct dax_device *dax_dev;
> +	int err;
> +	u64 part_off;
> +
> +	dax_dev = fs_dax_get_by_bdev(bdev->md_bdev, &part_off, NULL, NULL);
> +	if (!dax_dev)
> +		return -ENODEV;
> +
> +	err = map_superblock_for_dax(bdev, dax_dev);
> +	if (!err)
> +		bdev->dax_dev = dax_dev;
> +	else
> +		put_dax(dax_dev);
> +
> +	return err;
> +}
> +
> +void drbd_dax_close(struct drbd_backing_dev *bdev)
> +{
> +	put_dax(bdev->dax_dev);
> +}
> +
> +/**
> + * drbd_dax_map() - Map metadata for dax
> + * @bdev: backing device whose metadata is to be mapped
> + */
> +int drbd_dax_map(struct drbd_backing_dev *bdev)
> +{
> +	struct dax_device *dax_dev = bdev->dax_dev;
> +	sector_t first_sector = drbd_md_first_sector(bdev);
> +	sector_t al_sector = bdev->md.md_offset + bdev->md.al_offset;
> +	long want = md_sectors_to_pages(drbd_md_last_sector(bdev) + 1 - first_sector);
> +	pgoff_t pgoff = md_sector_to_pgoff(first_sector);
> +	long md_offset_byte = md_sector_offset_to_bytes(bdev->md.md_offset - first_sector);
> +	long al_offset_byte = md_sector_offset_to_bytes(al_sector - first_sector);
> +	void *kaddr;
> +	long len;
> +	int id;
> +
> +	id = dax_read_lock();
> +	len = dax_direct_access(dax_dev, pgoff, want, DAX_ACCESS, &kaddr, NULL);
> +	dax_read_unlock(id);
> +
> +	if (len < want)
> +		return -EIO;
> +
> +	bdev->md_on_pmem = kaddr + md_offset_byte;
> +	bdev->al_on_pmem = kaddr + al_offset_byte;
> +
> +	return 0;
> +}

[Severity: High]
Does discarding part_off in drbd_dax_open() cause DRBD to read and write
metadata at the wrong offset when bdev->md_bdev is a partition?

fs_dax_get_by_bdev() returns the dax_device for the whole disk and writes
the partition's starting byte offset into part_off.  Because
map_superblock_for_dax() and drbd_dax_map() pass partition-relative
pgoff values to dax_direct_access() without adding part_off, metadata
accesses hit the beginning of the parent PMEM disk instead of the
partition, corrupting the partition table or earlier partitions.

[Severity: High]
Can md_sector_to_pgoff() and md_sectors_to_pages() misalign pointers and
truncate the mapping when PAGE_SIZE > 4096 (such as on arm64 or powerpc
with 64 KiB pages)?

DRBD metadata offsets and sizes are 4 KiB-aligned, not PAGE_SIZE-aligned;
for example, internal metadata places the 4 KiB superblock in the last 8
sectors of the device.  In map_superblock_for_dax(), md_sector_to_pgoff()
shifts off the sub-page bits of bdev->md.md_offset, so kaddr points to the
start of the page rather than the superblock.  In drbd_dax_map(), kaddr
also starts at the page boundary pgoff << PAGE_SHIFT, but md_offset_byte
and al_offset_byte are calculated relative to first_sector instead of the
start of pgoff, and md_sectors_to_pages() right-shifts the sector count
down (yielding want == 0 if the metadata region is smaller than a page).

[ ... ]

> +void drbd_dax_al_begin_io_commit(struct drbd_device *device)
> +{
> +	struct lc_element *e;
> +
> +	spin_lock_irq(&device->al_lock);
> +
> +	list_for_each_entry(e, &device->act_log->to_be_changed, list)
> +		drbd_dax_al_update(device, e);
> +
> +	lc_committed(device->act_log);
> +
> +	spin_unlock_irq(&device->al_lock);
> +}

[Severity: High]
Does leaving __LC_DIRTY set in device->act_log->flags after
drbd_dax_al_begin_io_commit() cause later activity log locking to hang?

When an activity log lookup misses, __lc_get() sets __LC_DIRTY in
lc->flags.  lc_committed() clears pending_changes, but __LC_DIRTY is only
cleared by lc_unlock().  Unlike drbd_al_begin_io_commit(), this function
does not acquire drbd_al_try_lock_for_transaction(), call
lc_unlock(device->act_log), or wake device->al_wait.  Once __LC_DIRTY is
set, lc_try_lock(device->act_log) (which requires lc->flags == 0) always
fails, so check_al_extents() hangs forever in wait_event(device->al_wait,
drbd_al_try_lock(device)) and drbd_try_suspend_al() always fails.

> +int drbd_dax_al_initialize(struct drbd_device *device)
> +{
> +	struct al_on_pmem *al_on_pmem = device->ldev->al_on_pmem;
> +	__be32 *slots = al_on_pmem->slots;
> +	/* The activity log area holds the magic followed by the slots. */
> +	int i, al_slots = device->ldev->md.al_size_4k * (SZ_4K / sizeof(*slots)) - 1;
> +
> +	al_on_pmem->magic = cpu_to_be32(DRBD_AL_PMEM_MAGIC);
> +	/*
> +	 * Initialize all slots rather than just the configured number in case
> +	 * the configuration is later changed.
> +	 */
> +	for (i = 0; i < al_slots; i++) {
> +		unsigned int extent_nr = i < device->act_log->nr_elements ?
> +			lc_element_by_index(device->act_log, i)->lc_number :
> +			LC_FREE;
> +		slots[i] = cpu_to_be32(extent_nr);
> +	}
> +
> +	return 0;
> +}

[Severity: High]
Does drbd_dax_al_initialize() need to flush the initialized activity log
with arch_wb_cache_pmem() and write al_on_pmem->magic only after slots[]
are persisted?

Unlike drbd_dax_al_update(), drbd_dax_al_initialize() never calls
arch_wb_cache_pmem(), and it stores DRBD_AL_PMEM_MAGIC before populating
slots[].  If a crash or power failure occurs before the slot cache lines
are evicted from volatile CPU caches, the activity log on PMEM can have a
valid magic header with uninitialized or stale slots.

[Severity: Medium]
Can al_slots overflow a signed 32-bit int when al_size_4k is larger than
8 GiB?

check_activity_log_stripe_size() allows al_size_4k up to 4,194,304
(16 GiB).  Multiplying by SZ_4K / sizeof(*slots) (1024) produces up to
0xffffffff, which becomes negative when stored in int al_slots and causes
the loop to be skipped without initializing any slots.

[Severity: Medium]
Can drbd_dax_al_initialize() write stale extent numbers to PMEM when
device->act_log->to_be_changed has uncommitted elements?

Unlike the non-DAX drbd_al_initialize() path, which holds device->al_lock,
writes lc_new_number for entries on to_be_changed, and calls
lc_committed(device->act_log), drbd_dax_al_initialize() reads lc_number
without holding device->al_lock or committing to_be_changed.  Any pending
element on to_be_changed has its old lc_number written to PMEM and remains
on to_be_changed after the caller unlocks device->act_log.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261110162646.3028472-1-christoph.boehmwalder@linbit.com?part=6