[PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Brian Bunker]

During an array controller failover on an iSCSI host, multipathd
briefly put a path to one LUN into the map of another LUN, and I/O
meant for the first LUN landed on the second. See the v1 cover letter
for the full sequence.

Changes in v3, after Martin's review:

- No SG_IO. The re-add check now compares the udev WWID with one built
  from the VPD page 0x83 data the kernel caches in sysfs, so it does no
  I/O on the device and can't hang. The new get_sysfs_wwid() in
  libmultipath shares get_vpd_uid() and get_uid()'s WWID clean-up with
  the existing sysfs fallback, so both produce the same string.

- The uev_update_path() check that ignored a changed udev WWID is gone.
  With cached data it would be dangerous: if a LUN is really remapped
  and the kernel hasn't rescanned yet, sysfs still shows the old WWID,
  and ignoring the change would keep a path to one LUN in another LUN's
  map. Removing a path from its map is always safe; the add is the step
  that needs checking.

- check_path_wwid_change() is back to its upstream form.

Unchanged from v2: the check runs only on PATH_ADD_REINIT, i.e. only
for the re-add at the end of uev_update_path(), and only for paths
where can_recheck_wwid() is true. Ordinary adds never run it.

On mixing WWID sources: Martin is right that multipath-tools should
not use different methods in different places. But on RHEL 9 the udev
WWID for sd devices doesn't come from sysfs. The RHEL (and Fedora)
sg3_utils package installs upstream's 55-scsi-sg3_id.rules as
61-scsi-sg3_id.rules, with the comment "need to run after
60-persistent-storage.rules". So 60-persistent-storage.rules runs
first, and its "scsi_id --export --whitelisted -d $devnode" sets
ID_SERIAL; the sysfs-based sg3 rules then leave ID_SERIAL alone.
scsi_id sends INQUIRY over SG_IO (strace on the host shows
ioctl(SG_IO) for the standard INQUIRY and EVPD pages 0x00, 0x80 and
0x83 on every run). On the host where we hit this, the bad WWID came
from that SG_IO INQUIRY inside udev, not from cached kernel data.

Where sg3_utils' rules run first, as on openSUSE, which installs them
as 55-scsi-sg3_id.rules, ID_SERIAL is built from the same sysfs page
this check reads. With everything at defaults, which
can_recheck_wwid() requires, the designator selection is the same. On
that RHEL 9 host, ID_SERIAL equals "3" + the NAA designator decoded
from sysfs vpd_pg83 on all 18 Pure paths. So the check doesn't
introduce another way of computing the WWID. It compares udev's value
with the data udev is meant to derive it from.

What happens on a mismatch: the path is left out of its map and udev
is retriggered, up to retrigger_tries times. If udev and sysfs still
disagree after that, the path goes to INIT_FAILED, and the checker
later re-adds it through the normal path, which doesn't run the check.
So a persistent disagreement, for example a device where udev rules
pick a different designator, only delays the add (by roughly
retrigger_tries * retrigger_delay) and doesn't lose the path. A
transient bad value like ours is caught, because the retriggered udev
event returns the right WWID.

In our case, the kernel's cached page 0x83 for the path came from the
original scan, so unless the kernel rescanned the device during the
failover (we have no sign that it did, but didn't capture sysfs at the
time), it still held the correct WWID when the path was re-added, and
v3 would have refused the add.


Brian Bunker (1):
  multipathd: check re-added paths' WWIDs against sysfs

 libmultipath/discovery.c          | 42 ++++++++++++++++++++++---------
 libmultipath/discovery.h          |  1 +
 libmultipath/libmultipath.version |  5 ++++
 multipathd/cli_handlers.c         |  2 +-
 multipathd/main.c                 | 39 ++++++++++++++++++++--------
 multipathd/main.h                 | 12 ++++++++-
 6 files changed, 76 insertions(+), 25 deletions(-)

-- 
2.56.0

[PATCH v3 1/1] multipathd: check re-added paths' WWIDs against sysfs

[Brian Bunker]

multipathd takes a path's WWID from udev. When a change uevent reports
a different WWID, uev_update_path() removes the path from its map and
re-adds it through uev_add_path(), which places it in whatever map owns
the new WWID. If the udev value is wrong, or the uevent is stale by the
time it is processed, the path goes into the map of a different LUN and
I/O for one LUN is silently sent to another.

Before such a re-add puts the path into a map, compare its WWID with
one built from the VPD page 0x83 data the kernel caches in sysfs. If
they differ, leave the path out of the map: orphan it, clear its WWID
and mark it INIT_MISSING_UDEV, so the checker retriggers udev and the
path is re-added from fresh data. This reads no data from the device,
so it can't hang. The check only runs for paths where
can_recheck_wwid() is true, where the udev WWID and the one built from
sysfs come from the same designator selection.

To tell re-adds from ordinary adds, turn the need_do_map argument of
uev_add_path() and ev_add_path() into enum path_add_mode, with a new
PATH_ADD_REINIT value that behaves like PATH_ADD_MAP and is passed only
for the re-add at the end of uev_update_path(). Ordinary adds are
unchanged.

Add get_sysfs_wwid() to libmultipath. It shares get_vpd_uid() and the
WWID clean-up in get_uid() with the existing sysfs fallback, so both
build the WWID the same way.

Signed-off-by: Brian Bunker <brian@purestorage.com>
---
 libmultipath/discovery.c          | 42 ++++++++++++++++++++++---------
 libmultipath/discovery.h          |  1 +
 libmultipath/libmultipath.version |  5 ++++
 multipathd/cli_handlers.c         |  2 +-
 multipathd/main.c                 | 39 ++++++++++++++++++++--------
 multipathd/main.h                 | 12 ++++++++-
 6 files changed, 76 insertions(+), 25 deletions(-)

diff --git a/libmultipath/discovery.c b/libmultipath/discovery.c
index 344da19c..62286928 100644
--- a/libmultipath/discovery.c
+++ b/libmultipath/discovery.c
@@ -2180,8 +2180,7 @@ get_udev_uid(struct path * pp, const char *uid_attribute, struct udev_device *ud
 	return len;
 }
 
-static int
-get_vpd_uid(struct path * pp)
+static int get_vpd_uid(const struct path *pp, char *wwid, int len)
 {
 	struct udev_device *parent = pp->udev;
 
@@ -2195,7 +2194,7 @@ get_vpd_uid(struct path * pp)
 	if (!parent)
 		return -EINVAL;
 
-	return get_vpd_sysfs(parent, 0x83, pp->wwid, WWID_SIZE);
+	return get_vpd_sysfs(parent, 0x83, wwid, len);
 }
 
 /* based on code from s390-tools/dasdinfo/dasdinfo.c */
@@ -2236,7 +2235,7 @@ static ssize_t uid_fallback(struct path *pp, int path_state,
 		len = dasd_get_uid(pp);
 		*origin = "sysfs";
 	} else if (pp->bus == SYSFS_BUS_SCSI) {
-		len = get_vpd_uid(pp);
+		len = get_vpd_uid(pp, pp->wwid, WWID_SIZE);
 		*origin = "sysfs";
 		if (len < 0 && path_state == PATH_UP) {
 			condlog(1, "%s: failed to get sysfs uid: %s",
@@ -2299,6 +2298,17 @@ bool can_recheck_wwid(const struct path *pp)
 	return (pp->bus == SYSFS_BUS_SCSI && has_uid_fallback(pp));
 }
 
+static void tidy_wwid(char *wwid)
+{
+	size_t i;
+
+	strchop(wwid);
+	for (i = 0; wwid[i] != '\0'; i++) {
+		if (wwid[i] == '/')
+			wwid[i] = '_';
+	}
+}
+
 int
 get_uid (struct path * pp, int path_state, struct udev_device *udev,
 	 int allow_fallback)
@@ -2307,7 +2317,6 @@ get_uid (struct path * pp, int path_state, struct udev_device *udev,
 	ssize_t len = 0;
 	struct config *conf;
 	int used_fallback = 0;
-	size_t i;
 
 	if (!pp->uid_attribute) {
 		conf = get_multipath_config();
@@ -2347,18 +2356,27 @@ get_uid (struct path * pp, int path_state, struct udev_device *udev,
 			pp->wwid);
 		memset(pp->wwid, 0x0, WWID_SIZE);
 		return 1;
-	} else {
-		strchop(pp->wwid);
-		for (i = 0; pp->wwid[i] != '\0'; i++) {
-			if (pp->wwid[i] == '/')
-				pp->wwid[i] = '_';
-		}
-	}
+	} else
+		tidy_wwid(pp->wwid);
 	condlog((used_fallback)? 1 : 3, "%s: uid = %s (%s)", pp->dev,
 		*pp->wwid == '\0' ? "<empty>" : pp->wwid, origin);
 	return 0;
 }
 
+/*
+ * Get the WWID of a SCSI path from the VPD page 0x83 data the kernel
+ * caches in sysfs, built the same way as get_uid() does when it falls
+ * back to sysfs. This does no I/O on the device. It only matches the
+ * udev WWID if can_recheck_wwid(pp) is true.
+ */
+bool get_sysfs_wwid(const struct path *pp, char *wwid)
+{
+	if (pp->bus != SYSFS_BUS_SCSI || get_vpd_uid(pp, wwid, WWID_SIZE) <= 0)
+		return false;
+	tidy_wwid(wwid);
+	return true;
+}
+
 int pathinfo(struct path *pp, struct config *conf, int mask)
 {
 	int path_state;
diff --git a/libmultipath/discovery.h b/libmultipath/discovery.h
index 2ef1d94b..e433a6dd 100644
--- a/libmultipath/discovery.h
+++ b/libmultipath/discovery.h
@@ -56,6 +56,7 @@ ssize_t sysfs_get_inquiry(struct udev_device *udev,
 int sysfs_get_asymmetric_access_state(struct path *pp,
 				      char *buff, int buflen);
 bool can_recheck_wwid(const struct path *pp);
+bool get_sysfs_wwid(const struct path *pp, char *wwid);
 int get_uid(struct path * pp, int path_state, struct udev_device *udev,
 	    int allow_fallback);
 bool is_vpd_page_supported(int fd, int pg);
diff --git a/libmultipath/libmultipath.version b/libmultipath/libmultipath.version
index f631f453..4f8e1fd5 100644
--- a/libmultipath/libmultipath.version
+++ b/libmultipath/libmultipath.version
@@ -246,3 +246,8 @@ local:
 	*;
 };
 
+LIBMULTIPATH_34.1.0 {
+global:
+	get_sysfs_wwid;
+} LIBMULTIPATH_34.0.0;
+
diff --git a/multipathd/cli_handlers.c b/multipathd/cli_handlers.c
index f5a13c46..9b4a04c2 100644
--- a/multipathd/cli_handlers.c
+++ b/multipathd/cli_handlers.c
@@ -673,7 +673,7 @@ cli_add_path (void *v, struct strbuf *reply, void *data)
 			return 1;
 		}
 	}
-	return ev_add_path(pp, vecs, 1);
+	return ev_add_path(pp, vecs, PATH_ADD_MAP);
 blacklisted:
 	append_strbuf_str(reply, "blacklisted\n");
 	condlog(2, "%s: path blacklisted", param);
diff --git a/multipathd/main.c b/multipathd/main.c
index 9bce4d6b..45ba6aff 100644
--- a/multipathd/main.c
+++ b/multipathd/main.c
@@ -1128,7 +1128,7 @@ check_path_wwid_change(struct path *pp)
 static int uev_update_path (struct uevent *uev, struct vectors * vecs);
 
 static int
-uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
+uev_add_path(struct uevent *uev, struct vectors *vecs, enum path_add_mode mode)
 {
 	struct path *pp;
 	int ret = 0, i;
@@ -1180,7 +1180,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
 				     DI_ALL | DI_BLACKLIST);
 			pthread_cleanup_pop(1);
 			if (r == PATHINFO_OK && !prev_mpp)
-				ret = ev_add_path(pp, vecs, need_do_map);
+				ret = ev_add_path(pp, vecs, mode);
 			else if (r == PATHINFO_OK &&
 				 !strncmp(pp->wwid, prev_mpp->wwid, WWID_SIZE)) {
 				/*
@@ -1255,7 +1255,7 @@ uev_add_path (struct uevent *uev, struct vectors * vecs, int need_do_map)
 		conf = get_multipath_config();
 		pp->checkint = conf->checkint;
 		put_multipath_config(conf);
-		ret = ev_add_path(pp, vecs, need_do_map);
+		ret = ev_add_path(pp, vecs, mode);
 	} else {
 		condlog(0, "%s: failed to store path info, "
 			"dropping event",
@@ -1297,10 +1297,10 @@ sysfs_get_ro (struct path *pp)
  * 0: added
  * 1: error
  */
-int
-ev_add_path (struct path * pp, struct vectors * vecs, int need_do_map)
+int ev_add_path(struct path *pp, struct vectors *vecs, enum path_add_mode mode)
 {
 	struct multipath * mpp;
+	char sysfs_wwid[WWID_SIZE];
 	char *params __attribute__((cleanup(cleanup_charp))) = NULL;
 	int retries = 3;
 	int start_waiter = 0;
@@ -1315,6 +1315,23 @@ ev_add_path (struct path * pp, struct vectors * vecs, int need_do_map)
 		condlog(0, "%s: failed to get path uid", pp->dev);
 		goto fail; /* leave path added to pathvec */
 	}
+	/*
+	 * This path is being re-added because its udev data was missing or
+	 * changed. The uevent may have been queued long before we got here,
+	 * and adding a path to the map of a different LUN corrupts data, so
+	 * check the WWID against the VPD data the kernel has cached in
+	 * sysfs. On a mismatch, leave the path out and let the checker
+	 * retrigger udev to get a fresh WWID.
+	 */
+	if (mode == PATH_ADD_REINIT && can_recheck_wwid(pp) &&
+	    get_sysfs_wwid(pp, sysfs_wwid) &&
+	    strncmp(sysfs_wwid, pp->wwid, WWID_SIZE)) {
+		condlog(0, "%s: not adding path, wwid '%s' from udev doesn't match '%s' from sysfs",
+			pp->dev, pp->wwid, sysfs_wwid);
+		memset(pp->wwid, 0, WWID_SIZE);
+		pp->initialized = INIT_MISSING_UDEV;
+		goto fail;
+	}
 	mpp = find_mp_by_wwid(vecs->mpvec, pp->wwid);
 	if (mpp && pp->size && mpp->size != pp->size) {
 		condlog(0, "%s: failed to add new path %s, device size mismatch", mpp->alias, pp->dev);
@@ -1375,7 +1392,7 @@ rescan:
 	if (ro == 1)
 		mpp->force_readonly = 1;
 
-	if (!need_do_map)
+	if (mode == PATH_ADD_NO_MAP)
 		return 0;
 
 	if (!dm_map_present(mpp->alias)) {
@@ -1789,7 +1806,7 @@ out:
 	}
 	/* pp->initialized must not be INIT_PARTIAL if needs_reinit is set */
 	if (needs_reinit)
-		retval = uev_add_path(uev, vecs, 1);
+		retval = uev_add_path(uev, vecs, PATH_ADD_REINIT);
 	return retval;
 }
 
@@ -1905,13 +1922,13 @@ uev_trigger (struct uevent * uev, void * trigger_data)
 	 */
 	list_for_each_entry_safe(merge_uev, tmp, &uev->merge_node, node) {
 		if (!strncmp(merge_uev->action, "add", 3))
-			r += uev_add_path(merge_uev, vecs, 0);
+			r += uev_add_path(merge_uev, vecs, PATH_ADD_NO_MAP);
 		if (!strncmp(merge_uev->action, "remove", 6))
 			r += uev_remove_path(merge_uev, vecs, 0);
 	}
 
 	if (!strncmp(uev->action, "add", 3))
-		r += uev_add_path(uev, vecs, 1);
+		r += uev_add_path(uev, vecs, PATH_ADD_MAP);
 	if (!strncmp(uev->action, "remove", 6))
 		r += uev_remove_path(uev, vecs, 1);
 	if (!strncmp(uev->action, "change", 6))
@@ -3015,7 +3032,7 @@ update_uninitialized_path(struct vectors * vecs, struct path * pp)
 		pthread_cleanup_pop(1);
 		/* INIT_OK implies ret == PATHINFO_OK */
 		if (pp->initialized == INIT_OK) {
-			ev_add_path(pp, vecs, 1);
+			ev_add_path(pp, vecs, PATH_ADD_MAP);
 			pp->tick = 1;
 		} else if (ret == PATHINFO_SKIPPED) {
 			int i;
@@ -3036,7 +3053,7 @@ update_uninitialized_path(struct vectors * vecs, struct path * pp)
 					CHECK_PATH_REMOVED :
 					CHECK_PATH_SKIPPED;
 		}
-		ev_add_path(pp, vecs, 1);
+		ev_add_path(pp, vecs, PATH_ADD_MAP);
 		pp->tick = 1;
 	}
 	return CHECK_PATH_CHECKED;
diff --git a/multipathd/main.h b/multipathd/main.h
index b66208ec..d3826a89 100644
--- a/multipathd/main.h
+++ b/multipathd/main.h
@@ -36,7 +36,17 @@ enum daemon_status wait_for_state_change_if(enum daemon_status oldstate,
 					    unsigned long ms);
 void schedule_reconfigure(enum force_reload_types requested_type);
 int need_to_delay_reconfig (struct vectors *);
-int ev_add_path (struct path *, struct vectors *, int);
+/*
+ * PATH_ADD_REINIT is PATH_ADD_MAP for a path that is being re-added
+ * because its udev data was missing or changed. Its WWID is checked
+ * against sysfs before the path is put into a map.
+ */
+enum path_add_mode {
+	PATH_ADD_NO_MAP,
+	PATH_ADD_MAP,
+	PATH_ADD_REINIT,
+};
+int ev_add_path(struct path *, struct vectors *, enum path_add_mode);
 int ev_remove_path (struct path *, struct vectors *, int);
 int ev_add_map (char *, const char *, struct vectors *);
 int flush_map(struct multipath *, struct vectors *);
-- 
2.56.0

Re: [PATCH v3 1/1] multipathd: check re-added paths' WWIDs against sysfs

[Benjamin Marzinski]

On Mon, Nov 09, 2026 at 10:55:55AM -0800, Brian Bunker wrote:
> multipathd takes a path's WWID from udev. When a change uevent reports
> a different WWID, uev_update_path() removes the path from its map and
> re-adds it through uev_add_path(), which places it in whatever map owns
> the new WWID. If the udev value is wrong, or the uevent is stale by the
> time it is processed, the path goes into the map of a different LUN and
> I/O for one LUN is silently sent to another.
> 
> Before such a re-add puts the path into a map, compare its WWID with
> one built from the VPD page 0x83 data the kernel caches in sysfs. If
> they differ, leave the path out of the map: orphan it, clear its WWID
> and mark it INIT_MISSING_UDEV, so the checker retriggers udev and the
> path is re-added from fresh data. This reads no data from the device,
> so it can't hang. The check only runs for paths where
> can_recheck_wwid() is true, where the udev WWID and the one built from
> sysfs come from the same designator selection.
> 
> To tell re-adds from ordinary adds, turn the need_do_map argument of
> uev_add_path() and ev_add_path() into enum path_add_mode, with a new
> PATH_ADD_REINIT value that behaves like PATH_ADD_MAP and is passed only
> for the re-add at the end of uev_update_path(). Ordinary adds are
> unchanged.
> 
> Add get_sysfs_wwid() to libmultipath. It shares get_vpd_uid() and the
> WWID clean-up in get_uid() with the existing sysfs fallback, so both
> build the WWID the same way.
> 
> Signed-off-by: Brian Bunker <brian@purestorage.com>

I'm interested in seeing if this resolves Martin's issues, but as
far as I'm concerned:

Reviewed-by: Benjamin Marzinski <bmarzins@redhat.com>

Re: [PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Martin Wilck]

On Mon, 2026-11-09 at 10:55 -0800, Brian Bunker wrote:
> During an array controller failover on an iSCSI host, multipathd
> briefly put a path to one LUN into the map of another LUN, and I/O
> meant for the first LUN landed on the second. See the v1 cover letter
> for the full sequence.
> 
> Changes in v3, after Martin's review:
> 
> - No SG_IO. The re-add check now compares the udev WWID with one
> built
>   from the VPD page 0x83 data the kernel caches in sysfs, so it does
> no
>   I/O on the device and can't hang. The new get_sysfs_wwid() in
>   libmultipath shares get_vpd_uid() and get_uid()'s WWID clean-up
> with
>   the existing sysfs fallback, so both produce the same string.

I am confused. Does this really fix the problem in your use case?

You reported that the udev-derived WWID was false in some uevents, and
you said that, as the test was running under RHEL9, udev had obtained
the WWID from scsi_id, which uses SG_IO. While my previous argument
that SG_IO can block is valid, it's clear that *if* SG_IO works, it's a
better approximation for the REAL device WWID than the kernel's cached
value in sysfs, which might be stale. It makes only limited sense to me
that reading a cached sysfs value in multipathd would be more reliable
than what happened during udev rule processing.

Also, talking about consistency, on a RHEL system it would actually be
more consistent to use SG_IO than to read the WWID from sysfs.

I'd like to go back to your original problem description for a moment.

> 1. A change uevent for sdq (verify3) carried verify1's WWID. The next
>    uevent 4.5 s later carried the correct one. We don't know whether
>    the array answered the INQUIRY (VPD 0x83, run by scsi_id) wrongly
>    during the takeover or the bad value arose elsewhere in the udev
>    path; we have no packet capture.

"Elsewhere in the udev path" makes little sense to me. As we discussed,
udev on RHEL9 uses a VPD 83 INQUIRY, sent by scsi_id via SG_IO. The
WWID that multipathd reads from the uevent represents (part of) what
the storage had sent to scsi_id. If multipathd uses SG_IO and obtains a
different value, then either the storage must have answered one of the
two INQUIRYs wrongly, or something must have changed in the meantime.

(Of course we could have a blatant bug in multipathd that mixes up the
WWIDs in some corner case. But if that's the case, your patch wouldn't
help).

IMO, if we can't trust the reply of the storage array, all bets are
off. Why would the reply that multipathd obtains directly be more
trustworthy than what scsi_id had got? (Under SUSE you could see this
differently, because SUSE reads from sysfs, but you observed the issue
on RHEL).

You wrote that you'd seen this problem in an "array controller
failover" scenario while some test script was generating uevent
storms. I'd like to understand better what happens in this concrete
failover scenario. One possible case for a WWID change is that a
a Linux SCSI disk is removed, and another disk is added later, taking
over the previous device's dev_t. In this case we'd expect "remove" and
"add" uevents. With the uevent storm ongoing, these events may be
delayed. So multipathd may see a change event for the old device while
the kernel had already added the new one. This makes some sense, but
unfortunately it doesn't match your description:

> A change uevent for sdq (verify3) carried verify1's WWID. The next
> uevent 4.5 s later carried the correct one.

This sounds as if the WWID of sdq was that of "verify3", then
"verify1", and then "verify3" again. In my scenario above, sdq might
change from "verify3" to "verify1", but it wouldn't change back again,
unless there was another array failover in a very short time span.

Question about your test: Apparently the problem is reproducible,
otherwise you wouldn't be able to verify your patch. But you said "we
have no packet capture". If the test is actually reproducible, it
should be possible to create a capture and verify that the storage is
not the culprit. That would be very important IMO, because like I said
above, if this can happen, all bets are off.

I would really appreciate if we could clarify these open questions.

Back to the patch set:

IMO if you want to determine the *real* WWID, you _must_ use SG_IO. I
am contradicting my previous statement here, I know. I still hold that
SG_IO may block and that I'd rather avoid using it in multipathd. But
again, if you want to be certain (ignoring the fact that the storage
may provide false replies), SG_IO is you best bet. 

With this in mind, I suggest that you go back to v2 and SG_IO. Sorry
for having led you on the wrong track previously. 

Because I am generally sceptical of the use of SG_IO in multipathd in
its current form, and because the the overall idea is very similar to
the "recheck_wwid" feature, I further suggest that you make this code
path conditional on "recheck_wwid".

In the long run, we may want to redesign our WWID detection. We do have
a general framework for asynchronous jobs now. Thus we actually could
run asynchronous VPD inquries. This would eliminate the problem of
multipathd getting stuck on a hanging INQUIRY. But this needs more
thought and experiments.

Regards
Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)

Re: [PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Brian Bunker]

On Tue, 2026-11-10, Martin Wilck wrote:
> I am confused. Does this really fix the problem in your use case?
>
> You reported that the udev-derived WWID was false in some uevents, and
> you said that, as the test was running under RHEL9, udev had obtained
> the WWID from scsi_id, which uses SG_IO. While my previous argument
> that SG_IO can block is valid, it's clear that *if* SG_IO works, it's a
> better approximation for the REAL device WWID than the kernel's cached
> value in sysfs, which might be stale.

You're right. v3 would have caught our case only because the cached
page happened to still hold the value from the original scan. That's
luck, not a design. v4 goes back to SG_IO.

> "Elsewhere in the udev path" makes little sense to me. As we discussed,
> udev on RHEL9 uses a VPD 83 INQUIRY, sent by scsi_id via SG_IO. The
> WWID that multipathd reads from the uevent represents (part of) what
> the storage had sent to scsi_id. If multipathd uses SG_IO and obtains a
> different value, then either the storage must have answered one of the
> two INQUIRYs wrongly, or something must have changed in the meantime.

Agreed, and I'll drop that wording. I don't know where the wrong WWID
came from. For one change event on sdq (LUN 8), udev's ID_SERIAL was
the WWID of a different LUN (LUN 6). The logs show no SCSI device
removed or added, and the array's LUN map did not change. That leaves
the INQUIRY exchange itself: either the array answered for the wrong
LUN while it was promoting its SAN layer (it became primary about
450 ms before that event), or something on the initiator side did. We
have no packet capture, no udev event dump and no copy of the kernel's
cached page 0x83 from that moment, so I can't tell which.

The event itself was most likely one of the synthetic change events
our test tool generated: it opened the sd devices read-write, and
udev's "watch" turns each close into a change event, which reruns
scsi_id. In that hour there were about 3,600 such events on this path,
roughly one per second, and this is the only one we know returned the
wrong WWID. The kernel logged no ALUA unit attention for the path at
that time, so I don't think the failover generated the event. The
failover is why that one INQUIRY could have gone wrong. We have no log
that ties the event to a particular open/close, so that is inference.

> IMO, if we can't trust the reply of the storage array, all bets are
> off. Why would the reply that multipathd obtains directly be more
> trustworthy than what scsi_id had got?

It isn't more trustworthy, it's newer. The sequence on the host was:

  09:26:34.808  udev change event for sdq; ID_SERIAL is verify1's
                WWID
  09:26:34.833  uev_update_path() sees the change and calls
                ev_remove_path(); the reload of verify3's map then
                blocks in the resume until in-flight I/O drains
  09:26:39.221  the reload completes; sdq is freed
  09:26:39.235  uev_update_path() calls uev_add_path() with the same
                uevent, and sdq goes into verify1's map
  09:26:39.379  next change event for sdq; ID_SERIAL is verify3's
                WWID again

So the add acted on an answer that was 4.4 seconds old. An INQUIRY at
the time of the add would have reflected what the device said then,
and the one 140 ms later was correct. The check doesn't help against
storage that keeps giving the wrong answer. It stops multipathd from
moving a path into another LUN's map on an old answer.

> One possible case for a WWID change is that a
> a Linux SCSI disk is removed, and another disk is added later, taking
> over the previous device's dev_t.
[...]
> This sounds as if the WWID of sdq was that of "verify3", then
> "verify1", and then "verify3" again.

That's what happened, and it wasn't a remove and add. The kernel log
shows no SCSI device removed or added in that window, and sdq stayed
2:0:0:8 (65:0) throughout. Only that one udev event carried the wrong
WWID.

> Question about your test: Apparently the problem is reproducible,
> otherwise you wouldn't be able to verify your patch. But you said "we
> have no packet capture".

I should have been clearer: we've seen it once. We haven't reproduced
it. We've been running a v1-style patch on our test initiators since,
and it hasn't recurred, but at this rate that doesn't tell us much.
We've also changed our test tool to open the devices read-only, which
removes the flood of change events that made the window likely in the
first place. I'd like to get a capture, but I can't promise we'll
reproduce it.

> With this in mind, I suggest that you go back to v2 and SG_IO.
[...]
> I further suggest that you make this code
> path conditional on "recheck_wwid".

Will do; Ben suggested the same. v4 does the SG_IO check only on
PATH_ADD_REINIT, only when recheck_wwid is set, and drops the
uev_update_path() check from v2.

> In the long run, we may want to redesign our WWID detection. We do have
> a general framework for asynchronous jobs now. Thus we actually could
> run asynchronous VPD inquries.

That sounds like the right direction, and I'd be glad to help with it.

Brian

Re: [PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Martin Wilck]

On Wed, 2026-11-11 at 09:42 -0800, Brian Bunker wrote:
> On Tue, 2026-11-10, Martin Wilck wrote:
> 
> 
> > IMO, if we can't trust the reply of the storage array, all bets are
> > off. Why would the reply that multipathd obtains directly be more
> > trustworthy than what scsi_id had got?
> 
> It isn't more trustworthy, it's newer. The sequence on the host was:

Right, but "newer" doesn't mean "more accurate". You write below that
there is no indication that anything actually changed, so time
shouldn't be a factor. If a wrong VPD reply is possible once, it can
also occur multiple times. If it's just an unlikely failure, a purely
stochastical thing, it would help to simply repeat the same request; if
the probability for error is 1e-4, the probability for two errors in
succession would be 1e-8. However we don't know if it's stochastical.

> 
>   09:26:34.808  udev change event for sdq; ID_SERIAL is verify1's
>                 WWID
>   09:26:34.833  uev_update_path() sees the change and calls
>                 ev_remove_path(); the reload of verify3's map then
>                 blocks in the resume until in-flight I/O drains
>   09:26:39.221  the reload completes; sdq is freed
>   09:26:39.235  uev_update_path() calls uev_add_path() with the same
>                 uevent, and sdq goes into verify1's map
>   09:26:39.379  next change event for sdq; ID_SERIAL is verify3's
>                 WWID again
> 
> So the add acted on an answer that was 4.4 seconds old. An INQUIRY at
> the time of the add would have reflected what the device said then,
> and the one 140 ms later was correct. The check doesn't help against
> storage that keeps giving the wrong answer. It stops multipathd from
> moving a path into another LUN's map on an old answer.

Well you keep telling me that newer == better. Maybe you are right. We
can neither prove nor disprove that with the material we have.

> > One possible case for a WWID change is that a
> > a Linux SCSI disk is removed, and another disk is added later,
> > taking
> > over the previous device's dev_t.
> [...]
> > This sounds as if the WWID of sdq was that of "verify3", then
> > "verify1", and then "verify3" again.
> 
> That's what happened, and it wasn't a remove and add. The kernel log
> shows no SCSI device removed or added in that window, and sdq stayed
> 2:0:0:8 (65:0) throughout. Only that one udev event carried the wrong
> WWID.

Weird. Gamma radiation maybe? ;-)

> > Question about your test: Apparently the problem is reproducible,
> > otherwise you wouldn't be able to verify your patch. But you said
> > "we
> > have no packet capture".
> 
> I should have been clearer: we've seen it once. We haven't reproduced
> it. We've been running a v1-style patch on our test initiators since,

If it was my choice, I'd prefer repeating the test with the unpatched
code, but additional debugging instrumentation, in the hope to obtain
more insight into what actually happened.

> and it hasn't recurred, but at this rate that doesn't tell us much.

I'd go one step further and say that it tells us nothing at all. You
would need to be able to reproduce the issue in a well-defined time
frame in order to do a test that gives at least some confidence that
your patch fixed it.

However from the time line you provided, it makes sense that your patch
would have been effective in the error case.

Regards,
Martin

> We've also changed our test tool to open the devices read-only, which
> removes the flood of change events that made the window likely in the
> first place. I'd like to get a capture, but I can't promise we'll
> reproduce it.



> 
> > With this in mind, I suggest that you go back to v2 and SG_IO.
> [...]
> > I further suggest that you make this code
> > path conditional on "recheck_wwid".
> 
> Will do; Ben suggested the same. v4 does the SG_IO check only on
> PATH_ADD_REINIT, only when recheck_wwid is set, and drops the
> uev_update_path() check from v2.
> 
> > In the long run, we may want to redesign our WWID detection. We do
> > have
> > a general framework for asynchronous jobs now. Thus we actually
> > could
> > run asynchronous VPD inquries.
> 
> That sounds like the right direction, and I'd be glad to help with
> it.
> 
> Brian

-- 
Dr. Martin Wilck <mwilck@suse.com>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)

Re: [PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Brian Bunker]

On Wed, 2026-11-11, Martin Wilck wrote:
> Right, but "newer" doesn't mean "more accurate". You write below that
> there is no indication that anything actually changed, so time
> shouldn't be a factor. If a wrong VPD reply is possible once, it can
> also occur multiple times. If it's just an unlikely failure, a purely
> stochastical thing, it would help to simply repeat the same request;
> if the probability for error is 1e-4, the probability for two errors
> in succession would be 1e-8. However we don't know if it's
> stochastical.

Fair, "newer" was the wrong word, and we can't show what caused the bad
reply.

I think the patch stands without that, though. recheck_wwid already
exists to stop a path whose WWID has changed from staying in, or going
into, the wrong map: multipathd checks the WWID with the device when a
failed path comes back, and when a path is added with "multipathd add
path". The re-add in uev_update_path() is a third place where a path is
put into a map because of a WWID change, and the only one of the three
that acts purely on the udev value. v4 applies the same check there,
under the same option.

It also costs little: nothing unless recheck_wwid is set, and with it
set, one INQUIRY on a re-add that only happens after udev reports a
changed WWID. If it ever refuses an add, it logs the udev WWID and the
one the device returned. We'll run with recheck_wwid enabled on our test
initiators and watch for that message. If it shows up more than once,
we'll come back with those logs, which would be better evidence than the
one run we have, where we didn't know at the time that our test tool was
flooding udev.

> I'd go one step further and say that it tells us nothing at all.

Agreed.

> However from the time line you provided, it makes sense that your
> patch would have been effective in the error case.

Thanks.

Brian

udev / multipath-tools: WWID determination with SG_IO vs. sysfs

[Martin Wilck]

Brian, Ben, Hannes,

Following up the discussion on Brian's latest patch, I've researched a
bit into the SG_IO vs. sysfs topic. I wondered why, under RHEL and
other distributions, where scsi_id is being used for device
identification, udev rules hanging in SCSI device ID determination
doesn't seem to be an issue, whereas SUSE has switched to sg_inq --
inhex --raw on sysfs files more than 10 years ago for precisely that
reason.

# About SG_IO blocking in udev rules:

SUSE switched from scsi_id to sg_inq in 2014, before releasing SLE12.
We got reports about udev rules hanging in device detection, but
AFAICS, only for USB devices. Actually, for those, it seems that the
hanging syscall was not the SG_IO ioctl itself, but the initial open()
on the device node (we have no evidence about the reasons; it's likely
that device probing was stuck in the kernel because of misbehaving
cheap USB hardware).

The reason that these hangs in udev rules have been seen under
(open)SUSE and apparently not on RHEL or other distros is rather
trivial: In 60-persistent-storage.rules, scsi_id is only called if
ID_SERIAL is not set yet [1]. For those USB devices that SUSE has seen
trouble with, ID_SERIAL would be set by usb_id, and thus scsi_id is
never invoked; therfore no hang. OTOH, 55-scsi-sg3_id.rules calls
sg_inq unconditionally [2], which may cause hangs with badly behaving
USB devices. To fix that, SUSE switched from SG_IO to reading sysfs
files a while after it had switched to sg_inq.

So far I haven't found evidence for a hanging sg_inq on hardware that
actually matters for multipath. That doesn't mean that it can't happen,
network-based I/O can always suffer from timeouts or delays, of course,
we have observed it quite a few times. But if this was a frequent
issue, I would expect it to get noticed by users.

@Ben, have you ever observed SG_IO ioctls blocking in scsi_id in udev
rules?

@Hannes, am I overlooking something? Do you recall cases of hanging
sg_inq processes on devices other than USB?

# About SG_IO blocking in multipathd:

Broken USB devices are not an issue for multipathd. W have ignored USB
devices since 0.6.0, 51957eb ("Skip USB devices during discovery").
Since 0.8.5, 5bd2645 ("libmultipath: Allow discovery of USB devices"),
they can be re-enabled, but I suppose that that's not a frequently used
feature.

We mostly avoid ioctls for paths that are down or otherwise unusable in
multipathd, except for the checker, where we use threads. Still, even
if the checker has seen a path in UP state, it can happen that SG_IO
would stall when retrieving the priority. There is sporadic evidence
that this has happened, but not much (surprisingly few, actually, and
most reports that I could find are quite old already).

I am not aware of any cases where the recheck_wwid code had caused
hanging ioctls.

Brian's patch would add more such ioctls and thus increase the
likelihood of observing a stall. To be observed.


# About sysfs data being stale:

The kernel will update the VPD pages when a UNIT ATTENTION with
ASC/ASCQ 3f/03 occurs ("INQURIY data has changed"; since kernel v4.6).
Normally, the VPD page update happens synchronously before the uevent
is sent to user space. Therefore reading the sysfs attribute from the
udev rule is guaranteed (almost, see below) to return the data that was
current right after the UA had been processed (the uevent is sent from
a kernel thread, thus a small delay is possible). 

However, it is possible that the update of the VPD data is skipped
because the device is not running, or the queue is quiesced. In this
case, the sysfs VPD data would also be stale (but later SG_IO would be
likely to fail as well).

It is also possible that the VPD data changes between establishing the
UA and calling scsi_id or sg_inq, but the probability should be rather
low.

Similar considerations hold for "add" events; for a device just added,
I'd expect the VPD data to be correct with very high likelihood,
especially if it is in "running" state.

The kernel does *not* rescan devices on receiving a UA with ASC/ASCQ
3f/0e ("REPORT LUNS data has changed"). AFAIU, it isn't well defined
what condition a given storage array actually establishes when a LUN
mapping changes. 3f/03 and 3f/0e are both possible, as is 29/xx "Power
on or device reset occured". 3f/0e may actually be received on a
different LUN than the one that had actually changed.

So, of 3f/0e or 29/xx occurs, the VPD data may be stale. And of course
also if the LUN mapping changes without any event occuring, as
discussed in the past in the context of the "recheck_wwid" option. Only
actively writing to the "rescan" sysfs attribute would fix this.

Thinking about this, it might make sense to switch from sysfs to SG_IO
in udev rules if we observe a 3f/0e or 29/xx UA.

@Hannes, what do you think?

IIRC we've discussed handling UA events in multipathd before, but
nobody has started working on it so far. If we do, we'd need to handle
all 3 kinds of events.

Regards & sorry for the long post,
Martin


[1] https://github.com/systemd/systemd/blob/6ec3ff3f41b7569d96a475a11db716f5feb35a6f/rules.d/60-persistent-storage.rules.in#L75
[2] https://github.com/doug-gilbert/sg3_utils/blob/8799caa3741b2b1640970747aeb84f17dbbaad86/scripts/55-scsi-sg3_id.rules#L67

-- 
Dr. Martin Wilck <mwilck@suse.com>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)

Re: udev / multipath-tools: WWID determination with SG_IO vs. sysfs

[Benjamin Marzinski]

On Thu, Nov 12, 2026 at 08:00:58PM +0100, Martin Wilck wrote:
> Brian, Ben, Hannes,
> 
> Following up the discussion on Brian's latest patch, I've researched a
> bit into the SG_IO vs. sysfs topic. I wondered why, under RHEL and
> other distributions, where scsi_id is being used for device
> identification, udev rules hanging in SCSI device ID determination
> doesn't seem to be an issue, whereas SUSE has switched to sg_inq --
> inhex --raw on sysfs files more than 10 years ago for precisely that
> reason.
> 
> # About SG_IO blocking in udev rules:
> 
> SUSE switched from scsi_id to sg_inq in 2014, before releasing SLE12.
> We got reports about udev rules hanging in device detection, but
> AFAICS, only for USB devices. Actually, for those, it seems that the
> hanging syscall was not the SG_IO ioctl itself, but the initial open()
> on the device node (we have no evidence about the reasons; it's likely
> that device probing was stuck in the kernel because of misbehaving
> cheap USB hardware).
> 
> The reason that these hangs in udev rules have been seen under
> (open)SUSE and apparently not on RHEL or other distros is rather
> trivial: In 60-persistent-storage.rules, scsi_id is only called if
> ID_SERIAL is not set yet [1]. For those USB devices that SUSE has seen
> trouble with, ID_SERIAL would be set by usb_id, and thus scsi_id is
> never invoked; therfore no hang. OTOH, 55-scsi-sg3_id.rules calls
> sg_inq unconditionally [2], which may cause hangs with badly behaving
> USB devices. To fix that, SUSE switched from SG_IO to reading sysfs
> files a while after it had switched to sg_inq.
> 
> So far I haven't found evidence for a hanging sg_inq on hardware that
> actually matters for multipath. That doesn't mean that it can't happen,
> network-based I/O can always suffer from timeouts or delays, of course,
> we have observed it quite a few times. But if this was a frequent
> issue, I would expect it to get noticed by users.
> 
> @Ben, have you ever observed SG_IO ioctls blocking in scsi_id in udev
> rules?

Over the years I've seen occassional issues where udev workers time out.
Unfortunately, they are either from something obviously wrong (which
hasn't been getting the device ID, IIRC), or there usually isn't enough
data to determine why, and they can't be reproduced.
 
> @Hannes, am I overlooking something? Do you recall cases of hanging
> sg_inq processes on devices other than USB?
> 
> # About SG_IO blocking in multipathd:
> 
> Broken USB devices are not an issue for multipathd. W have ignored USB
> devices since 0.6.0, 51957eb ("Skip USB devices during discovery").
> Since 0.8.5, 5bd2645 ("libmultipath: Allow discovery of USB devices"),
> they can be re-enabled, but I suppose that that's not a frequently used
> feature.
> 
> We mostly avoid ioctls for paths that are down or otherwise unusable in
> multipathd, except for the checker, where we use threads. Still, even
> if the checker has seen a path in UP state, it can happen that SG_IO
> would stall when retrieving the priority. There is sporadic evidence
> that this has happened, but not much (surprisingly few, actually, and
> most reports that I could find are quite old already).
> 
> I am not aware of any cases where the recheck_wwid code had caused
> hanging ioctls.

I know some users have turned this on, if they have problems fixing
something that is remapping their LUNs without making sure that they
aren't in use. I don't have a good handle on how many users that is, but
I'm pretty sure it's not a large number. I have never heard of it
causing problems though.
 
> Brian's patch would add more such ioctls and thus increase the
> likelihood of observing a stall. To be observed.
> 

This is true, and unlike the existing recheck_wwids use and the
prioritizer, it won't be called right after the path checker determined
a path to be up. We do also do SG_IO ioctls in pathinfo when DI_IOCTL is
set (when we're adding a path, reconfiguring multipath, or initializing
a partial path). In that case we do check if the path is up according to
sysfs first, but we don't call the checker.

-Ben

> 
> # About sysfs data being stale:
> 
> The kernel will update the VPD pages when a UNIT ATTENTION with
> ASC/ASCQ 3f/03 occurs ("INQURIY data has changed"; since kernel v4.6).
> Normally, the VPD page update happens synchronously before the uevent
> is sent to user space. Therefore reading the sysfs attribute from the
> udev rule is guaranteed (almost, see below) to return the data that was
> current right after the UA had been processed (the uevent is sent from
> a kernel thread, thus a small delay is possible). 
> 
> However, it is possible that the update of the VPD data is skipped
> because the device is not running, or the queue is quiesced. In this
> case, the sysfs VPD data would also be stale (but later SG_IO would be
> likely to fail as well).
> 
> It is also possible that the VPD data changes between establishing the
> UA and calling scsi_id or sg_inq, but the probability should be rather
> low.
> 
> Similar considerations hold for "add" events; for a device just added,
> I'd expect the VPD data to be correct with very high likelihood,
> especially if it is in "running" state.
> 
> The kernel does *not* rescan devices on receiving a UA with ASC/ASCQ
> 3f/0e ("REPORT LUNS data has changed"). AFAIU, it isn't well defined
> what condition a given storage array actually establishes when a LUN
> mapping changes. 3f/03 and 3f/0e are both possible, as is 29/xx "Power
> on or device reset occured". 3f/0e may actually be received on a
> different LUN than the one that had actually changed.
> 
> So, of 3f/0e or 29/xx occurs, the VPD data may be stale. And of course
> also if the LUN mapping changes without any event occuring, as
> discussed in the past in the context of the "recheck_wwid" option. Only
> actively writing to the "rescan" sysfs attribute would fix this.
> 
> Thinking about this, it might make sense to switch from sysfs to SG_IO
> in udev rules if we observe a 3f/0e or 29/xx UA.
> 
> @Hannes, what do you think?
> 
> IIRC we've discussed handling UA events in multipathd before, but
> nobody has started working on it so far. If we do, we'd need to handle
> all 3 kinds of events.
> 
> Regards & sorry for the long post,
> Martin
> 
> 
> [1] https://github.com/systemd/systemd/blob/6ec3ff3f41b7569d96a475a11db716f5feb35a6f/rules.d/60-persistent-storage.rules.in#L75
> [2] https://github.com/doug-gilbert/sg3_utils/blob/8799caa3741b2b1640970747aeb84f17dbbaad86/scripts/55-scsi-sg3_id.rules#L67
> 
> -- 
> Dr. Martin Wilck <mwilck@suse.com>
> SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
> Germany
> Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
> 36809,AG Nürnberg)

Re: udev / multipath-tools: WWID determination with SG_IO vs. sysfs

[Martin Wilck]

On Thu, 2026-11-12 at 14:58 -0500, Benjamin Marzinski wrote:
> On Thu, Nov 12, 2026 at 08:00:58PM +0100, Martin Wilck wrote:
> 
>  
> > Brian's patch would add more such ioctls and thus increase the
> > likelihood of observing a stall. To be observed.
> > 
> 
> This is true, and unlike the existing recheck_wwids use and the
> prioritizer, it won't be called right after the path checker
> determined
> a path to be up. We do also do SG_IO ioctls in pathinfo when DI_IOCTL
> is
> set (when we're adding a path, reconfiguring multipath, or
> initializing
> a partial path). In that case we do check if the path is up according
> to
> sysfs first, but we don't call the checker.

So, would you prefer making it conditional on something else but
"recheck_wwid"?

Martin

-- 
Dr. Martin Wilck <mwilck@suse.com>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)