Re: [PATCH v3 0/1] multipathd: check re-added paths' WWIDs against sysfs

[Brian Bunker <brian@purestorage.com>]

On Tue, 2026-11-10, Martin Wilck wrote:
> I am confused. Does this really fix the problem in your use case?
>
> You reported that the udev-derived WWID was false in some uevents, and
> you said that, as the test was running under RHEL9, udev had obtained
> the WWID from scsi_id, which uses SG_IO. While my previous argument
> that SG_IO can block is valid, it's clear that *if* SG_IO works, it's a
> better approximation for the REAL device WWID than the kernel's cached
> value in sysfs, which might be stale.

You're right. v3 would have caught our case only because the cached
page happened to still hold the value from the original scan. That's
luck, not a design. v4 goes back to SG_IO.

> "Elsewhere in the udev path" makes little sense to me. As we discussed,
> udev on RHEL9 uses a VPD 83 INQUIRY, sent by scsi_id via SG_IO. The
> WWID that multipathd reads from the uevent represents (part of) what
> the storage had sent to scsi_id. If multipathd uses SG_IO and obtains a
> different value, then either the storage must have answered one of the
> two INQUIRYs wrongly, or something must have changed in the meantime.

Agreed, and I'll drop that wording. I don't know where the wrong WWID
came from. For one change event on sdq (LUN 8), udev's ID_SERIAL was
the WWID of a different LUN (LUN 6). The logs show no SCSI device
removed or added, and the array's LUN map did not change. That leaves
the INQUIRY exchange itself: either the array answered for the wrong
LUN while it was promoting its SAN layer (it became primary about
450 ms before that event), or something on the initiator side did. We
have no packet capture, no udev event dump and no copy of the kernel's
cached page 0x83 from that moment, so I can't tell which.

The event itself was most likely one of the synthetic change events
our test tool generated: it opened the sd devices read-write, and
udev's "watch" turns each close into a change event, which reruns
scsi_id. In that hour there were about 3,600 such events on this path,
roughly one per second, and this is the only one we know returned the
wrong WWID. The kernel logged no ALUA unit attention for the path at
that time, so I don't think the failover generated the event. The
failover is why that one INQUIRY could have gone wrong. We have no log
that ties the event to a particular open/close, so that is inference.

> IMO, if we can't trust the reply of the storage array, all bets are
> off. Why would the reply that multipathd obtains directly be more
> trustworthy than what scsi_id had got?

It isn't more trustworthy, it's newer. The sequence on the host was:

  09:26:34.808  udev change event for sdq; ID_SERIAL is verify1's
                WWID
  09:26:34.833  uev_update_path() sees the change and calls
                ev_remove_path(); the reload of verify3's map then
                blocks in the resume until in-flight I/O drains
  09:26:39.221  the reload completes; sdq is freed
  09:26:39.235  uev_update_path() calls uev_add_path() with the same
                uevent, and sdq goes into verify1's map
  09:26:39.379  next change event for sdq; ID_SERIAL is verify3's
                WWID again

So the add acted on an answer that was 4.4 seconds old. An INQUIRY at
the time of the add would have reflected what the device said then,
and the one 140 ms later was correct. The check doesn't help against
storage that keeps giving the wrong answer. It stops multipathd from
moving a path into another LUN's map on an old answer.

> One possible case for a WWID change is that a
> a Linux SCSI disk is removed, and another disk is added later, taking
> over the previous device's dev_t.
[...]
> This sounds as if the WWID of sdq was that of "verify3", then
> "verify1", and then "verify3" again.

That's what happened, and it wasn't a remove and add. The kernel log
shows no SCSI device removed or added in that window, and sdq stayed
2:0:0:8 (65:0) throughout. Only that one udev event carried the wrong
WWID.

> Question about your test: Apparently the problem is reproducible,
> otherwise you wouldn't be able to verify your patch. But you said "we
> have no packet capture".

I should have been clearer: we've seen it once. We haven't reproduced
it. We've been running a v1-style patch on our test initiators since,
and it hasn't recurred, but at this rate that doesn't tell us much.
We've also changed our test tool to open the devices read-only, which
removes the flood of change events that made the window likely in the
first place. I'd like to get a capture, but I can't promise we'll
reproduce it.

> With this in mind, I suggest that you go back to v2 and SG_IO.
[...]
> I further suggest that you make this code
> path conditional on "recheck_wwid".

Will do; Ben suggested the same. v4 does the SG_IO check only on
PATH_ADD_REINIT, only when recheck_wwid is set, and drops the
uev_update_path() check from v2.

> In the long run, we may want to redesign our WWID detection. We do have
> a general framework for asynchronous jobs now. Thus we actually could
> run asynchronous VPD inquries.

That sounds like the right direction, and I'd be glad to help with it.

Brian