[PATCH] multipath.rules.in: fix race in unit cancel-multipath-wait-$kernel

[PATCH] multipath.rules.in: fix race in unit cancel-multipath-wait-$kernel

[Antonio Alvarez Feijoo]

The transient unit cancel-multipath-wait-$kernel creates a timer and
service with "Conflicts=initrd-cleanup.service":

- sda: /usr/lib/udev/rules.d/56-multipath.rules ...
    --on-active=1.000000 ...
- sda: ... 'Running timer as unit: cancel-multipath-wait-sda.timer'
- sda: ... 'Will run service as unit:
    cancel-multipath-wait-sda.service'

This introduces a race not always reproducible: there is a 1 second
window where this transient unit can start after
initrd-cleanup.service/start is enqueued but is still held back. When
that happens, starting cancel-multipath-wait-$kernel.service pulls in
a stop job for initrd-cleanup.service and the queued start job is
cancelled, the initrd is left with an empty job queue and hangs
forever, before switch-root:

- initrd-parse-etc.service: Triggering OnSuccess= dependencies.
- initrd-cleanup.service: Installed new job
    initrd-cleanup.service/start as 115
- initrd-cleanup.service: starting held back, waiting for:
    dracut-pre-pivot.service
- initrd-cleanup.service: Job 115 initrd-cleanup.service/start
    finished, result=canceled
- initrd-cleanup.service: Installed new job
    initrd-cleanup.service/stop as 133
- initrd-cleanup.service: Job 133 initrd-cleanup.service/stop
    finished, result=done

"Conflicts=" is symmetric: if a unit has a "Conflicts=" requirement
on a set of other units, then starting it will stop all of them and
starting any of them will stop it. This is safe against
shutdown.target, whose jobs are enqueued irreversibly: a unit
starting during shutdown fails to start rather than cancelling the
shutdown. It is not safe against initrd-cleanup.service, whose start
job is an ordinary job enqueued by initrd-parse-etc.service via
"OnSuccess=".

Ordering "Before=initrd-cleanup.service" is enough and is not
destructive: a trigger already in flight completes before cleanup
runs, and a unit with no job does not delay it. Tearing the
transient units down before switch-root is already handled by the
isolate to initrd-switch-root.target, which stops everything that
is not "IgnoreOnIsolate=yes".

Signed-off-by: Antonio Alvarez Feijoo <antonio.feijoo@suse.com>
---
 multipath/multipath.rules.in | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/multipath/multipath.rules.in b/multipath/multipath.rules.in
index 2ac1972f..7b605dd4 100644
--- a/multipath/multipath.rules.in
+++ b/multipath/multipath.rules.in
@@ -68,8 +68,18 @@ ENV{.SAVED_FM_WAIT_UNTIL}=="?*", GOTO="pretend_mpath"
 # the --on-active timeout.
 #
 # We must trigger an "add" event because LVM2 will only act on those.
-
-RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel --description 'cancel waiting for multipath siblings of $kernel' --no-block --timer-property DefaultDependencies=no --timer-property Conflicts=shutdown.target --timer-property Before=shutdown.target --timer-property Conflicts=initrd-cleanup.service --timer-property Before=initrd-cleanup.service --timer-property AccuracySec=500ms --property DefaultDependencies=no --property Conflicts=shutdown.target --property Before=shutdown.target --property Conflicts=initrd-cleanup.service --property Before=initrd-cleanup.service --on-active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger --action=add $sys$devpath"
+#
+# These units are ordered Before=initrd-cleanup.service so that a trigger that
+# is already in flight completes before the initrd switches root. They must
+# *not* use Conflicts=initrd-cleanup.service: unlike shutdown.target, whose
+# jobs are enqueued irreversibly, initrd-cleanup.service/start is a plain job
+# enqueued by initrd-parse-etc.service via OnSuccess=. Starting a unit that
+# conflicts with it therefore cancels that job instead of failing to start,
+# and OnSuccess= is never retried, so the initrd would hang forever with an
+# empty job queue. Stopping these units before switch-root is already
+# guaranteed by the isolate to initrd-switch-root.target.
+
+RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel --description 'cancel waiting for multipath siblings of $kernel' --no-block --timer-property DefaultDependencies=no --timer-property Conflicts=shutdown.target --timer-property Before=shutdown.target --timer-property Before=initrd-cleanup.service --timer-property AccuracySec=500ms --property DefaultDependencies=no --property Conflicts=shutdown.target --property Before=shutdown.target --property Before=initrd-cleanup.service --on-active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger --action=add $sys$devpath"
 
 LABEL="pretend_mpath"
 ENV{DM_MULTIPATH_DEVICE_PATH}="1"
-- 
2.51.0

Re: [PATCH] multipath.rules.in: fix race in unit cancel-multipath-wait-$kernel

[Martin Wilck]

On Fri, 2026-08-28 at 11:25 +0200, Antonio Alvarez Feijoo wrote:
> The transient unit cancel-multipath-wait-$kernel creates a timer and
> service with "Conflicts=initrd-cleanup.service":
> 
> - sda: /usr/lib/udev/rules.d/56-multipath.rules ...
>     --on-active=1.000000 ...
> - sda: ... 'Running timer as unit: cancel-multipath-wait-sda.timer'
> - sda: ... 'Will run service as unit:
>     cancel-multipath-wait-sda.service'
> 
> This introduces a race not always reproducible: there is a 1 second
> window where this transient unit can start after
> initrd-cleanup.service/start is enqueued but is still held back. When
> that happens, starting cancel-multipath-wait-$kernel.service pulls in
> a stop job for initrd-cleanup.service and the queued start job is
> cancelled, the initrd is left with an empty job queue and hangs
> forever, before switch-root:
> 
> - initrd-parse-etc.service: Triggering OnSuccess= dependencies.
> - initrd-cleanup.service: Installed new job
>     initrd-cleanup.service/start as 115
> - initrd-cleanup.service: starting held back, waiting for:
>     dracut-pre-pivot.service
> - initrd-cleanup.service: Job 115 initrd-cleanup.service/start
>     finished, result=canceled
> - initrd-cleanup.service: Installed new job
>     initrd-cleanup.service/stop as 133
> - initrd-cleanup.service: Job 133 initrd-cleanup.service/stop
>     finished, result=done
> 
> "Conflicts=" is symmetric: if a unit has a "Conflicts=" requirement
> on a set of other units, then starting it will stop all of them and
> starting any of them will stop it. This is safe against
> shutdown.target, whose jobs are enqueued irreversibly: a unit
> starting during shutdown fails to start rather than cancelling the
> shutdown. It is not safe against initrd-cleanup.service, whose start
> job is an ordinary job enqueued by initrd-parse-etc.service via
> "OnSuccess=".
> 
> Ordering "Before=initrd-cleanup.service" is enough and is not
> destructive: a trigger already in flight completes before cleanup
> runs, and a unit with no job does not delay it. Tearing the
> transient units down before switch-root is already handled by the
> isolate to initrd-switch-root.target, which stops everything that
> is not "IgnoreOnIsolate=yes".
> 
> Signed-off-by: Antonio Alvarez Feijoo <antonio.feijoo@suse.com>

Reviewed-by: Martin Wilck <mwilck@suse.com>

@Ben and others, I wasn't certain about this at first. Below is my flow
of thoughts from the SUSE bugzilla, through which I came to think that
it's correct. But this is quite subtle, so please give it some own
thoughts ;-)

Martin

The general idea of the "cancel" timer is this:

1. uevent for a block device arrives, the WWID is yet unknown to
multipath
2. multipathd sees the event and temporarily sets SYSTEMD_READY=0,
avoiding other subsystems like MD or LVM grabbing it, and waiting for
potential other paths to appear
3. udev starts the transient timer unit
3a. some other path to the same volume appears, and the timer is
cancelled (we are not discussing this case here)
4. when the timer expires, it starts "udevadm trigger"
5. the kernel generates a new uevent for the same block device
6. while handling the uevent, "multipath -u" is run. 
7. realizes that the wait for other paths has timed out, and remove
SYSTEMD_READY, opening it for other layers to grab.

There's a race window between 4 and 6 which systemd cannot handle.
systemd does not "know" that a uevent has been triggered, so even if 3)
or 4) happen before the start of initrd-cleanup.service, 5, 6, 7 may
happen afterwards.

In earlier versions of multipathd and dracut "multipath -u" would
attempt to start multipathd.service via socket activation
(multipathd.socket), which would then stop initrd-cleanup.service, as
it Conflicts: it. Since 0.9.8, multipathd.socket is disabled by
default, and we haven't included it in the initrd any more for a while,
too [*].

It's important to understand that the "cancel" timer+service are just a
safeguard for the case that no new uevent for the device in question
occurs. If we're in the initrd, we can be sure that every uevent is
going to be replayed in the udev coldplug phase. 

We still need to start the transient timer in the initrd for handling
the case that the device in question is needed for the root file
system, and is not multipathed. But when initrd-cleanup.service is
started, the root device has been discovered already, and we don't need
to bother about the "cancel" timer and service. It doesn't matter if
they run or not.

In the worst case, Antonio's patch will cause these units to be started
after initrd-cleanup.service is started, which may cause an extra "add"
uevent for the block device in question. But that shouldn't be a
problem because such an event is going to be replayed after switching
root, anyway.

IIUC, if the "Conflicts" is removed, the timer unit will continue
running through the switch root phase, and will be taken up by systemd
through serialization 

[*] upstream dracut version history related to multipathd.socket:

247d2b ("fix(multipathd.service): adapt to upstream multipath-tools
unit file") (dracut 056) added socket dep to the multipathd.service
unit file.
02e646f ("fix(multipath): install multipathd.socket") (dracut 058) made
sure the socket unit file was installed.
297525c ("fix(multipath): remove dependency on multipathd.socket")
(dracut 060) removed both

5e87b68 ("refactor(multipath): remove custom multipathd.service")
(dracut 107) removed the .service file.

> ---
>  multipath/multipath.rules.in | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/multipath/multipath.rules.in
> b/multipath/multipath.rules.in
> index 2ac1972f..7b605dd4 100644
> --- a/multipath/multipath.rules.in
> +++ b/multipath/multipath.rules.in
> @@ -68,8 +68,18 @@ ENV{.SAVED_FM_WAIT_UNTIL}=="?*",
> GOTO="pretend_mpath"
>  # the --on-active timeout.
>  #
>  # We must trigger an "add" event because LVM2 will only act on
> those.
> -
> -RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel
> --description 'cancel waiting for multipath siblings of $kernel' --
> no-block --timer-property DefaultDependencies=no --timer-property
> Conflicts=shutdown.target --timer-property Before=shutdown.target --
> timer-property Conflicts=initrd-cleanup.service --timer-property
> Before=initrd-cleanup.service --timer-property AccuracySec=500ms --
> property DefaultDependencies=no --property Conflicts=shutdown.target
> --property Before=shutdown.target --property Conflicts=initrd-
> cleanup.service --property Before=initrd-cleanup.service --on-
> active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger
> --action=add $sys$devpath"
> +#
> +# These units are ordered Before=initrd-cleanup.service so that a
> trigger that
> +# is already in flight completes before the initrd switches root.
> They must
> +# *not* use Conflicts=initrd-cleanup.service: unlike
> shutdown.target, whose
> +# jobs are enqueued irreversibly, initrd-cleanup.service/start is a
> plain job
> +# enqueued by initrd-parse-etc.service via OnSuccess=. Starting a
> unit that
> +# conflicts with it therefore cancels that job instead of failing to
> start,
> +# and OnSuccess= is never retried, so the initrd would hang forever
> with an
> +# empty job queue. Stopping these units before switch-root is
> already
> +# guaranteed by the isolate to initrd-switch-root.target.
> +
> +RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel
> --description 'cancel waiting for multipath siblings of $kernel' --
> no-block --timer-property DefaultDependencies=no --timer-property
> Conflicts=shutdown.target --timer-property Before=shutdown.target --
> timer-property Before=initrd-cleanup.service --timer-property
> AccuracySec=500ms --property DefaultDependencies=no --property
> Conflicts=shutdown.target --property Before=shutdown.target --
> property Before=initrd-cleanup.service --on-
> active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger
> --action=add $sys$devpath"
>  
>  LABEL="pretend_mpath"
>  ENV{DM_MULTIPATH_DEVICE_PATH}="1"

-- 
Dr. Martin Wilck <mwilck@suse.com>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Jochen Jaser, Andrew McDonald (HRB 36809,AG Nürnberg)

Re: [PATCH] multipath.rules.in: fix race in unit cancel-multipath-wait-$kernel

[Benjamin Marzinski]

On Fri, Aug 28, 2026 at 03:57:38PM +0200, Martin Wilck wrote:
> @Ben and others, I wasn't certain about this at first. Below is my flow
> of thoughts from the SUSE bugzilla, through which I came to think that
> it's correct. But this is quite subtle, so please give it some own
> thoughts ;-)
> 
> Martin
> 
> IIUC, if the "Conflicts" is removed, the timer unit will continue
> running through the switch root phase, and will be taken up by systemd
> through serialization 

But like Antonio said, isolate will stop the timer, since it's not a
dependency of initrd-switch-root.target, right? I'm a little fuzzy on
exactly how all the units get stopped, and you might be right that a
udev rule could run and trigger a timer after the isolate could catch
it. But I think you are right that this would be harmless if it could
occur.

-Ben

> 
> [*] upstream dracut version history related to multipathd.socket:
> 
> 247d2b ("fix(multipathd.service): adapt to upstream multipath-tools
> unit file") (dracut 056) added socket dep to the multipathd.service
> unit file.
> 02e646f ("fix(multipath): install multipathd.socket") (dracut 058) made
> sure the socket unit file was installed.
> 297525c ("fix(multipath): remove dependency on multipathd.socket")
> (dracut 060) removed both
> 
> 5e87b68 ("refactor(multipath): remove custom multipathd.service")
> (dracut 107) removed the .service file.
> 
> > ---
> >  multipath/multipath.rules.in | 14 ++++++++++++--
> >  1 file changed, 12 insertions(+), 2 deletions(-)
> > 
> > diff --git a/multipath/multipath.rules.in
> > b/multipath/multipath.rules.in
> > index 2ac1972f..7b605dd4 100644
> > --- a/multipath/multipath.rules.in
> > +++ b/multipath/multipath.rules.in
> > @@ -68,8 +68,18 @@ ENV{.SAVED_FM_WAIT_UNTIL}=="?*",
> > GOTO="pretend_mpath"
> >  # the --on-active timeout.
> >  #
> >  # We must trigger an "add" event because LVM2 will only act on
> > those.
> > -
> > -RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel
> > --description 'cancel waiting for multipath siblings of $kernel' --
> > no-block --timer-property DefaultDependencies=no --timer-property
> > Conflicts=shutdown.target --timer-property Before=shutdown.target --
> > timer-property Conflicts=initrd-cleanup.service --timer-property
> > Before=initrd-cleanup.service --timer-property AccuracySec=500ms --
> > property DefaultDependencies=no --property Conflicts=shutdown.target
> > --property Before=shutdown.target --property Conflicts=initrd-
> > cleanup.service --property Before=initrd-cleanup.service --on-
> > active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger
> > --action=add $sys$devpath"
> > +#
> > +# These units are ordered Before=initrd-cleanup.service so that a
> > trigger that
> > +# is already in flight completes before the initrd switches root.
> > They must
> > +# *not* use Conflicts=initrd-cleanup.service: unlike
> > shutdown.target, whose
> > +# jobs are enqueued irreversibly, initrd-cleanup.service/start is a
> > plain job
> > +# enqueued by initrd-parse-etc.service via OnSuccess=. Starting a
> > unit that
> > +# conflicts with it therefore cancels that job instead of failing to
> > start,
> > +# and OnSuccess= is never retried, so the initrd would hang forever
> > with an
> > +# empty job queue. Stopping these units before switch-root is
> > already
> > +# guaranteed by the isolate to initrd-switch-root.target.
> > +
> > +RUN+="@SYSDIR_BIN@/systemd-run --unit=cancel-multipath-wait-$kernel
> > --description 'cancel waiting for multipath siblings of $kernel' --
> > no-block --timer-property DefaultDependencies=no --timer-property
> > Conflicts=shutdown.target --timer-property Before=shutdown.target --
> > timer-property Before=initrd-cleanup.service --timer-property
> > AccuracySec=500ms --property DefaultDependencies=no --property
> > Conflicts=shutdown.target --property Before=shutdown.target --
> > property Before=initrd-cleanup.service --on-
> > active=$env{FIND_MULTIPATHS_WAIT_UNTIL} @SYSDIR_BIN@/udevadm trigger
> > --action=add $sys$devpath"
> >  
> >  LABEL="pretend_mpath"
> >  ENV{DM_MULTIPATH_DEVICE_PATH}="1"
> 
> -- 
> Dr. Martin Wilck <mwilck@suse.com>
> SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
> Germany
> Geschäftsführer: Jochen Jaser, Andrew McDonald (HRB 36809,AG Nürnberg)