Chicken and egg problem with multipath-tools

Chicken and egg problem with multipath-tools

[Andrei B.]

Hi, 

I've been trying to install multipath-tools on a couple of linux servers with FC storage that I work with. 
They run Linux Slackware.
I've built from source, created an installable package (Slackware style) and deployed it. 

I have discovered the following problem (chicken and egg style):

- multipathd, AFAIK, should start before LVM, because if several volumes need be assembled by LVM, they should exist before LVM starts. 
- multipathd, AFAIK, should start before any service that has it's data stored on the remote storage. 

I am not familiar [enough] with the inner workings of the startup sequence on redhat/debian like systems. 
Slackware's startup sequence is much simpler and I placed multipathd in file rc.S (which is the first one called by init) right before LVM. 

And here lies the chicken-egg problem. At this point, only the root filesystem is mounted and it is mounted read-only. It is the correct and proper startup sequence in Slackware. Filesystems are checked for errors after LVM starts and then root is remounted read-write and the rest of the filesystems are mounted. 

The problem is that multipathd wants to create a PID file under /var (which is at this time readonly). This fails and multipathd fails to start. 
It took me a lot of testing and reboots until I managed to determine this is the actual problem.

The stop-gap solution I have found, so far, is to load dm-multipath module and run multipath -v0 before LVM, thus creating the multipath devices, then later, after the filesystems are remounted rw, start multipathd. 

Can you suggest a better way of doing this, so that the system can do multipath management during the startup sequence and filesystem checks, from the creation of the devices to the starting of multipathd later ? 

Thank you. 

Re: Chicken and egg problem with multipath-tools

[Gabriel de Perthuis]

> The problem is that multipathd wants to create a PID file under /var
> (which is at this time readonly). This fails and multipathd fails to
> start.

Distros handle this by putting pidfiles in /run, a tmpfs set up at early 
boot. There might be some magic to ensure that later mounts don't shadow 
it.

Re: Chicken and egg problem with multipath-tools

[Andrei B]

Well, that would be an idea. However, multipathd has the pidfile
hardcoded. It cannot be changed at runtime, nor can it be disabled.

I know, I could compile it with a different hardcoded path, but that's
really a bad idea, because then if multipathd were to be restarted
anytime later after boot, it would then fail because of the same reason.


On 2013-03-03 01:42 PM, Gabriel de Perthuis wrote:
>> The problem is that multipathd wants to create a PID file under /var
>> (which is at this time readonly). This fails and multipathd fails to
>> start.
> Distros handle this by putting pidfiles in /run, a tmpfs set up at early 
> boot. There might be some magic to ensure that later mounts don't shadow 
> it.
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Chicken and egg problem with multipath-tools

[Hannes Reinecke]

On 03/02/2013 11:43 AM, Andrei B. wrote:
> Hi,
>
> I've been trying to install multipath-tools on a couple of linux servers with FC storage that I work with.
> They run Linux Slackware.
> I've built from source, created an installable package (Slackware style) and deployed it.
>
> I have discovered the following problem (chicken and egg style):
>
> - multipathd, AFAIK, should start before LVM, because if several volumes need be assembled by LVM,
 > they should exist before LVM starts.
> - multipathd, AFAIK, should start before any service that has it's data stored on the remote storage.
>
> I am not familiar [enough] with the inner workings of the startup sequence on redhat/debian like systems.
> Slackware's startup sequence is much simpler and I placed multipathd in file rc.S (which is the first
 > one called by init) right before LVM.
>
> And here lies the chicken-egg problem. At this point, only the root filesystem is mounted and
 > it is mounted read-only. It is the correct and proper startup 
sequence in Slackware. Filesystems
 > are checked for errors after LVM starts and then root is 
remounted read-write and the rest
 > of the filesystems are mounted.
>
> The problem is that multipathd wants to create a PID file under /var (which is at this time readonly).
 > This fails and multipathd fails to start.
> It took me a lot of testing and reboots until I managed to determine this is the actual problem.
>
Hmm. Why? multipathd can happily live without a PID file; SUSE has 
been doing this for years (for precisely the same reason).
Check the SUSE init scripts.

> The stop-gap solution I have found, so far, is to load dm-multipath module and run multipath -v0
> before LVM, thus creating the multipath devices, then later, after the filesystems are remounted
 > rw, start multipathd.
>
Bah.
Just start multipathd without creating a pid file.
Then you can check if multipathd runs via the multipathd CLI.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: Chicken and egg problem with multipath-tools

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/03/2013 09:01 PM, Andrei B wrote:
> Well, that would be an idea. However, multipathd has the pidfile 
> hardcoded. It cannot be changed at runtime, nor can it be
> disabled.

It's not pretty but if you really want the daemon running this early
(most distros without /run don't bother) you can modify the init
script to mount some writable (e.g. ramfs/tmpfs) location over the top
of the read-only /var during start up.

E.g.:

mount -t ramfs none /var/run/

> I know, I could compile it with a different hardcoded path, but
> that's really a bad idea, because then if multipathd were to be
> restarted anytime later after boot, it would then fail because of
> the same reason.

Since the daemon only manages path monitoring and failback when paths
are down most distros that need to cope with read only /var (i.e.
pre-systemd and /run) do not start multipathd until later in the boot
process when /var has been made writable.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlE0fDYACgkQ6YSQoMYUY97ZEgCgsgdZfaVC8D1ZCfhY93UPA9Sh
BEEAniaeRZTap/rbp6pbanE78DpD3myy
=ZTTO
-----END PGP SIGNATURE-----

Re: Chicken and egg problem with multipath-tools

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/02/2013 10:43 AM, Andrei B. wrote:
> Hi,
> 
> I've been trying to install multipath-tools on a couple of linux
> servers with FC storage that I work with. They run Linux
> Slackware. I've built from source, created an installable package
> (Slackware style) and deployed it.
> 
> I have discovered the following problem (chicken and egg style):
> 
> - multipathd, AFAIK, should start before LVM, because if several
> volumes need be assembled by LVM, they should exist before LVM
> starts.

The daemon is not required to discovery paths; just call "multipath"
in the initramfs or early userspace to create mapped devices. Start
the daemon later when you have writable root/var (or hack around it as
I mentioned in my other reply).

> - multipathd, AFAIK, should start before any service that has it's
> data stored on the remote storage.

No need usually. There is a small window where a path failure during
boot will not be recoverable however in over five years supporting
distros using this approach I've never seen that happen in practice
(the timing is tight - it must be after mapping the device but before
userspace gets going and it must require the daemon to intervene to
reinstate paths to carry on).

> I am not familiar [enough] with the inner workings of the startup
> sequence on redhat/debian like systems. Slackware's startup
> sequence is much simpler and I placed multipathd in file rc.S
> (which is the first one called by init) right before LVM.

Call multipath instead. Then start multipathd after your LVM devices
are up and /var is writable.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlE0fVcACgkQ6YSQoMYUY96cnQCeI/AYeKvu36kpdefUeLT4mNvw
1ycAoNQlgCK2D+fyHOjrEsPilzWMz+sn
=qWjC
-----END PGP SIGNATURE-----

Re: Chicken and egg problem with multipath-tools

[Andrei B.]

--- On Mon, 3/4/13, Hannes Reinecke <hare@suse.de> wrote:

> From: Hannes Reinecke <hare@suse.de>

> > The problem is that multipathd wants to create a PID
> file under /var (which is at this time readonly).
> > This fails and multipathd fails to start.
> > It took me a lot of testing and reboots until I managed
> to determine this is the actual problem.
> > 

> Hmm. Why? multipathd can happily live without a PID file;
> SUSE has been doing this for years (for precisely the same
> reason).
> Check the SUSE init scripts.

Official (from author's website) multipath-tools contain a multipathd that has hardcoded PIDFILE location and there is no runtime option to change its location or disable it. Both manpage and source show this. 
Maybe SUSE has a modified multipathd with added option to control PIDFILE creation. 

Re: Chicken and egg problem with multipath-tools

[Hannes Reinecke]

On 03/04/2013 12:00 PM, Andrei B. wrote:
>
>
> --- On Mon, 3/4/13, Hannes Reinecke <hare@suse.de> wrote:
>
>> From: Hannes Reinecke <hare@suse.de>
>
>>> The problem is that multipathd wants to create a PID
>> file under /var (which is at this time readonly).
>>> This fails and multipathd fails to start.
>>> It took me a lot of testing and reboots until I managed
>> to determine this is the actual problem.
>>>
>
>> Hmm. Why? multipathd can happily live without a PID file;
>> SUSE has been doing this for years (for precisely the same
>> reason).
>> Check the SUSE init scripts.
>
> Official (from author's website) multipath-tools contain a
 > multipathd that has hardcoded PIDFILE location and there is
 > no runtime option to change its location or disable it.
 > Both manpage and source show this.
> Maybe SUSE has a modified multipathd with added option to
 > control PIDFILE creation.
>

multipathd/main.c:1996
         /* Startup complete, create logfile */
         pid_rc = pidfile_create(DEFAULT_PIDFILE, daemon_pid);
         /* Ignore errors, we can live without */


No option, but multipathd doesn't _need_ to create a pidfile.
And it certainly won't abort if it cannot create one.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: Chicken and egg problem with multipath-tools

[Bryn M. Reeves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/04/2013 11:13 AM, Hannes Reinecke wrote:
> multipathd/main.c:1996 /* Startup complete, create logfile */ 
> pid_rc = pidfile_create(DEFAULT_PIDFILE, daemon_pid); /* Ignore
> errors, we can live without */
> 
> 
> No option, but multipathd doesn't _need_ to create a pidfile. And
> it certainly won't abort if it cannot create one.

Maybe an old version (prior to commit 6cda789 in May 2011)?

- -       if (pidfile_create(DEFAULT_PIDFILE, getpid())) {
- -               if (logsink)
- -                       log_thread_stop();
+       if (pidfile_create(DEFAULT_PIDFILE, getpid()))
+               /* Ignore errors, we can live without */
+               condlog(1, "failed to create pidfile");

- -               exit(1);
- -       }

If this is the case then using a more recent multipath-tools would
definitely be a good plan.

Regards,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlE0hTgACgkQ6YSQoMYUY96+8wCdG/zOh6fDAOW2LzCNitHNTiek
VzkAn3ovG1230AQMPV/uRcVk8pA1emN/
=o2M/
-----END PGP SIGNATURE-----