Re: multipath prio_callout broke from 5.2 to 5.3

Re: multipath prio_callout broke from 5.2 to 5.3

[christophe.varoqui]

John,

Redhat-shiped multipathd populates upon start-up a private mem-backed filesystem with binaries it needs.
Prio callouts in the form "$SHELL /path/to/myscript" seem to confuse the logic.
If you prio callout is of general interest, may be we can port it upstream (as a shared object).
If you are interested, please describe and post the source.

Regards,
cvaroqui

----- Mail Original -----
De: "John A. Sullivan III" <jsullivan@opensourcedevel.com>
À: "device-mapper development" <dm-devel@redhat.com>
Envoyé: Dimanche 12 Avril 2009 06h07:55 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [dm-devel] multipath prio_callout broke from 5.2 to 5.3

On Sat, 2009-04-11 at 23:54 -0400, John A. Sullivan III wrote:
> Hello, all.  We are facing a serious problem with dm-multipath after our
> upgrade.  We use a bash script to set priorities for failover.  We
> understand multipathd cannot use a bash script directly so it has been
> carefully crafted to use only internal commands and is loaded as:
> 
> prio_callout            "/bin/bash /usr/local/sbin/mpath_prio_ssi %n"
> 
> This has been working perfectly fine.  We upgraded our test lab to
> CentOS 5.3, device-mapper-multipath.x86_64 0.4.7-23.el5_3.2, kernel
> 2.6.29.1 (the 2.6.18 default causes a kernel panic with iSCSI).
> Suddenly, it is breaking.  /var/log/messages is filled with:
> 
> Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/dasd_id : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/gnbd_import : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/dasd_id
> Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/gnbd_import
> Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdc
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdd
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sde
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> 
> The first several messages are expected but not the latter ones.  If we
> run the call from the command line, e.g.,
> "/bin/bash /usr/local/sbin/mpath_prio_ssi sdc" it works perfectly fine.
> 
> What has changed and how do we fix it? I'll include a sample script
> below.  The script is dynamically created just before launching
> multipathd:
> 
> #!/bin/bash
> # if not passed any device name, return a priority of 0
> if [ -z "${1}" ];then
>         echo 0
>         exit
> fi
> 
> DEVS="lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:17f534f0-74af-e61b-a716-b8ac8e219dac-lun-0 -> ../../sdj
> lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:47c5e722-10d3-66c7-a952-d3d79732da9c-lun-0 -> ../../sdr
> lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:520e823d-342c-6668-9477-fad130b148d7-lun-0 -> ../../sdn"
> 
> LIST="172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->99
> 172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->49
> 172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->24"
> 
> FOUND=0
> IFSORIG=${IFS}
> IFS=$'\n'
> for LINE in ${DEVS}
> do
>         ENTRY=${LINE%/${1}}
>         if [ ${#ENTRY} -ne ${#LINE} ];then # We found the line
>                 FOUND=1
>                 break
>         fi
> done
> if [ "$FOUND" = "0" ];then  # This is not an iSCSI device
>         echo 0
>         exit
> fi
> DEV="${ENTRY##* ip-}"
> #DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
> #DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
> DEV="${DEV%%-lun-[0-9]* ->*}"
> PRIORITY=0
> for LINE in ${LIST}
> do
>         DISK=${LINE%->*}
>         if [ "${DEV}" = "${DISK}" ];then
>                 PRIORITY="${LINE##*->}"
>                 break
>         fi
> done
> echo ${PRIORITY}
> 
> I did notice the semantics of /dev/disk/by-path changed and we adapted
> to that.  We were planning to move this to production on Thursday so
> this has thrown a huge spanner in the works.  Any help would be greatly
> appreciated.  Thanks - John

I've just notice that my console is filled with:

/bin/bash: /usr/local/sbin/mpath_prio_ssi: No such file or directory

but it is indeed there and owned by root and executable.  I've quintuple
checked! Has multipathd been changed so it cannot read anything from
disk even if invoked from within bash? Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

Thank you.  I'll detail our script and the logic behind it in a separate
email in case it is helpful to others.

In the meantime, we have a critical problem where the script which was
working perfectly in 5.2 is now broken in 5.3.  Is there any way to
deconfuse the 5.3 multipathd or any other immediate solution? - John

On Sun, 2009-04-12 at 09:13 +0200, christophe.varoqui@free.fr wrote:
> John,
> 
> Redhat-shiped multipathd populates upon start-up a private mem-backed filesystem with binaries it needs.
> Prio callouts in the form "$SHELL /path/to/myscript" seem to confuse the logic.
> If you prio callout is of general interest, may be we can port it upstream (as a shared object).
> If you are interested, please describe and post the source.
> 
> Regards,
> cvaroqui
> 
> ----- Mail Original -----
> De: "John A. Sullivan III" <jsullivan@opensourcedevel.com>
> À: "device-mapper development" <dm-devel@redhat.com>
> Envoyé: Dimanche 12 Avril 2009 06h07:55 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
> Objet: Re: [dm-devel] multipath prio_callout broke from 5.2 to 5.3
> 
> On Sat, 2009-04-11 at 23:54 -0400, John A. Sullivan III wrote:
> > Hello, all.  We are facing a serious problem with dm-multipath after our
> > upgrade.  We use a bash script to set priorities for failover.  We
> > understand multipathd cannot use a bash script directly so it has been
> > carefully crafted to use only internal commands and is loaded as:
> > 
> > prio_callout            "/bin/bash /usr/local/sbin/mpath_prio_ssi %n"
> > 
> > This has been working perfectly fine.  We upgraded our test lab to
> > CentOS 5.3, device-mapper-multipath.x86_64 0.4.7-23.el5_3.2, kernel
> > 2.6.29.1 (the 2.6.18 default causes a kernel panic with iSCSI).
> > Suddenly, it is breaking.  /var/log/messages is filled with:
> > 
> > Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/dasd_id : No such file or directory
> > Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/gnbd_import : No such file or directory
> > Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/dasd_id
> > Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
> > Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/gnbd_import
> > Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory
> > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdc
> > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdd
> > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sde
> > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > 
> > The first several messages are expected but not the latter ones.  If we
> > run the call from the command line, e.g.,
> > "/bin/bash /usr/local/sbin/mpath_prio_ssi sdc" it works perfectly fine.
> > 
> > What has changed and how do we fix it? I'll include a sample script
> > below.  The script is dynamically created just before launching
> > multipathd:
> > 
> > #!/bin/bash
> > # if not passed any device name, return a priority of 0
> > if [ -z "${1}" ];then
> >         echo 0
> >         exit
> > fi
> > 
> > DEVS="lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:17f534f0-74af-e61b-a716-b8ac8e219dac-lun-0 -> ../../sdj
> > lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:47c5e722-10d3-66c7-a952-d3d79732da9c-lun-0 -> ../../sdr
> > lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:520e823d-342c-6668-9477-fad130b148d7-lun-0 -> ../../sdn"
> > 
> > LIST="172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->99
> > 172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->49
> > 172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->24"
> > 
> > FOUND=0
> > IFSORIG=${IFS}
> > IFS=$'\n'
> > for LINE in ${DEVS}
> > do
> >         ENTRY=${LINE%/${1}}
> >         if [ ${#ENTRY} -ne ${#LINE} ];then # We found the line
> >                 FOUND=1
> >                 break
> >         fi
> > done
> > if [ "$FOUND" = "0" ];then  # This is not an iSCSI device
> >         echo 0
> >         exit
> > fi
> > DEV="${ENTRY##* ip-}"
> > #DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
> > #DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
> > DEV="${DEV%%-lun-[0-9]* ->*}"
> > PRIORITY=0
> > for LINE in ${LIST}
> > do
> >         DISK=${LINE%->*}
> >         if [ "${DEV}" = "${DISK}" ];then
> >                 PRIORITY="${LINE##*->}"
> >                 break
> >         fi
> > done
> > echo ${PRIORITY}
> > 
> > I did notice the semantics of /dev/disk/by-path changed and we adapted
> > to that.  We were planning to move this to production on Thursday so
> > this has thrown a huge spanner in the works.  Any help would be greatly
> > appreciated.  Thanks - John
> 
> I've just notice that my console is filled with:
> 
> /bin/bash: /usr/local/sbin/mpath_prio_ssi: No such file or directory
> 
> but it is indeed there and owned by root and executable.  I've quintuple
> checked! Has multipathd been changed so it cannot read anything from
> disk even if invoked from within bash? Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[Benjamin Marzinski]

On Mon, Apr 13, 2009 at 05:00:05AM -0400, John A. Sullivan III wrote:
> Thank you.  I'll detail our script and the logic behind it in a separate
> email in case it is helpful to others.
> 
> In the meantime, we have a critical problem where the script which was
> working perfectly in 5.2 is now broken in 5.3.  Is there any way to
> deconfuse the 5.3 multipathd or any other immediate solution? - John

What christophe said is correct. In RHEL 5.3, multipath started copying
all of the necessary callouts into it own private namespace. It scans
through your config file, and pulls out all the binaries.  However,
there are two problems that are affecting you.  First, it only pulls the
command, "/bin/bash" in you case, not the arguments, which for
you include a script to run.  Second, it's private namespace only
consists of /sbin, /bin, /tmp, a couple of virtual filesystems, like
/proc and /sys (well, actually there are a couple of others, like /etc,
that multipath needs to start up, but you shouldn't rely on them being
there all the time, since you can lose access to them if the device
they're on goes down)

There are two ways to deal with this.  First is to rewrite the
prioritizer in C.  I realize that this is a pain, but it will be
necessary to run on RHEL6 and new fedora machines, which use upstream's
prio functions instead of callout binaries.

The second, quicker way is to move your callout to /sbin and add a dummy
device section to make sure it gets picked up.

devices {
...
	device {
		vendor       "dummy"
		product      "dummy"
		prio_callout "/sbin/mpath_prio_ssi"
	}
}

This will cause multipathd to copy your script into the private
namespace, and everything should work, with one exception.

bash is not a statically linked executable.  It links to libraries,
and multipathd doesn't make its own copies of them.  Under normal
operation this will work (/lib is also in multipathd's
private namespace). However, if you lose access to /lib, bash won't
work, and multipathd won't be able to restore access to your devices.
If you aren't planning on multipathing / or /lib you might choose to
ignore this (The exact same problem exists in 5.2).

I don't believe that there is a statically linked shell in RHEL 5.
This is another reason to convert your callout to a C program. Or
you can recompile bash with static linking.

-Ben

> On Sun, 2009-04-12 at 09:13 +0200, christophe.varoqui@free.fr wrote:
> > John,
> > 
> > Redhat-shiped multipathd populates upon start-up a private mem-backed filesystem with binaries it needs.
> > Prio callouts in the form "$SHELL /path/to/myscript" seem to confuse the logic.
> > If you prio callout is of general interest, may be we can port it upstream (as a shared object).
> > If you are interested, please describe and post the source.
> > 
> > Regards,
> > cvaroqui
> > 
> > ----- Mail Original -----
> > De: "John A. Sullivan III" <jsullivan@opensourcedevel.com>
> > À: "device-mapper development" <dm-devel@redhat.com>
> > Envoyé: Dimanche 12 Avril 2009 06h07:55 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
> > Objet: Re: [dm-devel] multipath prio_callout broke from 5.2 to 5.3
> > 
> > On Sat, 2009-04-11 at 23:54 -0400, John A. Sullivan III wrote:
> > > Hello, all.  We are facing a serious problem with dm-multipath after our
> > > upgrade.  We use a bash script to set priorities for failover.  We
> > > understand multipathd cannot use a bash script directly so it has been
> > > carefully crafted to use only internal commands and is loaded as:
> > > 
> > > prio_callout            "/bin/bash /usr/local/sbin/mpath_prio_ssi %n"
> > > 
> > > This has been working perfectly fine.  We upgraded our test lab to
> > > CentOS 5.3, device-mapper-multipath.x86_64 0.4.7-23.el5_3.2, kernel
> > > 2.6.29.1 (the 2.6.18 default causes a kernel panic with iSCSI).
> > > Suddenly, it is breaking.  /var/log/messages is filled with:
> > > 
> > > Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/dasd_id : No such file or directory
> > > Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/gnbd_import : No such file or directory
> > > Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/dasd_id
> > > Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
> > > Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/gnbd_import
> > > Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory
> > > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdc
> > > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdd
> > > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > > Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sde
> > > Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> > > 
> > > The first several messages are expected but not the latter ones.  If we
> > > run the call from the command line, e.g.,
> > > "/bin/bash /usr/local/sbin/mpath_prio_ssi sdc" it works perfectly fine.
> > > 
> > > What has changed and how do we fix it? I'll include a sample script
> > > below.  The script is dynamically created just before launching
> > > multipathd:
> > > 
> > > #!/bin/bash
> > > # if not passed any device name, return a priority of 0
> > > if [ -z "${1}" ];then
> > >         echo 0
> > >         exit
> > > fi
> > > 
> > > DEVS="lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:17f534f0-74af-e61b-a716-b8ac8e219dac-lun-0 -> ../../sdj
> > > lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:47c5e722-10d3-66c7-a952-d3d79732da9c-lun-0 -> ../../sdr
> > > lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:520e823d-342c-6668-9477-fad130b148d7-lun-0 -> ../../sdn"
> > > 
> > > LIST="172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->99
> > > 172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->49
> > > 172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->24"
> > > 
> > > FOUND=0
> > > IFSORIG=${IFS}
> > > IFS=$'\n'
> > > for LINE in ${DEVS}
> > > do
> > >         ENTRY=${LINE%/${1}}
> > >         if [ ${#ENTRY} -ne ${#LINE} ];then # We found the line
> > >                 FOUND=1
> > >                 break
> > >         fi
> > > done
> > > if [ "$FOUND" = "0" ];then  # This is not an iSCSI device
> > >         echo 0
> > >         exit
> > > fi
> > > DEV="${ENTRY##* ip-}"
> > > #DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
> > > #DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
> > > DEV="${DEV%%-lun-[0-9]* ->*}"
> > > PRIORITY=0
> > > for LINE in ${LIST}
> > > do
> > >         DISK=${LINE%->*}
> > >         if [ "${DEV}" = "${DISK}" ];then
> > >                 PRIORITY="${LINE##*->}"
> > >                 break
> > >         fi
> > > done
> > > echo ${PRIORITY}
> > > 
> > > I did notice the semantics of /dev/disk/by-path changed and we adapted
> > > to that.  We were planning to move this to production on Thursday so
> > > this has thrown a huge spanner in the works.  Any help would be greatly
> > > appreciated.  Thanks - John
> > 
> > I've just notice that my console is filled with:
> > 
> > /bin/bash: /usr/local/sbin/mpath_prio_ssi: No such file or directory
> > 
> > but it is indeed there and owned by root and executable.  I've quintuple
> > checked! Has multipathd been changed so it cannot read anything from
> > disk even if invoked from within bash? Thanks - John
> -- 
> John A. Sullivan III
> Open Source Development Corporation
> +1 207-985-7880
> jsullivan@opensourcedevel.com
> 
> http://www.spiritualoutreach.com
> Making Christianity intelligible to secular society
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

On Mon, 2009-04-13 at 13:57 -0500, Benjamin Marzinski wrote:
> On Mon, Apr 13, 2009 at 05:00:05AM -0400, John A. Sullivan III wrote:
> > Thank you.  I'll detail our script and the logic behind it in a separate
> > email in case it is helpful to others.
> > 
> > In the meantime, we have a critical problem where the script which was
> > working perfectly in 5.2 is now broken in 5.3.  Is there any way to
> > deconfuse the 5.3 multipathd or any other immediate solution? - John
> 
> What christophe said is correct. In RHEL 5.3, multipath started copying
> all of the necessary callouts into it own private namespace. It scans
> through your config file, and pulls out all the binaries.  However,
> there are two problems that are affecting you.  First, it only pulls the
> command, "/bin/bash" in you case, not the arguments, which for
> you include a script to run.  Second, it's private namespace only
> consists of /sbin, /bin, /tmp, a couple of virtual filesystems, like
> /proc and /sys (well, actually there are a couple of others, like /etc,
> that multipath needs to start up, but you shouldn't rely on them being
> there all the time, since you can lose access to them if the device
> they're on goes down)
> 
> There are two ways to deal with this.  First is to rewrite the
> prioritizer in C.  I realize that this is a pain, but it will be
> necessary to run on RHEL6 and new fedora machines, which use upstream's
> prio functions instead of callout binaries.
> 
> The second, quicker way is to move your callout to /sbin and add a dummy
> device section to make sure it gets picked up.
> 
> devices {
> ...
> 	device {
> 		vendor       "dummy"
> 		product      "dummy"
> 		prio_callout "/sbin/mpath_prio_ssi"
> 	}
> }
> 
> This will cause multipathd to copy your script into the private
> namespace, and everything should work, with one exception.
> 
> bash is not a statically linked executable.  It links to libraries,
> and multipathd doesn't make its own copies of them.  Under normal
> operation this will work (/lib is also in multipathd's
> private namespace). However, if you lose access to /lib, bash won't
> work, and multipathd won't be able to restore access to your devices.
> If you aren't planning on multipathing / or /lib you might choose to
> ignore this (The exact same problem exists in 5.2).
> 
> I don't believe that there is a statically linked shell in RHEL 5.
> This is another reason to convert your callout to a C program. Or
> you can recompile bash with static linking.
> 
> -<snip>
Thanks very much for the explanation.  If I understand correctly, 5.2
also copied into a ramfs but not a separate namespace and that's why it
worked in 5.2?

In any event, we attempted to implement the less preferred method for
the sake of time right now (none of us are particularly adept at C and
are not sure how we'd feed the configuration file if it is not safe to
pull files from disk).  We moved mpath_prio_ssi to /sbin and called it
directly in multipath.conf, i.e.,
prio_callout            "/sbin/mpath_prio_ssi %n"

It still does not work but this time we get:
Apr 13 15:33:15 kvm01 multipathd: error calling out /sbin/mpath_prio_ssi
sdq
Apr 13 15:33:15 kvm01 multipathd: /sbin/mpath_prio_ssi exitted with 255

If we revert to
prio_callout            "/bin/bash /sbin/mpath_prio_ssi %n"
we return to:
Apr 13 15:34:43 kvm01 multipathd: error calling
out /bin/bash /sbin/mpath_prio_ssi sdc
Apr 13 15:34:43 kvm01 multipathd: /bin/bash exitted with 127

We thought the script might need an explicit exit code so we changed
everything to exit 0 but that did not fix the problem.  Any idea why we
are getting this 255 error? Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[Benjamin Marzinski]

On Mon, Apr 13, 2009 at 03:56:05PM -0400, John A. Sullivan III wrote:
> On Mon, 2009-04-13 at 13:57 -0500, Benjamin Marzinski wrote:
> > On Mon, Apr 13, 2009 at 05:00:05AM -0400, John A. Sullivan III wrote:
> > > Thank you.  I'll detail our script and the logic behind it in a separate
> > > email in case it is helpful to others.
> > > 
> > > In the meantime, we have a critical problem where the script which was
> > > working perfectly in 5.2 is now broken in 5.3.  Is there any way to
> > > deconfuse the 5.3 multipathd or any other immediate solution? - John
> > 
> > What christophe said is correct. In RHEL 5.3, multipath started copying
> > all of the necessary callouts into it own private namespace. It scans
> > through your config file, and pulls out all the binaries.  However,
> > there are two problems that are affecting you.  First, it only pulls the
> > command, "/bin/bash" in you case, not the arguments, which for
> > you include a script to run.  Second, it's private namespace only
> > consists of /sbin, /bin, /tmp, a couple of virtual filesystems, like
> > /proc and /sys (well, actually there are a couple of others, like /etc,
> > that multipath needs to start up, but you shouldn't rely on them being
> > there all the time, since you can lose access to them if the device
> > they're on goes down)
> > 
> > There are two ways to deal with this.  First is to rewrite the
> > prioritizer in C.  I realize that this is a pain, but it will be
> > necessary to run on RHEL6 and new fedora machines, which use upstream's
> > prio functions instead of callout binaries.
> > 
> > The second, quicker way is to move your callout to /sbin and add a dummy
> > device section to make sure it gets picked up.
> > 
> > devices {
> > ...
> > 	device {
> > 		vendor       "dummy"
> > 		product      "dummy"
> > 		prio_callout "/sbin/mpath_prio_ssi"
> > 	}
> > }
> > 
> > This will cause multipathd to copy your script into the private
> > namespace, and everything should work, with one exception.
> > 
> > bash is not a statically linked executable.  It links to libraries,
> > and multipathd doesn't make its own copies of them.  Under normal
> > operation this will work (/lib is also in multipathd's
> > private namespace). However, if you lose access to /lib, bash won't
> > work, and multipathd won't be able to restore access to your devices.
> > If you aren't planning on multipathing / or /lib you might choose to
> > ignore this (The exact same problem exists in 5.2).
> > 
> > I don't believe that there is a statically linked shell in RHEL 5.
> > This is another reason to convert your callout to a C program. Or
> > you can recompile bash with static linking.
> > 
> > -<snip>
> Thanks very much for the explanation.  If I understand correctly, 5.2
> also copied into a ramfs but not a separate namespace and that's why it
> worked in 5.2?

Not quite.  multipathd had a private namespace in 5.2. but it didn't
unmount all of the unnecessary mountpoints.  This was changed in 5.3 for
two reasons.

1. Otherwise if you unmounted a filesystem that had been mounted before
you started multipathd, and then tried to remove the device, you
couldn't, since the private namespace still had it open.

2. To catch configurations like yours.  In RHEL 5.2, multipathd started
up and worked, but if you ever lost access to /usr/local/sbin,
multipathd would stop working.  By unmounting the filesystems that could
potentially disappear (or at least most of them), you can force people
to do things in a way that makes multipathd fault tolerant.

In rhel 5.2, multipath didn't make a private, in-memory copy of your
script. It just used the one on the regular filesystem, which is the
very thing that the private namespace was trying to avoid.

> In any event, we attempted to implement the less preferred method for
> the sake of time right now (none of us are particularly adept at C and
> are not sure how we'd feed the configuration file if it is not safe to
> pull files from disk).  We moved mpath_prio_ssi to /sbin and called it
> directly in multipath.conf, i.e.,
> prio_callout            "/sbin/mpath_prio_ssi %n"

Sorry for the confusion.  You still need to call your script with
/bin/bash in your actual device section, just like you originally were.
But you also need a dummy device section to cause multipathd to pull
that script into the private namespace. In the dummy device section, you
need to reference the script directly. This is because multipathd only
pulls in commands, not their arguments (even if the argument is a script
to run).  When I tested this setup before my first email, my
multipath.conf devices section looked like this:

devices {
        device {
                vendor          "WINSYS"
                product         "SF2372"
                path_grouping_policy    group_by_prio
                prio_callout            "/bin/bash /sbin/mpath_prio_one"
        }
        device {
                vendor          "dummy"
                product         "dummy"
                prio_callout    "/sbin/mpath_prio_one"
        }
}

mpath_prio_one is a bash script that just echos 1.

-Ben
> 
> It still does not work but this time we get:
> Apr 13 15:33:15 kvm01 multipathd: error calling out /sbin/mpath_prio_ssi
> sdq
> Apr 13 15:33:15 kvm01 multipathd: /sbin/mpath_prio_ssi exitted with 255
> 
> If we revert to
> prio_callout            "/bin/bash /sbin/mpath_prio_ssi %n"
> we return to:
> Apr 13 15:34:43 kvm01 multipathd: error calling
> out /bin/bash /sbin/mpath_prio_ssi sdc
> Apr 13 15:34:43 kvm01 multipathd: /bin/bash exitted with 127
> 
> We thought the script might need an explicit exit code so we changed
> everything to exit 0 but that did not fix the problem.  Any idea why we
> are getting this 255 error? Thanks - John
> -- 
> John A. Sullivan III
> Open Source Development Corporation
> +1 207-985-7880
> jsullivan@opensourcedevel.com
> 
> http://www.spiritualoutreach.com
> Making Christianity intelligible to secular society
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath prio_callout broke from 5.2 to 5.3

[Benjamin Marzinski]

Just to clear one more thing up. There is nothing special about the
"dummy" strings in the dummy section.  You could use 
"this is gross hack" and it would work just as well.  The only thing is
to choose a string that you can be reasonably sure that storage hardware
makers won't use for their vendor and product names on their devices.

-Ben

> devices {
>         device {
>                 vendor          "WINSYS"
>                 product         "SF2372"
>                 path_grouping_policy    group_by_prio
>                 prio_callout            "/bin/bash /sbin/mpath_prio_one"
>         }
>         device {
>                 vendor          "dummy"
>                 product         "dummy"
>                 prio_callout    "/sbin/mpath_prio_one"
>         }
> }

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

On Mon, 2009-04-13 at 15:37 -0500, Benjamin Marzinski wrote:
> On Mon, Apr 13, 2009 at 03:56:05PM -0400, John A. Sullivan III wrote:
> > On Mon, 2009-04-13 at 13:57 -0500, Benjamin Marzinski wrote:
> > > On Mon, Apr 13, 2009 at 05:00:05AM -0400, John A. Sullivan III wrote:
> > > > Thank you.  I'll detail our script and the logic behind it in a separate
> > > > email in case it is helpful to others.
> > > > 
> > > > In the meantime, we have a critical problem where the script which was
> > > > working perfectly in 5.2 is now broken in 5.3.  Is there any way to
> > > > deconfuse the 5.3 multipathd or any other immediate solution? - John
> > > 
> > > What christophe said is correct. In RHEL 5.3, multipath started copying
> > > all of the necessary callouts into it own private namespace. It scans
> > > through your config file, and pulls out all the binaries.  However,
> > > there are two problems that are affecting you.  First, it only pulls the
> > > command, "/bin/bash" in you case, not the arguments, which for
> > > you include a script to run.  Second, it's private namespace only
> > > consists of /sbin, /bin, /tmp, a couple of virtual filesystems, like
> > > /proc and /sys (well, actually there are a couple of others, like /etc,
> > > that multipath needs to start up, but you shouldn't rely on them being
> > > there all the time, since you can lose access to them if the device
> > > they're on goes down)
> > > 
> > > There are two ways to deal with this.  First is to rewrite the
> > > prioritizer in C.  I realize that this is a pain, but it will be
> > > necessary to run on RHEL6 and new fedora machines, which use upstream's
> > > prio functions instead of callout binaries.
> > > 
> > > The second, quicker way is to move your callout to /sbin and add a dummy
> > > device section to make sure it gets picked up.
> > > 
> > > devices {
> > > ...
> > > 	device {
> > > 		vendor       "dummy"
> > > 		product      "dummy"
> > > 		prio_callout "/sbin/mpath_prio_ssi"
> > > 	}
> > > }
> > > 
> > > This will cause multipathd to copy your script into the private
> > > namespace, and everything should work, with one exception.
> > > 
> > > bash is not a statically linked executable.  It links to libraries,
> > > and multipathd doesn't make its own copies of them.  Under normal
> > > operation this will work (/lib is also in multipathd's
> > > private namespace). However, if you lose access to /lib, bash won't
> > > work, and multipathd won't be able to restore access to your devices.
> > > If you aren't planning on multipathing / or /lib you might choose to
> > > ignore this (The exact same problem exists in 5.2).
> > > 
> > > I don't believe that there is a statically linked shell in RHEL 5.
> > > This is another reason to convert your callout to a C program. Or
> > > you can recompile bash with static linking.
> > > 
> > > -<snip>
> > Thanks very much for the explanation.  If I understand correctly, 5.2
> > also copied into a ramfs but not a separate namespace and that's why it
> > worked in 5.2?
> 
> Not quite.  multipathd had a private namespace in 5.2. but it didn't
> unmount all of the unnecessary mountpoints.  This was changed in 5.3 for
> two reasons.
> 
> 1. Otherwise if you unmounted a filesystem that had been mounted before
> you started multipathd, and then tried to remove the device, you
> couldn't, since the private namespace still had it open.
> 
> 2. To catch configurations like yours.  In RHEL 5.2, multipathd started
> up and worked, but if you ever lost access to /usr/local/sbin,
> multipathd would stop working.  By unmounting the filesystems that could
> potentially disappear (or at least most of them), you can force people
> to do things in a way that makes multipathd fault tolerant.
> 
> In rhel 5.2, multipath didn't make a private, in-memory copy of your
> script. It just used the one on the regular filesystem, which is the
> very thing that the private namespace was trying to avoid.
> 
> > In any event, we attempted to implement the less preferred method for
> > the sake of time right now (none of us are particularly adept at C and
> > are not sure how we'd feed the configuration file if it is not safe to
> > pull files from disk).  We moved mpath_prio_ssi to /sbin and called it
> > directly in multipath.conf, i.e.,
> > prio_callout            "/sbin/mpath_prio_ssi %n"
> 
> Sorry for the confusion.  You still need to call your script with
> /bin/bash in your actual device section, just like you originally were.
> But you also need a dummy device section to cause multipathd to pull
> that script into the private namespace. In the dummy device section, you
> need to reference the script directly. This is because multipathd only
> pulls in commands, not their arguments (even if the argument is a script
> to run).  When I tested this setup before my first email, my
> multipath.conf devices section looked like this:
> 
> devices {
>         device {
>                 vendor          "WINSYS"
>                 product         "SF2372"
>                 path_grouping_policy    group_by_prio
>                 prio_callout            "/bin/bash /sbin/mpath_prio_one"
>         }
>         device {
>                 vendor          "dummy"
>                 product         "dummy"
>                 prio_callout    "/sbin/mpath_prio_one"
>         }
> }
> 
> mpath_prio_one is a bash script that just echos 1.
> 
> -Ben
Ah, got it.  It worked. Thanks very, very much - John
> > 
> > It still does not work but this time we get:
> > Apr 13 15:33:15 kvm01 multipathd: error calling out /sbin/mpath_prio_ssi
> > sdq
> > Apr 13 15:33:15 kvm01 multipathd: /sbin/mpath_prio_ssi exitted with 255
> > 
> > If we revert to
> > prio_callout            "/bin/bash /sbin/mpath_prio_ssi %n"
> > we return to:
> > Apr 13 15:34:43 kvm01 multipathd: error calling
> > out /bin/bash /sbin/mpath_prio_ssi sdc
> > Apr 13 15:34:43 kvm01 multipathd: /bin/bash exitted with 127
> > 
> > We thought the script might need an explicit exit code so we changed
> > everything to exit 0 but that did not fix the problem.  Any idea why we
> > are getting this 255 error? Thanks - John
> > -- 
> > John A. Sullivan III
> > Open Source Development Corporation
> > +1 207-985-7880
> > jsullivan@opensourcedevel.com
> > 
> > http://www.spiritualoutreach.com
> > Making Christianity intelligible to secular society
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[Ty! Boyack]

This thread has been great information since I'm looking at the same 
type of thing.  However it raises a couple of (slightly off-topic) 
questions for me. 

My recent upgrade to fedora 10 broke my prio_callout bash script just 
like you described, but my getuid_callout (a bash script that calls 
udevadm, grep, sed, and iscsi_id) runs just fine.  Are the two callouts 
handled differently?

Also, is there an easy way to know what tools are in the private 
namespace already?  My prio_callout script calls two other binaries: 
/sbin/udevadm and grep.  If I go to C-code, handling grep's functions 
myself is no problem, but I'm not confident about re-implementing what 
udevadm does.  Can I assume that since /sbin/udevadm is in /sbin that it 
will be available to call via exec()?  Or would I be right back where we 
are with the bash scripting, as in having to include a dummy device as 
you described?

Finally, in my case I've got two redundant iscsi networks, one is 1GbE, 
and the other is 10GbE.  In the past I've always had symetric paths, so 
I've used round-robin/multibus.  But I want to focus traffic on the 
10GbE path, so I was looking at using the prio callout.  Is this even 
necessary?  Or will round-robin/multibus take full advantage of both 
paths?  I can see round-robin on that setup resulting in either around 
11Gbps or 2 Gbps, depending on whether the slower link becomes a 
limiting factor.  I'm just wondering if I am making things unnecessarily 
complex by trying to set priorities.

Thanks for all the help.

-Ty!

-- 
-===========================-
  Ty! Boyack
  NREL Unix Network Manager
  ty@nrel.colostate.edu
  (970) 491-1186
-===========================-

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

On Thu, 2009-04-23 at 12:08 -0600, Ty! Boyack wrote:
> This thread has been great information since I'm looking at the same 
> type of thing.  However it raises a couple of (slightly off-topic) 
> questions for me. 
> 
> My recent upgrade to fedora 10 broke my prio_callout bash script just 
> like you described, but my getuid_callout (a bash script that calls 
> udevadm, grep, sed, and iscsi_id) runs just fine.  Are the two callouts 
> handled differently?
> 
> Also, is there an easy way to know what tools are in the private 
> namespace already?  My prio_callout script calls two other binaries: 
> /sbin/udevadm and grep.  If I go to C-code, handling grep's functions 
> myself is no problem, but I'm not confident about re-implementing what 
> udevadm does.  Can I assume that since /sbin/udevadm is in /sbin that it 
> will be available to call via exec()?  Or would I be right back where we 
> are with the bash scripting, as in having to include a dummy device as 
> you described?
> 
> Finally, in my case I've got two redundant iscsi networks, one is 1GbE, 
> and the other is 10GbE.  In the past I've always had symetric paths, so 
> I've used round-robin/multibus.  But I want to focus traffic on the 
> 10GbE path, so I was looking at using the prio callout.  Is this even 
> necessary?  Or will round-robin/multibus take full advantage of both 
> paths?  I can see round-robin on that setup resulting in either around 
> 11Gbps or 2 Gbps, depending on whether the slower link becomes a 
> limiting factor.  I'm just wondering if I am making things unnecessarily 
> complex by trying to set priorities.
> 
> Thanks for all the help.
> 
> -Ty!
> 
I can't answer the questions regarding the internals.  I did make sure
my bash scripts called not external applications and I placed everything
in /sbin.

I did find I was able to pick and choose which connections had which
priorities - that was the whole purpose of my script.  In my case, there
were many networks and I wanted prioritized failover to try to balance
the load across interfaces and keep failover traffic on the same switch
rather than crossing a bonded link to another switch.  I did it by cross
referencing the mappings in /dev/disk/by-path with a prioritized list of
mappings.  I believe I posted the entire setup in an earlier e-mail.  If
you'd like, I can post the details again.

As I reread your post a little more closely, I wonder if using multibus
as you describe will not slow you down to the lowest common denominator.
I know when I tested with RAID0 across several interfaces to load
balance traffic (this seemed to give better average performance across a
wide range of I/O patterns than multi-bus with varying rr_min_io
settings), I had three e1000e NICs and one on board NIC. When I replaced
the on-board with another e1000e, I saw a substantial performance
improvement.  I don't know if that will be your experience for sure but
pass it along as a caveat. Hope this helps - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[Ty! Boyack]

Thanks John.  I think that you and I are doing very similar things, but 
there is one thing in your technique that would cause me problems.  I 
start multipathd at system boot time, but my iscsi devices get connected 
(and disconnected) later as the system runs, so the list you generate 
when you start multipathd (to map /dev/sdX names to their 
/dev/disk/by-path counterpart) is not available when multipathd starts 
for me. 

However, it seems we are indeed facing the same issue:  We want to be 
able to specify path priorities based on some criteria in the 
/dev/disk/by-path name.  I usually get this from '/sbin/udevadm info 
--query=env --name=/dev/sdX', and in fact I usually only care about the 
ID_PATH variable out of that.  Would you also be able to get the 
information you need out of this type of output? (If the 'env' query is 
not enough, maybe 'all' would be better)

Ben mentioned that if this was something that was a common need that 
maybe a shared object could be added upstream to make this a general 
solution.  I'm thinking that a module could be written that would do 
this type of query on the device, and then look up the priority in a 
simple expression file that might look something like:

<regular expression><priority>

In my case I could just look for something like /ID_PATH=ip-10.0.x/ to 
see if it is on the particular network in question, and then set the 
priority.  You might search for entire iqn names.  But this would be 
flexible to allow anyone to set priority based on the udev parameters of 
vendor, model, serial numbers, iqn path, etc.

I don't know if it is feasible to query udev in this environment -- 
perhaps someone closer to the internals could answer that.  (It looks 
like it could also be pulled from /sys, but I'm not too familiar with 
that structure, and we would need to make sure it was not too dependent 
on kernel changes which might change that structure).

Thoughts?

-Ty!

John A. Sullivan III wrote:
> On Thu, 2009-04-23 at 12:08 -0600, Ty! Boyack wrote:
>   
>> This thread has been great information since I'm looking at the same 
>> type of thing.  However it raises a couple of (slightly off-topic) 
>> questions for me. 
>>
>> My recent upgrade to fedora 10 broke my prio_callout bash script just 
>> like you described, but my getuid_callout (a bash script that calls 
>> udevadm, grep, sed, and iscsi_id) runs just fine.  Are the two callouts 
>> handled differently?
>>
>> Also, is there an easy way to know what tools are in the private 
>> namespace already?  My prio_callout script calls two other binaries: 
>> /sbin/udevadm and grep.  If I go to C-code, handling grep's functions 
>> myself is no problem, but I'm not confident about re-implementing what 
>> udevadm does.  Can I assume that since /sbin/udevadm is in /sbin that it 
>> will be available to call via exec()?  Or would I be right back where we 
>> are with the bash scripting, as in having to include a dummy device as 
>> you described?
>>
>> Finally, in my case I've got two redundant iscsi networks, one is 1GbE, 
>> and the other is 10GbE.  In the past I've always had symetric paths, so 
>> I've used round-robin/multibus.  But I want to focus traffic on the 
>> 10GbE path, so I was looking at using the prio callout.  Is this even 
>> necessary?  Or will round-robin/multibus take full advantage of both 
>> paths?  I can see round-robin on that setup resulting in either around 
>> 11Gbps or 2 Gbps, depending on whether the slower link becomes a 
>> limiting factor.  I'm just wondering if I am making things unnecessarily 
>> complex by trying to set priorities.
>>
>> Thanks for all the help.
>>
>> -Ty!
>>
>>     
> I can't answer the questions regarding the internals.  I did make sure
> my bash scripts called not external applications and I placed everything
> in /sbin.
>
> I did find I was able to pick and choose which connections had which
> priorities - that was the whole purpose of my script.  In my case, there
> were many networks and I wanted prioritized failover to try to balance
> the load across interfaces and keep failover traffic on the same switch
> rather than crossing a bonded link to another switch.  I did it by cross
> referencing the mappings in /dev/disk/by-path with a prioritized list of
> mappings.  I believe I posted the entire setup in an earlier e-mail.  If
> you'd like, I can post the details again.
>
> As I reread your post a little more closely, I wonder if using multibus
> as you describe will not slow you down to the lowest common denominator.
> I know when I tested with RAID0 across several interfaces to load
> balance traffic (this seemed to give better average performance across a
> wide range of I/O patterns than multi-bus with varying rr_min_io
> settings), I had three e1000e NICs and one on board NIC. When I replaced
> the on-board with another e1000e, I saw a substantial performance
> improvement.  I don't know if that will be your experience for sure but
> pass it along as a caveat. Hope this helps - John
>   


-- 
-===========================-
  Ty! Boyack
  NREL Unix Network Manager
  ty@nrel.colostate.edu
  (970) 491-1186
-===========================-

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

On Thu, 2009-04-23 at 21:09 -0600, Ty! Boyack wrote:
> Thanks John.  I think that you and I are doing very similar things, but 
> there is one thing in your technique that would cause me problems.  I 
> start multipathd at system boot time, but my iscsi devices get connected 
> (and disconnected) later as the system runs, so the list you generate 
> when you start multipathd (to map /dev/sdX names to their 
> /dev/disk/by-path counterpart) is not available when multipathd starts 
> for me. 
We do intentionally delay multipathd until later in the process.  We
first add some logic to ensure the SAN is available (for example when
bootstrapping the entire environment), then setup the sessions, then
activate multipath, then set the read ahead buffer and finally setup the
final devices, e.g., RAID0, encryption, etc. (although we have found
encryption intolerably slow on iSCSI).
> 
> However, it seems we are indeed facing the same issue:  We want to be 
> able to specify path priorities based on some criteria in the 
> /dev/disk/by-path name.  I usually get this from '/sbin/udevadm info 
> --query=env --name=/dev/sdX', and in fact I usually only care about the 
> ID_PATH variable out of that.  Would you also be able to get the 
> information you need out of this type of output? (If the 'env' query is 
> not enough, maybe 'all' would be better)
It sounds like you know much more about this than I do.  I'm one of
those management types who has been plunged back into engineering while
awaiting our funding so I'm fumbling my way through badly.  I would
assume your approach is superior to mine.
> 
> Ben mentioned that if this was something that was a common need that 
> maybe a shared object could be added upstream to make this a general 
> solution.  I'm thinking that a module could be written that would do 
> this type of query on the device, and then look up the priority in a 
> simple expression file that might look something like:
> 
> <regular expression><priority>
I think there is a problem in using a separate file in that it is not
available to multipathd.  Somehow, it needs to be incorporated into
whatever multipathd has in memory. That is why we have our priomaker
script which sews together the list and the bash script into the final
bash script which is then loaded into multipathd.  Then again, maybe we
can use the "dummy device" trick to pull the expression file into the
multipathd namespace.
> 
> In my case I could just look for something like /ID_PATH=ip-10.0.x/ to 
> see if it is on the particular network in question, and then set the 
> priority.  You might search for entire iqn names.  But this would be 
> flexible to allow anyone to set priority based on the udev parameters of 
> vendor, model, serial numbers, iqn path, etc.
> 
> I don't know if it is feasible to query udev in this environment -- 
> perhaps someone closer to the internals could answer that.  (It looks 
> like it could also be pulled from /sys, but I'm not too familiar with 
> that structure, and we would need to make sure it was not too dependent 
> on kernel changes which might change that structure).
> 
> Thoughts?
> 
> -Ty!
> 
> John A. Sullivan III wrote:
> > On Thu, 2009-04-23 at 12:08 -0600, Ty! Boyack wrote:
> >   
> >> This thread has been great information since I'm looking at the same 
> >> type of thing.  However it raises a couple of (slightly off-topic) 
> >> questions for me. 
> >>
> >> My recent upgrade to fedora 10 broke my prio_callout bash script just 
> >> like you described, but my getuid_callout (a bash script that calls 
> >> udevadm, grep, sed, and iscsi_id) runs just fine.  Are the two callouts 
> >> handled differently?
> >>
> >> Also, is there an easy way to know what tools are in the private 
> >> namespace already?  My prio_callout script calls two other binaries: 
> >> /sbin/udevadm and grep.  If I go to C-code, handling grep's functions 
> >> myself is no problem, but I'm not confident about re-implementing what 
> >> udevadm does.  Can I assume that since /sbin/udevadm is in /sbin that it 
> >> will be available to call via exec()?  Or would I be right back where we 
> >> are with the bash scripting, as in having to include a dummy device as 
> >> you described?
> >>
> >> Finally, in my case I've got two redundant iscsi networks, one is 1GbE, 
> >> and the other is 10GbE.  In the past I've always had symetric paths, so 
> >> I've used round-robin/multibus.  But I want to focus traffic on the 
> >> 10GbE path, so I was looking at using the prio callout.  Is this even 
> >> necessary?  Or will round-robin/multibus take full advantage of both 
> >> paths?  I can see round-robin on that setup resulting in either around 
> >> 11Gbps or 2 Gbps, depending on whether the slower link becomes a 
> >> limiting factor.  I'm just wondering if I am making things unnecessarily 
> >> complex by trying to set priorities.
> >>
> >> Thanks for all the help.
> >>
> >> -Ty!
> >>
> >>     
> > I can't answer the questions regarding the internals.  I did make sure
> > my bash scripts called not external applications and I placed everything
> > in /sbin.
> >
> > I did find I was able to pick and choose which connections had which
> > priorities - that was the whole purpose of my script.  In my case, there
> > were many networks and I wanted prioritized failover to try to balance
> > the load across interfaces and keep failover traffic on the same switch
> > rather than crossing a bonded link to another switch.  I did it by cross
> > referencing the mappings in /dev/disk/by-path with a prioritized list of
> > mappings.  I believe I posted the entire setup in an earlier e-mail.  If
> > you'd like, I can post the details again.
> >
> > As I reread your post a little more closely, I wonder if using multibus
> > as you describe will not slow you down to the lowest common denominator.
> > I know when I tested with RAID0 across several interfaces to load
> > balance traffic (this seemed to give better average performance across a
> > wide range of I/O patterns than multi-bus with varying rr_min_io
> > settings), I had three e1000e NICs and one on board NIC. When I replaced
> > the on-board with another e1000e, I saw a substantial performance
> > improvement.  I don't know if that will be your experience for sure but
> > pass it along as a caveat. Hope this helps - John
> >   
> 
> 
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[Benjamin Marzinski]

On Thu, Apr 23, 2009 at 12:08:32PM -0600, Ty! Boyack wrote:
> This thread has been great information since I'm looking at the same type 
> of thing.  However it raises a couple of (slightly off-topic) questions for 
> me. 
> My recent upgrade to fedora 10 broke my prio_callout bash script just like 
> you described, but my getuid_callout (a bash script that calls udevadm, 
> grep, sed, and iscsi_id) runs just fine.  Are the two callouts handled 
> differently?

Fedora 10 uses the same method as upstream, priority callouts have been
replaced by priority modules. These are dynamic shared objects that get
loaded by multipath.  Because of this, multipath doesn't use a private
namespace, so it can use scripts without restrictions for the getuid_callout.

>
> Also, is there an easy way to know what tools are in the private namespace 
> already?  My prio_callout script calls two other binaries: /sbin/udevadm 
> and grep.  If I go to C-code, handling grep's functions myself is no 
> problem, but I'm not confident about re-implementing what udevadm does.  
> Can I assume that since /sbin/udevadm is in /sbin that it will be available 
> to call via exec()?  Or would I be right back where we are with the bash 
> scripting, as in having to include a dummy device as you described?

Sorry, the C code is necessary now.

> Finally, in my case I've got two redundant iscsi networks, one is 1GbE, and 
> the other is 10GbE.  In the past I've always had symetric paths, so I've 
> used round-robin/multibus.  But I want to focus traffic on the 10GbE path, 
> so I was looking at using the prio callout.  Is this even necessary?  Or 
> will round-robin/multibus take full advantage of both paths?  I can see 
> round-robin on that setup resulting in either around 11Gbps or 2 Gbps, 
> depending on whether the slower link becomes a limiting factor.  I'm just 
> wondering if I am making things unnecessarily complex by trying to set 
> priorities.

With round-robin, you will send half your IO to the slow path.  A
priority callout makes sense here.

> Thanks for all the help.
>
> -Ty!
>
> -- 
> -===========================-
>  Ty! Boyack
>  NREL Unix Network Manager
>  ty@nrel.colostate.edu
>  (970) 491-1186
> -===========================-
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath prio_callout broke from 5.2 to 5.3

[Christopher Chen]

On Thu, Apr 23, 2009 at 11:27 PM, Benjamin Marzinski
<bmarzins@redhat.com> wrote:
> On Thu, Apr 23, 2009 at 12:08:32PM -0600, Ty! Boyack wrote:
>> This thread has been great information since I'm looking at the same type
>> of thing.  However it raises a couple of (slightly off-topic) questions for
>> me.
>> My recent upgrade to fedora 10 broke my prio_callout bash script just like
>> you described, but my getuid_callout (a bash script that calls udevadm,
>> grep, sed, and iscsi_id) runs just fine.  Are the two callouts handled
>> differently?
>
> Fedora 10 uses the same method as upstream, priority callouts have been
> replaced by priority modules. These are dynamic shared objects that get
> loaded by multipath.  Because of this, multipath doesn't use a private
> namespace, so it can use scripts without restrictions for the getuid_callout.
>
>>
>> Also, is there an easy way to know what tools are in the private namespace
>> already?  My prio_callout script calls two other binaries: /sbin/udevadm
>> and grep.  If I go to C-code, handling grep's functions myself is no
>> problem, but I'm not confident about re-implementing what udevadm does.
>> Can I assume that since /sbin/udevadm is in /sbin that it will be available
>> to call via exec()?  Or would I be right back where we are with the bash
>> scripting, as in having to include a dummy device as you described?
>
> Sorry, the C code is necessary now.
>
>> Finally, in my case I've got two redundant iscsi networks, one is 1GbE, and
>> the other is 10GbE.  In the past I've always had symetric paths, so I've
>> used round-robin/multibus.  But I want to focus traffic on the 10GbE path,
>> so I was looking at using the prio callout.  Is this even necessary?  Or
>> will round-robin/multibus take full advantage of both paths?  I can see
>> round-robin on that setup resulting in either around 11Gbps or 2 Gbps,
>> depending on whether the slower link becomes a limiting factor.  I'm just
>> wondering if I am making things unnecessarily complex by trying to set
>> priorities.
>
> With round-robin, you will send half your IO to the slow path.  A
> priority callout makes sense here.
>
>> Thanks for all the help.

I ended up biting the bullet and writing a C program to do the
priority calculations.

m2c

cc

-- 
Chris Chen <muffaleta@gmail.com>
"I want the kind of six pack you can't drink."
-- Micah

multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

Hello, all.  We are facing a serious problem with dm-multipath after our
upgrade.  We use a bash script to set priorities for failover.  We
understand multipathd cannot use a bash script directly so it has been
carefully crafted to use only internal commands and is loaded as:

prio_callout            "/bin/bash /usr/local/sbin/mpath_prio_ssi %n"

This has been working perfectly fine.  We upgraded our test lab to
CentOS 5.3, device-mapper-multipath.x86_64 0.4.7-23.el5_3.2, kernel
2.6.29.1 (the 2.6.18 default causes a kernel panic with iSCSI).
Suddenly, it is breaking.  /var/log/messages is filled with:

Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/dasd_id : No such file or directory
Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/gnbd_import : No such file or directory
Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/dasd_id
Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/gnbd_import
Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory
Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdc
Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdd
Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sde
Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127

The first several messages are expected but not the latter ones.  If we
run the call from the command line, e.g.,
"/bin/bash /usr/local/sbin/mpath_prio_ssi sdc" it works perfectly fine.

What has changed and how do we fix it? I'll include a sample script
below.  The script is dynamically created just before launching
multipathd:

#!/bin/bash
# if not passed any device name, return a priority of 0
if [ -z "${1}" ];then
        echo 0
        exit
fi

DEVS="lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:17f534f0-74af-e61b-a716-b8ac8e219dac-lun-0 -> ../../sdj
lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:47c5e722-10d3-66c7-a952-d3d79732da9c-lun-0 -> ../../sdr
lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:520e823d-342c-6668-9477-fad130b148d7-lun-0 -> ../../sdn"

LIST="172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->99
172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->49
172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->24"

FOUND=0
IFSORIG=${IFS}
IFS=$'\n'
for LINE in ${DEVS}
do
        ENTRY=${LINE%/${1}}
        if [ ${#ENTRY} -ne ${#LINE} ];then # We found the line
                FOUND=1
                break
        fi
done
if [ "$FOUND" = "0" ];then  # This is not an iSCSI device
        echo 0
        exit
fi
DEV="${ENTRY##* ip-}"
#DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
#DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
DEV="${DEV%%-lun-[0-9]* ->*}"
PRIORITY=0
for LINE in ${LIST}
do
        DISK=${LINE%->*}
        if [ "${DEV}" = "${DISK}" ];then
                PRIORITY="${LINE##*->}"
                break
        fi
done
echo ${PRIORITY}

I did notice the semantics of /dev/disk/by-path changed and we adapted
to that.  We were planning to move this to production on Thursday so
this has thrown a huge spanner in the works.  Any help would be greatly
appreciated.  Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: multipath prio_callout broke from 5.2 to 5.3

[John A. Sullivan III]

On Sat, 2009-04-11 at 23:54 -0400, John A. Sullivan III wrote:
> Hello, all.  We are facing a serious problem with dm-multipath after our
> upgrade.  We use a bash script to set priorities for failover.  We
> understand multipathd cannot use a bash script directly so it has been
> carefully crafted to use only internal commands and is loaded as:
> 
> prio_callout            "/bin/bash /usr/local/sbin/mpath_prio_ssi %n"
> 
> This has been working perfectly fine.  We upgraded our test lab to
> CentOS 5.3, device-mapper-multipath.x86_64 0.4.7-23.el5_3.2, kernel
> 2.6.29.1 (the 2.6.18 default causes a kernel panic with iSCSI).
> Suddenly, it is breaking.  /var/log/messages is filled with:
> 
> Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/dasd_id : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: cannot open /sbin/gnbd_import : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/dasd_id
> Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/dasd_id in ramfs : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: [copy.c] cannot open /sbin/gnbd_import
> Apr 11 23:17:15 kvm01 multipathd: cannot copy /sbin/gnbd_import in ramfs : No such file or directory
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdc
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sdd
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> Apr 11 23:17:15 kvm01 multipathd: error calling out /bin/bash /usr/local/sbin/mpath_prio_ssi sde
> Apr 11 23:17:15 kvm01 multipathd: /bin/bash exitted with 127
> 
> The first several messages are expected but not the latter ones.  If we
> run the call from the command line, e.g.,
> "/bin/bash /usr/local/sbin/mpath_prio_ssi sdc" it works perfectly fine.
> 
> What has changed and how do we fix it? I'll include a sample script
> below.  The script is dynamically created just before launching
> multipathd:
> 
> #!/bin/bash
> # if not passed any device name, return a priority of 0
> if [ -z "${1}" ];then
>         echo 0
>         exit
> fi
> 
> DEVS="lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:17f534f0-74af-e61b-a716-b8ac8e219dac-lun-0 -> ../../sdj
> lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:47c5e722-10d3-66c7-a952-d3d79732da9c-lun-0 -> ../../sdr
> lrwxrwxrwx 1 root root  9 Apr 11 23:13 ip-172.x.x.30:3260-iscsi-iqn.1986-03.com.sun:02:520e823d-342c-6668-9477-fad130b148d7-lun-0 -> ../../sdn"
> 
> LIST="172.x.x.78:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->99
> 172.x.x.46:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->49
> 172.x.x.62:3260-iscsi-iqn.1986-03.com.sun:02:adb0cf37-9a23-6fc9-922a-eb4540bee1c9->24"
> 
> FOUND=0
> IFSORIG=${IFS}
> IFS=$'\n'
> for LINE in ${DEVS}
> do
>         ENTRY=${LINE%/${1}}
>         if [ ${#ENTRY} -ne ${#LINE} ];then # We found the line
>                 FOUND=1
>                 break
>         fi
> done
> if [ "$FOUND" = "0" ];then  # This is not an iSCSI device
>         echo 0
>         exit
> fi
> DEV="${ENTRY##* ip-}"
> #DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
> #DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
> DEV="${DEV%%-lun-[0-9]* ->*}"
> PRIORITY=0
> for LINE in ${LIST}
> do
>         DISK=${LINE%->*}
>         if [ "${DEV}" = "${DISK}" ];then
>                 PRIORITY="${LINE##*->}"
>                 break
>         fi
> done
> echo ${PRIORITY}
> 
> I did notice the semantics of /dev/disk/by-path changed and we adapted
> to that.  We were planning to move this to production on Thursday so
> this has thrown a huge spanner in the works.  Any help would be greatly
> appreciated.  Thanks - John

I've just notice that my console is filled with:

/bin/bash: /usr/local/sbin/mpath_prio_ssi: No such file or directory

but it is indeed there and owned by root and executable.  I've quintuple
checked! Has multipathd been changed so it cannot read anything from
disk even if invoked from within bash? Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society