Re: Problems with path groups in multipath-tools

Re: Problems with path groups in multipath-tools

[Juha Koivisto]

I'm not sure if my previous message is going to make it to the list (by
now it hasn't yet, at least according to the web archive), but I solved
my problem already. It turns out that in order to use a shell script as
prio_callout (in multipath.conf), the command needs to be explicitly
executed with /bin/sh.

This works (at least for me):

/etc/multipath.conf:
---cut---
devices {
        device {
                vendor                  "DGC     "
                product                 "*"
                path_grouping_policy    group_by_prio
                prio_callout            "/bin/sh /usr/local/sbin/get_path_prio %n"
                #getuid_callout         "/sbin/scsi_id -g -s /block/%n"
                #path_checker           emc_clariion
                path_selector           "round-robin 0"
                hardware_handler        "1 emc"
        }
}
---cut---

/usr/local/sbin/get_path_prio:
---cut---
#/bin/sh

PATH=/bin:/usr/bin

prio=1
if [ -e /sys/block/$1/device ]; then
        target=`ls -l /sys/block/$1/device | tr / "\n" | grep target.:.:`
        if [ -e /sys/class/fc_transport/$target/port_name ]; then
                id=`cut -c 10 < /sys/class/fc_transport/$target/port_name`
                case $id in
                        0 | 1 | 2 | 3 )
                                prio=1
                                ;;
                        8 | 9 | a | b | A | B )
                                prio=2
                                ;;
                esac
        fi
fi

echo $prio
---cut---

I don't know if I should use emc_clarion as path_checker (it sees the
non-active paths as "ready", while the default readsector0 reports them
as faulty), but I guess that's not very important right now.

Please Cc: me in replies, I'm not subscribed to the list.

-- 
Juha.Koivisto@hut.fi

Re: Re: Problems with path groups in multipath-tools

[Greg Freemyer]

Your problem is the very first line of your script

#/bin/sh

should be

#!/bin/sh

Note the missing ! in your line.

#! is a magic sequence to Liunx/UNIX to run the program as a script. 
The interpreter is whatever is on the rest of the line.

ie. #!/bin/perl is for a perl script.

Greg
On Fri, 25 Feb 2005 14:56:13 +0200, Juha Koivisto wrote:
> I'm not sure if my previous message is going to make it to the list (by
> now it hasn't yet, at least according to the web archive), but I solved
> my problem already. It turns out that in order to use a shell script as
> prio_callout (in multipath.conf), the command needs to be explicitly
> executed with /bin/sh.
> 
> This works (at least for me):
> 
> /etc/multipath.conf:
> ---cut---
> devices {
>         device {
>                 vendor                  "DGC     "
>                 product                 "*"
>                 path_grouping_policy    group_by_prio
>                 prio_callout            "/bin/sh /usr/local/sbin/get_path_prio %n"
>                 #getuid_callout         "/sbin/scsi_id -g -s /block/%n"
>                 #path_checker           emc_clariion
>                 path_selector           "round-robin 0"
>                 hardware_handler        "1 emc"
>         }
> }
> ---cut---
> 
> /usr/local/sbin/get_path_prio:
> ---cut---
> #/bin/sh
> 
> PATH=/bin:/usr/bin
> 
> prio=1
> if [ -e /sys/block/$1/device ]; then
>         target=`ls -l /sys/block/$1/device | tr / "\n" | grep target.:.:`
>         if [ -e /sys/class/fc_transport/$target/port_name ]; then
>                 id=`cut -c 10 < /sys/class/fc_transport/$target/port_name`
>                 case $id in
>                         0 | 1 | 2 | 3 )
>                                 prio=1
>                                 ;;
>                         8 | 9 | a | b | A | B )
>                                 prio=2
>                                 ;;
>                 esac
>         fi
> fi
> 
> echo $prio
> ---cut---
> 
> I don't know if I should use emc_clarion as path_checker (it sees the
> non-active paths as "ready", while the default readsector0 reports them
> as faulty), but I guess that's not very important right now.
> 
> Please Cc: me in replies, I'm not subscribed to the list.
> 
> --
> Juha.Koivisto@hut.fi
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: Re: Problems with path groups in multipath-tools

[Juha Koivisto]

On Fri, Feb 25, 2005 at 10:07:43AM -0500, Greg Freemyer wrote:
> Your problem is the very first line of your script
> 
> #/bin/sh

Uh... yes, I see that very clearly now. You don't want to know how many
hours I had spent debugging this (and then it turns out to be a simple
typo). The reason I missed it was that I had tested the script from the
command line and it worked, so I assumed the problem had to be in the
multipath tool. (I would have thought bash needs ". scriptname" to
execute a text file as a script in the current shell, but apparently
not.)

Anyway, thanks.

It seems that my original message still hasn't made it to the list, so
I will send it again. Even though the actual problem was solved already,
I noticed some other strangeness (and possibly real bugs), to be
described shortly...

-- 
Juha.Koivisto@hut.fi

Re: Re: Problems with path groups in multipath-tools

[Greg Freemyer]

On Fri, 25 Feb 2005 18:31:18 +0200, Juha Koivisto <Juha.Koivisto@hut.fi> wrote:
> On Fri, Feb 25, 2005 at 10:07:43AM -0500, Greg Freemyer wrote:
> > Your problem is the very first line of your script
> >
> > #/bin/sh
> 
> Uh... yes, I see that very clearly now. You don't want to know how many
> hours I had spent debugging this (and then it turns out to be a simple
> typo). The reason I missed it was that I had tested the script from the
> command line and it worked, so I assumed the problem had to be in the
> multipath tool. (I would have thought bash needs ". scriptname" to
> execute a text file as a script in the current shell, but apparently
> not.)
> 
Actually none of the shells require the #!/bin/sh type of thing if you
are invoking the script from the shell you want to execute it.

The purpose of it is to allow you to invoke the script from somewhere
other than the shell.

A simple example, if you write a csh script, then bash would require
#!/bin/csh to cause the shell to be swapped out.  Without the
#!/bin/csh, bash assumes it is native bash script.

Greg

Re: Re: Problems with path groups in multipath-tools

[Juha Koivisto]

[This is a repost of my original message which didn't seem to go to
the list (apologies if it actually did). The problem, which turned out
to be an embarrassing blindness to a missing "!", has been solved
already, but this might still contain some useful information and/or
bug reports, so here it goes.

Again, please Cc: replies to me, as I'm not on the list (yet - maybe
I should...)]

-----

Sorry if this is slightly off-topic for dm-devel, I wasn't sure if there
is a better list for the multipath userspace tools.

Anyway, for the last couple of days I've been trying to get a multipath
setup running, and while it seems I got most of the other problems sorted
out, I'm now stuck at getting the path groups right. [Solved now; see
script posted earlier in this thread (but add the missing "!")]

First, our hardware and fabric topology:
  - 2 Emulex LP9002L HBA's
  - EMC CX300 (two storage processors, one of them active and the other
    standby (can't access the disks); two FC ports each)
  - one path to each of the four ports (each HBA sees one active and
    one standby port through its own FC switch)

And software:
  - kernel 2.6.11-rc4 with patches 00003 through 00010 from
    http://sources.redhat.com/dm/ applied (00001 and 00002 were already
    in the rc4 kernel); dm_multipath and dm_emc modules loaded
  - lpfc driver v8.0.24 from http://sourceforge.net/projects/lpfcxxxx/
  - multipath-tools v0.4.2 (23/01, 2005) from the debian package

The problem:

I would (obviously) want to put the active paths (which are sda and sdd
at the moment) into one group, and the standby paths (currently sdb and
sdc) into another.

Unfortunately, with this combination of hardware and drivers, all paths
report the same node_name and serial number, so grouping by those isn't
working. It seems that the necessary information would in theory be
available with a patched /sbin/scsi_id (I remember reading about it
here, but has the patch been released yet?)

Until then the only way seems to be writing a prio_callout script that
would examine the port_name of the corresponding fc_transport target in
sysfs (giving the SP port WWN which is the only uniquely identifying
piece of information that I could find):

$ cat /sys/class/fc_transport/target*/port_name
0x5006016130209865
0x5006016830209865
0x5006016930209865
0x5006016030209865

Here, if the eighth digit is 0 or 1, the path is on storage processor A;
if 8 or 9, it's on storage processor B. (I believe this numbering scheme
is universal for the EMC CX?00 series; in the higher-end models with more
ports the numbers would be 0/1/2/3 and 8/9/a/b if I'm not mistaken.)

[broken script deleted]

Apart from the script, multipath seems to be doing something strange
when using group_by_prio. This is what I get with no multipath.conf:

# multipath -d -v 2 -p group_by_prio
Unknown page code '0xc0'
Unknown page code '0xc0'
Unknown page code '0xc0'
Unknown page code '0xc0'
create: 36006016024601200ea747d273d7fd911
[size=1 TB][features="0"][hwhandler="1 emc"]
\_ round-robin 0 [first]
  \_ 0:0:0:0 sda  8:0     [ready ]
  \_ 0:0:1:0 sdb  8:16    [ready ]
  \_ 1:0:1:0 sdd  8:48    [ready ]
\_ round-robin 0 
  \_ 1:0:0:0 sdc  8:32    [ready ]

(The "Unknown page code" messages are printed by /sbin/scsi_id which
doesn't support the command that multipath is trying to use for EMC
hardware).

So the paths get grouped three and one - I would understand each path
in its own group, or all in one, or the intended two and two, but this
I can't figure out. Judging by the debug output [see below], it would
appear to be a bug in the multipath tool.

[Also note that all paths are marked ready, even though in reality half
of them are failing. Below is the output when using the readsector0
path_checker instead of the default emc_clariion.] This also happens
with an empty "devices { }" section, so having that in the config file
at all masks out the built-in defaults. Not sure if this is a bug or a
feature.

# multipath -d -v 2 -p group_by_prio     
create: 36006016024601200ea747d273d7fd911
[size=1 TB][features="0"][hwhandler="0"]
\_ round-robin 0 [first]
  \_ 0:0:0:0 sda  8:0     [ready ]
  \_ 0:0:1:0 sdb  8:16    [faulty]
  \_ 1:0:1:0 sdd  8:48    [ready ]
\_ round-robin 0 
  \_ 1:0:0:0 sdc  8:32    [faulty]


[Deleted lengthy debug output (of multipath-tools compiled with
DEBUG=1) which is irrelevant now that the actual problem in the script
was found. It did show, though, that prio = 1 for all of the above
paths, which still end up in two groups. I can post it again if someone
really wants to see it.]

-- 
Juha.Koivisto@hut.fi