path priorities on Sun's 6140

path priorities on Sun's 6140

[James Fillman]

I'm running RHEL5 with QLogic HBA's and a Sun 6140 SAN. The host type
I'm using for my servers is 'Solaris (with DMP)'. This turns AVT mode
one. For some reason, the two controllers are returning the same
priority value to my priority call out program (mpath_prio_tpc).

Can anyone briefly explain what the mpath_prio_tpc utility does and
where these priority values come from?

The output of multipath -v2 -ll is:

[root@vlxq02md cluster]# multipath -v2 -ll
mdi_logging (3600a0b800029f8e80000064d46e565e7) dm-9 SUN,CSM200_R
[size=100G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:0:2 sdd 8:48  [active][ready]
 \_ 1:0:1:2 sdg 8:96  [active][ready]

Where is should be:

[root@vlxq02md cluster]# multipath -v2 -ll
mdi_logging (3600a0b800029f8e80000064d46e565e7) dm-9 SUN,CSM200_R
[size=100G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=3][active]
 \_ 1:0:0:2 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=0][active]
 \_ 1:0:1:2 sdg 8:96  [active][ready]

I've gotten this working at my other data center with the same
hardware/software configuration. At the moment, I'm pointing my finger
at the SAN hardware but I don't have much to back it up with.

here's my multipath.conf file:

blacklist
{
         devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"
}

defaults
{
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    group_by_prio
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
        path_checker            readsector0
        rr_min_io               100
        rr_weight               priorities
        failback                immediate
        no_path_retry           queue
        user_friendly_name      yes
}

devnode_blacklist
{
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"
        devnode "^hd[a-z][0-9]*"
        devnode "^cciss!c[0-9]d[0-9]*"
}

multipaths
{
         multipath
        {
                wwid                   3600a0b800029cc400000120946e56616
                alias                  mdi_deploys
        }
         multipath
        {
                wwid                   3600a0b800029f8e80000064d46e565e7
                alias                  mdi_logging
        }
         multipath
        {
                wwid                   3600a0b800029cc4000001140468531fa
                alias                  primary_logging
        }
}

devices
{
        device
        {
                vendor                  "SUN"
                product                 "CSM200_R"
                path_grouping_policy    group_by_prio
                path_checker            readsector0
                prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
                getuid_callout          "/sbin/scsi_id -g -u -s
/block/%n"
                rr_weight               uniform
                rr_min_io               1000
                features                "1 queue_if_no_path"
        }
}

cheers,
--james

Re: path priorities on Sun's 6140

[Eckebrecht von Pappenheim]

Hi,

> Can anyone briefly explain what the mpath_prio_tpc utility does and
> where these priority values come from?
> 
> The output of multipath -v2 -ll is:
> 
> [root@vlxq02md cluster]# multipath -v2 -ll
> mdi_logging (3600a0b800029f8e80000064d46e565e7) dm-9 SUN,CSM200_R
> [size=100G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:0:2 sdd 8:48  [active][ready]
>  \_ 1:0:1:2 sdg 8:96  [active][ready]

you are sure your two pathes are using different controllers? A SAN setup
problem which makes the HBA use only 1 of the two controllers seems to be a
very good reason for this multipath output.

Do you use a single-port HBA? 

With my dual-port HBA I get:

Raid5_for_IMAP-Volume (3600a0b80002906e20000089345bf1a2b) dm-1 SUN,CSM200_R
[size=1.4T][features=0][hwhandler=0]
\_ round-robin 0 [prio=12][active]
 \_ 7:0:1:1  sdac 65:192 [active][ready] 
 \_ 6:0:3:1  sds  65:32  [active][ready] 
\_ round-robin 0 [prio=2][enabled]
 \_ 7:0:3:1  sdam 66:96  [active][ready] 
 \_ 6:0:1:1  sdi  8:128  [active][ready]


with multipath.conf:

devices {
        device {
                vendor SUN
                product CSM200_R
                path_grouping_policy group_by_prio
                path_checker tur
                prio_callout "mpath_prio_tpc /dev/%n"
                features "1 queue_if_no_path"
        }
}


BTW: A polling interval of 10 seconds is too slow for 6140 firmware
updates, 
I had to use 5 seconds because the controllers are rebooting quite fast...

regards,
Eckebrecht

-- 
Eckebrecht von Pappenheim
Netzwerkadministration
Heise Zeitschriften Verlag GmbH & Co KG
Helstorferstr. 7, D-30625 Hannover, FRG
E-Mail: evp@heise.de  - Tel: +49 511 5352 242  -  Fax: +49 511 5352 479
http://www.heise.de/

Registergericht: Amtsgericht Hannover HRA 26709
 
Persönlich haftende Gesellschafterin:  
Heise Zeitschriften Verlag Geschäftsführung GmbH
Registergericht: Amtsgericht Hannover, HRB 60405
Geschäftsführer: Ansgar Heise, Steven P. Steinkraus, Dr. Alfons Schräder

Re: path priorities on Sun's 6140

[Tore Anderson]

* James Fillman

> I'm running RHEL5 with QLogic HBA's and a Sun 6140 SAN. The host type
> I'm using for my servers is 'Solaris (with DMP)'. This turns AVT
> mode one. For some reason, the two controllers are returning the same
> priority value to my priority call out program (mpath_prio_tpc).

   Try path_grouping_policy group_by_serial?

> Can anyone briefly explain what the mpath_prio_tpc utility does and 
> where these priority values come from?

   It calls out to the supplied SCSI device and returns a different
  integer depending on if the controller is the preferred owner for the
  volume and if it actually is.  The values has changed quite a bit from
  version to version, but if I recall correctly they are in your version:

   Device is a path to the preferred and active controller = 6
   Device is a path to the preferred and inactive controller = 4
   Device is a path to the least preferred and active controller = 3
   Device is a path to the least preferred and inactive controller = 1

   Since you have group_by_prio and they end up in the same path group it
  seems the prio callout doesn't work (you can test this yourself by
  running «mpath_prio_tpc /dev/sdx»).  When I used AVT I set the hosts to
  host type AIX_FO, which seemed to work fine for me at least.  Try that?

   You're also using path_checker readsector0 with AVT which is really
  bad.  Every time multipathd checks a path to the passive controller the
  volume vil move there, which will interrupt I/O for several seconds.
  Use the tur checker instead.

   However, are you _really_ sure you want AVT mode?  Support for RDAC
  mode was recently added to dm-multipath (both a hardware handler and
  a path checker), and using this is normally vastly superior to AVT
  mode.  The Linux kernel itself (partition scan on boot) as well as a
  lot of applications (LVM, mdadm, fdisk, etc.) believe reading from
  block devices is a harmless thing to do.  However with AVT mode this
  I/O will cause a volume to transfer and I/O to be interrupted, and
  paths will fail.  With AVT you won't be able to boot node A in a
  cluster without interrupting the I/O flow of the rest of the nodes, nor
  manage LVM on any node in a cluster without also interrupting I/O
  (unless you've configured LVM to stay away from your multipathed
  devices).

   PS:  If you have I/O failures happening on your hosts when you do
  changes to the storage domains setup, this is due to a mis-feature in
  the 6140 that makes it dim its fibre ports whenever a change is made.
  They said this was done to make all hosts relogin and automatically
  discover new volumes with no manual rescan needed, but the fabric
  relogin interrupts I/O and causes path failures.  But there is hope.  I
  was told yesterday that in the next firmware release we can opt to
  toggle this «feature» off.  Go pester Sun to get it.  ;-)

Regards
-- 
Tore Anderson

RE: path priorities on Sun's 6140

[James Fillman]

Thanks for getting back to me Tore, Hannes, and Eckebrecht.

I've downloaded and installed the latest device-mapper-multipath rpm
from Redhat. It's still only beta but I figured I give it a try. I see
that there's an rdac checker in there but I don't see a hardware
handler.

Is anyone running RHEL against a Sun 6140 SAN? Trying to get a
definitive answer from Redhat on their support for this SAN has been
impossible. The mailing list hasn't been too helpful either.

If RDAC is the way to go
  1. do I need to use a hardware handler? There doesn't seem to be one
in their latest RPM.
  2. What Host type do I chose on the SAN?
  3. What should the path grouping policy by? Tore: you mentioned using
group_by_serial
  4. What should the path checker be?

I'm going to take a stab at it and guess this, can anyone confirm?:
/etc/multipath.conf

defaults
{
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    group_by_serial
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
        path_checker            tur
        rr_min_io               100
        rr_weight               uniform
        failback                immediate
        no_path_retry           queue
        user_friendly_name      yes
}
devices
{
        device
        {
                vendor                  "SUN"
                product                 "CSM200_R"
                path_grouping_policy    group_by_prio
                path_checker            tur
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
                getuid_callout          "/sbin/scsi_id -g -u -s
/block/%n"
                rr_weight               uniform
                rr_min_io               1000
                features                "1 queue_if_no_path"
        }
}

cheers,
--james

RE: path priorities on Sun's 6140

[James Fillman]

Ok. So I just realized that if I want to try the new RDAC support, then
I have to install the new beta kernel as well. It includes the RDAC
hardware handler.

I'm under serious pressure to get my cluster up and running asap and I
don't think I want to be installing a bunch of beta packages onto my
production servers. Especially kernels.

So this might mean not exploring the RDAC solution. Redhat's docs say
that they support a Sun OpenStorage D280 and our Sun rep says the 6140
is essentially identical. If that's the case, what am I don't wrong?

cheers,
--james

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of James Fillman
Sent: Monday, September 17, 2007 11:18 AM
To: device-mapper development
Subject: RE: [dm-devel] path priorities on Sun's 6140

Thanks for getting back to me Tore, Hannes, and Eckebrecht.

I've downloaded and installed the latest device-mapper-multipath rpm
from Redhat. It's still only beta but I figured I give it a try. I see
that there's an rdac checker in there but I don't see a hardware
handler.

Is anyone running RHEL against a Sun 6140 SAN? Trying to get a
definitive answer from Redhat on their support for this SAN has been
impossible. The mailing list hasn't been too helpful either.

If RDAC is the way to go
  1. do I need to use a hardware handler? There doesn't seem to be one
in their latest RPM.
  2. What Host type do I chose on the SAN?
  3. What should the path grouping policy by? Tore: you mentioned using
group_by_serial
  4. What should the path checker be?

I'm going to take a stab at it and guess this, can anyone confirm?:
/etc/multipath.conf

defaults
{
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    group_by_serial
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
        path_checker            tur
        rr_min_io               100
        rr_weight               uniform
        failback                immediate
        no_path_retry           queue
        user_friendly_name      yes
}
devices
{
        device
        {
                vendor                  "SUN"
                product                 "CSM200_R"
                path_grouping_policy    group_by_prio
                path_checker            tur
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
                getuid_callout          "/sbin/scsi_id -g -u -s
/block/%n"
                rr_weight               uniform
                rr_min_io               1000
                features                "1 queue_if_no_path"
        }
}

cheers,
--james


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: path priorities on Sun's 6140

[Tore Anderson]

* James Fillman

> Ok. So I just realized that if I want to try the new RDAC support,
> then I have to install the new beta kernel as well. It includes the
> RDAC hardware handler.
> 
> I'm under serious pressure to get my cluster up and running asap and
> I don't think I want to be installing a bunch of beta packages onto
> my production servers. Especially kernels.

To me the benefit of RDAC mode would far outweigh the disadvantage of
knowing you're running a beta kernel, but that's your call...

With AVT mode you will have I/O interruptions and volume failovers when
a node in your cluster boots.  It's unavoidable.  You'll probably want
to test really well how the rest of the cluster behaves in such situations.

> So this might mean not exploring the RDAC solution. Redhat's docs say
> that they support a Sun OpenStorage D280 and our Sun rep says the
> 6140 is essentially identical. If that's the case, what am I don't
> wrong?

The original hardware is called Engenio 3994 - you might be able to
confirm that the D280 is the same by Googling some.  (The IBM DS4700
model 72 is also the same, by the way).  RH probably supports it in AVT
mode.  Your configuration looks okay to me, different
path_grouping_policy then what I use but as long as the prio_callout
works correctly the end result should be the same.  My config looks like
this on a RHEL4 box running in AVT mode:

device {
        vendor                  SUN.*
        product                 CSM200_R.*
        path_grouping_policy    group_by_serial
        path_checker            tur
        prio_callout            "/usr/local/sbin/mpath_prio_tpc /dev/%n"
        failback                immediate
        no_path_retry           queue
}

This gives me output from 'multipath -ll' like this in a dual fabric
topology:

nfs_ny (3600a0b80002625ae0000080745dbe5ca)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:0:1 sdc 8:32  [active][ready]
 \_ 2:0:1:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=12][active]
 \_ 1:0:1:1 sde 8:64  [active][ready]
 \_ 2:0:0:1 sdg 8:96  [active][ready]

SCSI targets 1:0:0 and 2:0:1 are to controller A, 1:0:1 and 2:0:0 are to
controller B (the preferred one for LUN 1).  I can verify that the
prio_callout works correctly by testing one path:

$ /usr/local/sbin/mpath_prio_tpc /dev/sde
6

Did you try host type AIX_FO, by the way?  That was the one I found to
work the best with AVT mode.

Regards
-- 
Tore Anderson

RE: path priorities on Sun's 6140

[James Fillman]

Ok, I've made the attempt to install all the necessary RHEL5.1 beta
rpm's to get rdac working and unfortunately I had roll back. I had to
install a bunch of gfs, cluster, and lvm rpm's, along with the new
kernel and it seemed to work at first. I then fired up my cluster and it
was very unstable.

I've downloaded and compiled the latest multipath source (0.4.8). Now I
need to compile the rdac hardware handler against my current kernel
source. Can anyone provide me with the latest, patched source? 

Tore, I know you have it! After scanning through the posts, it looks
like you've had the same problems I have and you're running the same
hardware(Sun 6140). I could really use your help getting rdac working.
Redhat totally gave me the cold shoulder when I opened a ticket with
them. At this point, I need to keep the stock redhat kernel but am
willing to build all the multipath stuff against it. I've got all the
userland stuff compiled. Now I'm hoping I can get instructions on how to
compile the rdac hw handler against my kernel.

Thanks,
--james


-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Tore Anderson
Sent: Tuesday, September 18, 2007 12:40 AM
To: device-mapper development
Subject: Re: [dm-devel] path priorities on Sun's 6140

* James Fillman

> Ok. So I just realized that if I want to try the new RDAC support,
> then I have to install the new beta kernel as well. It includes the
> RDAC hardware handler.
> 
> I'm under serious pressure to get my cluster up and running asap and
> I don't think I want to be installing a bunch of beta packages onto
> my production servers. Especially kernels.

To me the benefit of RDAC mode would far outweigh the disadvantage of
knowing you're running a beta kernel, but that's your call...

With AVT mode you will have I/O interruptions and volume failovers when
a node in your cluster boots.  It's unavoidable.  You'll probably want
to test really well how the rest of the cluster behaves in such
situations.

> So this might mean not exploring the RDAC solution. Redhat's docs say
> that they support a Sun OpenStorage D280 and our Sun rep says the
> 6140 is essentially identical. If that's the case, what am I don't
> wrong?

The original hardware is called Engenio 3994 - you might be able to
confirm that the D280 is the same by Googling some.  (The IBM DS4700
model 72 is also the same, by the way).  RH probably supports it in AVT
mode.  Your configuration looks okay to me, different
path_grouping_policy then what I use but as long as the prio_callout
works correctly the end result should be the same.  My config looks like
this on a RHEL4 box running in AVT mode:

device {
        vendor                  SUN.*
        product                 CSM200_R.*
        path_grouping_policy    group_by_serial
        path_checker            tur
        prio_callout            "/usr/local/sbin/mpath_prio_tpc /dev/%n"
        failback                immediate
        no_path_retry           queue
}

This gives me output from 'multipath -ll' like this in a dual fabric
topology:

nfs_ny (3600a0b80002625ae0000080745dbe5ca)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:0:1 sdc 8:32  [active][ready]
 \_ 2:0:1:1 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=12][active]
 \_ 1:0:1:1 sde 8:64  [active][ready]
 \_ 2:0:0:1 sdg 8:96  [active][ready]

SCSI targets 1:0:0 and 2:0:1 are to controller A, 1:0:1 and 2:0:0 are to
controller B (the preferred one for LUN 1).  I can verify that the
prio_callout works correctly by testing one path:

$ /usr/local/sbin/mpath_prio_tpc /dev/sde
6

Did you try host type AIX_FO, by the way?  That was the one I found to
work the best with AVT mode.

Regards
-- 
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: path priorities on Sun's 6140

[Tore Anderson]

* James Fillman

> Ok, I've made the attempt to install all the necessary RHEL5.1 beta 
> rpm's to get rdac working and unfortunately I had roll back. I had to
> install a bunch of gfs, cluster, and lvm rpm's, along with the new 
> kernel and it seemed to work at first. I then fired up my cluster and
> it was very unstable.

Hm.  I thought it would work to upgrade only the kernel.  It seems very
strange that you have to upgrade all those userland utilities.  Did you
try with some RPM frontend like up2date that makes "clever" choices for
you, mayhaps?  If so try downloading the RPM and installing it directly
with the rpm utility.

When looking at the package description of the 2.6.18-36 RPM, I don't
really see any conflict that should cause this - the conflicts are the
exact same as in the latest RHEL5 non-beta kernel (2.6.18-8.1.10), so
there's no reason why it wouldn't work to install the beta kernel
directly without upgrading any other packages.  I can't see any, at least.

> I've downloaded and compiled the latest multipath source (0.4.8). Now
> I need to compile the rdac hardware handler against my current kernel
> source. Can anyone provide me with the latest, patched source?

The source RPM is available from RHN.  Go to Channels, then All
channels, find Red Hat Enterprise Linux (v. 5 for [your arch here]),
expand it and find the associated beta channel.  Enter it, go to
Packages and search for "kernel", follow the appropriate link and you
should have a source RPM download link there.

> Tore, I know you have it! After scanning through the posts, it looks 
> like you've had the same problems I have and you're running the same 
> hardware(Sun 6140). I could really use your help getting rdac
> working. Redhat totally gave me the cold shoulder when I opened a
> ticket with them. At this point, I need to keep the stock redhat
> kernel but am willing to build all the multipath stuff against it.
> I've got all the userland stuff compiled. Now I'm hoping I can get
> instructions on how to compile the rdac hw handler against my kernel.

I'm using 2.6.23-rcX or the kernel from RHEL 5.1 beta.  Or you can try
to grab the RHEL5 (2.6.18-8.1.10) kernel sources as well as the source
RPM for 2.6.18-36, copy over driver/md/dm-mpath-rdac.c and also add the
relevant changes to Makefile and Kconfig, and with some luck you'll be
able to build a module that's insertable on 2.6.18-8.1.10.  But first
I'd try some more to install the kernel from 5.1 beta.  I'm quite
confident it should work just fine.

Regards
-- 
Tore Anderson

RE: path priorities on Sun's 6140

[James Fillman]

I'm happy to say that I've gotten things working. Friday night I
grabbled the src RPM from the RHEL5.1 beta channel for the 2.6.36
kernel, pulled out the patch file for the rdac hardware handler, applied
it to the current 2.6.18-8.1.10 kernel, rebuilt the rpm and voila.
Things work great. I also had to install the beta
device-mapper-multipath and kpartx rpm's to get it working but after
some extensive testing, thing are working exactly as expected.

Thanks for all your help. It's much appreciated.

FYI: The reason why I had to install a bunch of addition cluster and lvm
beta rpm's along with the beta kernel is because the dlm kernel modules
in the beta kernel are too new for the userland tools. clvmd reported
version mismatches and all the cman stuff was just too unstable.

cheers,
--james



-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Tore Anderson
Sent: Sunday, September 23, 2007 11:48 PM
To: device-mapper development
Subject: Re: [dm-devel] path priorities on Sun's 6140

* James Fillman

> Ok, I've made the attempt to install all the necessary RHEL5.1 beta 
> rpm's to get rdac working and unfortunately I had roll back. I had to
> install a bunch of gfs, cluster, and lvm rpm's, along with the new 
> kernel and it seemed to work at first. I then fired up my cluster and
> it was very unstable.

Hm.  I thought it would work to upgrade only the kernel.  It seems very
strange that you have to upgrade all those userland utilities.  Did you
try with some RPM frontend like up2date that makes "clever" choices for
you, mayhaps?  If so try downloading the RPM and installing it directly
with the rpm utility.

When looking at the package description of the 2.6.18-36 RPM, I don't
really see any conflict that should cause this - the conflicts are the
exact same as in the latest RHEL5 non-beta kernel (2.6.18-8.1.10), so
there's no reason why it wouldn't work to install the beta kernel
directly without upgrading any other packages.  I can't see any, at
least.

> I've downloaded and compiled the latest multipath source (0.4.8). Now
> I need to compile the rdac hardware handler against my current kernel
> source. Can anyone provide me with the latest, patched source?

The source RPM is available from RHN.  Go to Channels, then All
channels, find Red Hat Enterprise Linux (v. 5 for [your arch here]),
expand it and find the associated beta channel.  Enter it, go to
Packages and search for "kernel", follow the appropriate link and you
should have a source RPM download link there.

> Tore, I know you have it! After scanning through the posts, it looks 
> like you've had the same problems I have and you're running the same 
> hardware(Sun 6140). I could really use your help getting rdac
> working. Redhat totally gave me the cold shoulder when I opened a
> ticket with them. At this point, I need to keep the stock redhat
> kernel but am willing to build all the multipath stuff against it.
> I've got all the userland stuff compiled. Now I'm hoping I can get
> instructions on how to compile the rdac hw handler against my kernel.

I'm using 2.6.23-rcX or the kernel from RHEL 5.1 beta.  Or you can try
to grab the RHEL5 (2.6.18-8.1.10) kernel sources as well as the source
RPM for 2.6.18-36, copy over driver/md/dm-mpath-rdac.c and also add the
relevant changes to Makefile and Kconfig, and with some luck you'll be
able to build a module that's insertable on 2.6.18-8.1.10.  But first
I'd try some more to install the kernel from 5.1 beta.  I'm quite
confident it should work just fine.

Regards
-- 
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

RE: path priorities on Sun's 6140

[Hannes Reinecke]

Am Mo 17 Sep 2007 20:36:32 CEST schrieb James Fillman <JFillman@cucbc.com>:

> Ok. So I just realized that if I want to try the new RDAC support, then
> I have to install the new beta kernel as well. It includes the RDAC
> hardware handler.
>
> I'm under serious pressure to get my cluster up and running asap and I
> don't think I want to be installing a bunch of beta packages onto my
> production servers. Especially kernels.

Well, I'm pretty sure their competitor has it with SP1 :-)

> So this might mean not exploring the RDAC solution. Redhat's docs say
> that they support a Sun OpenStorage D280 and our Sun rep says the 6140
> is essentially identical. If that's the case, what am I don't wrong?
>
If you don't have the RDAC hw handler you'll basically have to switch the
array to AVT mode. It works quite well, provided you don't boot the nodes
too often.

Cheers,

Hannes

Re: path priorities on Sun's 6140

[Hannes Reinecke]

James Fillman wrote:
> I'm running RHEL5 with QLogic HBA's and a Sun 6140 SAN. The host type
> I'm using for my servers is 'Solaris (with DMP)'. This turns AVT mode
> one. For some reason, the two controllers are returning the same
> priority value to my priority call out program (mpath_prio_tpc).
> 
> Can anyone briefly explain what the mpath_prio_tpc utility does and
> where these priority values come from?
> 
> The output of multipath -v2 -ll is:
> 
> [root@vlxq02md cluster]# multipath -v2 -ll
> mdi_logging (3600a0b800029f8e80000064d46e565e7) dm-9 SUN,CSM200_R
> [size=100G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:0:2 sdd 8:48  [active][ready]
>  \_ 1:0:1:2 sdg 8:96  [active][ready]
> 
> Where is should be:
> 
> [root@vlxq02md cluster]# multipath -v2 -ll
> mdi_logging (3600a0b800029f8e80000064d46e565e7) dm-9 SUN,CSM200_R
> [size=100G][features=1 queue_if_no_path][hwhandler=0]
> \_ round-robin 0 [prio=3][active]
>  \_ 1:0:0:2 sdd 8:48  [active][ready]
> \_ round-robin 0 [prio=0][active]
>  \_ 1:0:1:2 sdg 8:96  [active][ready]
> 
> I've gotten this working at my other data center with the same
> hardware/software configuration. At the moment, I'm pointing my finger
> at the SAN hardware but I don't have much to back it up with.
> 
> here's my multipath.conf file:
> 
> blacklist
> {
>          devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"
> }
> 
> defaults
> {
>         udev_dir                /dev
>         polling_interval        10
>         selector                "round-robin 0"
>         path_grouping_policy    group_by_prio
>         getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
>         prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
>         path_checker            readsector0
>         rr_min_io               100
>         rr_weight               priorities
>         failback                immediate
>         no_path_retry           queue
>         user_friendly_name      yes
> }
> 
> devnode_blacklist
> {
>         devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"
>         devnode "^hd[a-z][0-9]*"
>         devnode "^cciss!c[0-9]d[0-9]*"
> }
> 
> multipaths
> {
>          multipath
>         {
>                 wwid                   3600a0b800029cc400000120946e56616
>                 alias                  mdi_deploys
>         }
>          multipath
>         {
>                 wwid                   3600a0b800029f8e80000064d46e565e7
>                 alias                  mdi_logging
>         }
>          multipath
>         {
>                 wwid                   3600a0b800029cc4000001140468531fa
>                 alias                  primary_logging
>         }
> }
> 
> devices
> {
>         device
>         {
>                 vendor                  "SUN"
>                 product                 "CSM200_R"
>                 path_grouping_policy    group_by_prio
>                 path_checker            readsector0
>                 prio_callout            "/sbin/mpath_prio_tpc /dev/%n"
>                 getuid_callout          "/sbin/scsi_id -g -u -s
> /block/%n"
>                 rr_weight               uniform
>                 rr_min_io               1000
>                 features                "1 queue_if_no_path"
>         }
> }
> 
Oh, please _don't_ use 

rr_weight priorities

The implementation is broken (I send a patch once but it got rejected)
and it's completely useless when using 'group_by_prio'.
rr_weight = priorities is meant to adjust the round-robin scheduler by
the priority of each path. However, when using group_by_prio you'll the
same priority for all paths in a group. So effectively you only increase
(actually, you would increase if it would be working properly) the rr_min_io
value by the priority value.
So please try without this setting.

As for the AVT setting: sg_vpd can decode the RDAC mode pages; please send
the output of

sg_vpd -p vac <devname>

That should give us some hints what's going on here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)