multipath: rdac on IBM 1746 (SAS connection to DS3512)

multipath: rdac on IBM 1746 (SAS connection to DS3512)

[Peter Volkov]

Hi. We have Bladecenter H with HS22 servers inside. All servers have 
0b:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1064ET
PCI-Express Fusion-MPT SAS (rev 10)
with two possible external SAS connections which we use to connect them
with DS3512 (IBM datastorage) using two SAS cables and thus multipath
configuration is in place. We use ubuntu 10.04 here and since in-stock
scsi_dh_rdac kernel module does not support IBM 1746 devices I've
patched it and rebuild kernel. Now, as I modproble module I see the
following in dmesg:

[   27.861388] sd 0:0:2:1: rdac: LUN 1 (owned)
[   27.861836] sd 0:0:3:1: rdac: LUN 1 (unowned)
[   27.861840] rdac: device handler registered

That's Ok, but multipath output does not have rdac:

root@btv1:~# multipath -l
vod (360080e50001baa46000025834d501798) dm-0 IBM     ,1746      FASt
[size=13T][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][active]
 \_ 0:0:2:1 sdc 8:32  [active][undef]
\_ round-robin 0 [prio=0][enabled]
 \_ 0:0:3:1 sdd 8:48  [active][undef]


Is this normal? On Gentoo system connected to the same DS I see:

mon1 ~ # multipath -l
vod (360080e50001baa46000025834d501798) dm-0 IBM,1746      FAStT
size=13T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1
rdac' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| `- 0:0:2:1 sdc 8:32 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 0:0:3:1 sdd 8:48 active undef running

Could you tell me what's hwhandler and why it's 0 in one case and 1 in
another.

Many thanks for any explanations in advance,
-- 
Peter.

Re: multipath: rdac on IBM 1746 (SAS connection to DS3512)

[Christophe Varoqui]

> root@btv1:~# multipath -l
> vod (360080e50001baa46000025834d501798) dm-0 IBM     ,1746      FASt
> [size=13T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][active]
>  \_ 0:0:2:1 sdc 8:32  [active][undef]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 0:0:3:1 sdd 8:48  [active][undef]
> 
> 
> Is this normal? On Gentoo system connected to the same DS I see:
> 
There you can say the multipath tools version is outdated on your ubuntu
version (still old output tree format).

> mon1 ~ # multipath -l
> vod (360080e50001baa46000025834d501798) dm-0 IBM,1746      FAStT
> size=13T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1
> rdac' wp=rw
> |-+- policy='round-robin 0' prio=-1 status=active
> | `- 0:0:2:1 sdc 8:32 active undef running
> `-+- policy='round-robin 0' prio=-1 status=enabled
>   `- 0:0:3:1 sdd 8:48 active undef running
> 
> Could you tell me what's hwhandler and why it's 0 in one case and 1 in
> another.
> 
Here the multipath tools version is more recent. More storage device
definitions get added every month. The hardware handler is one of the
properties of such a storage device definition. This property is used to
format the multipath device mapper string.

On your ubuntu setup, multipath tools picked the default hardware
handler (none, ie 0) whereas on your gentoo setup the hardware handler
is explicitely set for use with your storage device.

Now 2 choices :
1/ upgrade multipath-tools
or
2/ set up your device definition in multipath.conf to override internal
defaults

Regards,
-- 
Christophe Varoqui <christophe.varoqui@opensvc.com>
OpenSVC

Re: multipath: rdac on IBM 1746 (SAS connection to DS3512)

[Peter Volkov]

Thank you for detailed answer Christophe.

В Срд, 09/02/2011 в 10:49 +0100, Christophe Varoqui пишет:
> On your ubuntu setup, multipath tools picked the default hardware
> handler (none, ie 0) whereas on your gentoo setup the hardware handler
> is explicitely set for use with your storage device.
> 
> Now 2 choices :
> 1/ upgrade multipath-tools
> or
> 2/ set up your device definition in multipath.conf to override internal
> defaults

I'll try to upgrade multipath-tools, but as for second choice it looks
like following configuration should work:

root@btv1:~# cat /etc/multipath.conf 
defaults {
	path_checker	rdac
}

blacklist {
	devnode "^sd[a-b]"
}

multipaths {
	multipath {
		wwid			360080e50001baa46000025834d501798
		alias			vod
		path_checker    rdac
	}
}

I even think that rdac definition in inside multipath is redundant but
if I remove it nothing changes. And looking at verbose output it looks
like it reads configfile (at least sda and sdb devices are blacklisted
and it tells that I have rdac defined in configfile).

root@btv1:~# multipath -v3 -ll
ram0: device node name blacklisted
[...snip...]
ram15: device node name blacklisted
loop0: device node name blacklisted
[...snip...]
loop7: device node name blacklisted
sda: device node name blacklisted
sdb: device node name blacklisted
sdc: not found in pathvec
sdc: mask = 0x5
sdc: dev_t = 8:32
sdc: size = 27341848576
sdc: subsystem = scsi
sdc: vendor = IBM     
sdc: product = 1746      FAStT 
sdc: rev = 1070
sdc: h:b:t:l = 0:0:2:1
sdd: not found in pathvec
sdd: mask = 0x5
sdd: dev_t = 8:48
sdd: size = 27341848576
sdd: subsystem = scsi
sdd: vendor = IBM     
sdd: product = 1746      FAStT 
sdd: rev = 1070
sdd: h:b:t:l = 0:0:3:1
md0: device node name blacklisted
dm-0: device node name blacklisted
===== paths list =====
uuid hcil    dev dev_t pri dm_st  chk_st  vend/prod/rev            
     0:0:2:1 sdc 8:32  -1  [undef][undef] IBM     ,1746      FAStT 
     0:0:3:1 sdd 8:48  -1  [undef][undef] IBM     ,1746      FAStT 
params = 0 0 2 1 round-robin 0 1 1 8:32 1000 round-robin 0 1 1 8:48
1000 
status = 2 0 0 0 2 1 A 0 1 0 8:32 A 0 E 0 1 0 8:48 A 0 
sdc: mask = 0x4
sdc: path checker = rdac (config file default)
sdc: state = 2
sdc: mask = 0x8
sdc: getprio = NULL (internal default)
sdc: prio = 1
sdd: mask = 0x4
sdd: path checker = rdac (config file default)
sdd: state = 4
sdd: mask = 0x8
sdd: getprio = NULL (internal default)
sdd: prio = 1
vod (360080e50001baa46000025834d501798) dm-0 IBM     ,1746      FASt
[size=13T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
 \_ 0:0:2:1 sdc 8:32  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 0:0:3:1 sdd 8:48  [active][ghost]


-- 
Peter.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath: rdac on IBM 1746 (SAS connection to DS3512)

[Malahal Naineni]

Peter Volkov [peter.volkov@gmail.com] wrote:
> Thank you for detailed answer Christophe.
> 
> В Срд, 09/02/2011 в 10:49 +0100, Christophe Varoqui пишет:
> > On your ubuntu setup, multipath tools picked the default hardware
> > handler (none, ie 0) whereas on your gentoo setup the hardware handler
> > is explicitely set for use with your storage device.
> > 
> > Now 2 choices :
> > 1/ upgrade multipath-tools
> > or
> > 2/ set up your device definition in multipath.conf to override internal
> > defaults
> 
> I'll try to upgrade multipath-tools, but as for second choice it looks
> like following configuration should work:
> 
> root@btv1:~# cat /etc/multipath.conf 
> defaults {
> 	path_checker	rdac
> }

Checker name is in the controller section and that default may
overwrite the above. BTW, checker name is different from
hardware handler.

> 
> blacklist {
> 	devnode "^sd[a-b]"
> }
> 
> multipaths {
> 	multipath {
> 		wwid			360080e50001baa46000025834d501798
> 		alias			vod
> 		path_checker    rdac

The above should work for path checker but path checker is different
from hardware handler.

Thanks, Malahal.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath: partitions are not created on IBM 1746 (SAS connection to DS3512)

[Peter Volkov]

Thank you Christophe and Malahal.

В Срд, 09/02/2011 в 11:22 +0100, Christophe Varoqui пишет:
> On mer., 2011-02-09 at 13:08 +0300, Peter Volkov wrote:
> > root@btv1:~# cat /etc/multipath.conf 
> > defaults {
> > 	path_checker	rdac
> > }
> > 
> the 'path_checker' param does not set the hwhandler.
> the keyword for that is 'hardware_handler'.

Is this an error in man multipath.conf then (it mentions rdac in
path_checker section)?

Anyway I've updated multipath-tools on ubuntu and this fixed initial
issue, but now I experience different problem with multipath. For
unknown reasons multipath does not automagically create partitions on
Gentoo system, although everything works if I run `kpartx -a /dev/dm-0`
manually. Could you help me to investigate this issue?

I've upgraded multipath tools to the current git with no Gentoo patches
applied, udev to 164, and lvm2 to 2.02.84 but still multipath even
without configuration file (defaults) creates only following devices:

 # ls -l /dev/mapper/360080e50001baa46000025834d501798* 
lrwxrwxrwx 1 root root 7 Feb 12 17:01 /dev/mapper/360080e50001baa46000025834d501798 -> ../dm-0
 # ls -l /dev/dm*                                      
brw-rw---- 1 root disk 253, 0 Feb 12 17:01 /dev/dm-0

and after

 # kpartx -a /dev/dm-0

partition appear:

 # ls -l /dev/mapper/360080e50001baa46000025834d501798*
lrwxrwxrwx 1 root root 7 Feb 12 17:01 /dev/mapper/360080e50001baa46000025834d501798 -> ../dm-0
lrwxrwxrwx 1 root root 7 Feb 12 17:03 /dev/mapper/360080e50001baa46000025834d501798p1 -> ../dm-1

This can be problem in my kernel configuration but I've compared with
ubuntu config and failed to find any evident differences. Could you
suggest how to debug this?


Have a nice weekend,
-- 
Peter.

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath: partitions are not created on IBM 1746 (SAS connection to DS3512)

[Christophe Varoqui]

On sam., 2011-02-12 at 12:12 +0300, Peter Volkov wrote:
> Thank you Christophe and Malahal.
> 
> В Срд, 09/02/2011 в 11:22 +0100, Christophe Varoqui пишет:
> > On mer., 2011-02-09 at 13:08 +0300, Peter Volkov wrote:
> > > root@btv1:~# cat /etc/multipath.conf 
> > > defaults {
> > > 	path_checker	rdac
> > > }
> > > 
> > the 'path_checker' param does not set the hwhandler.
> > the keyword for that is 'hardware_handler'.
> 
> Is this an error in man multipath.conf then (it mentions rdac in
> path_checker section)?
> 
No, rdac is also a valid valid value for path_checker.

> Anyway I've updated multipath-tools on ubuntu and this fixed initial
> issue, but now I experience different problem with multipath. For
> unknown reasons multipath does not automagically create partitions on
> Gentoo system, although everything works if I run `kpartx -a /dev/dm-0`
> manually. Could you help me to investigate this issue?
> 
You must keep the distribution udev rules. And report bugs to them. Nor
multipath or multipathd care about partitions. Those create multipath
device maps and upon creation udev is responsible for calling kpartx.

Check the multipath-tools 'make install' output to see what it may
unduly install as udev rules.

Regards,
cvaroqui


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel