CLARiiON failover modes and DM.

CLARiiON failover modes and DM.

[Paul Cote]

[-- Attachment #1.1: Type: text/plain, Size: 123 bytes --]

Hi

Is there a specific failover mode that the CX must be set for DM
interoperability?

thanks in advance,

Paul


[-- Attachment #1.2: Type: text/html, Size: 981 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: CLARiiON failover modes and DM.

[Tore Anderson]

* Paul Cote

> Is there a specific failover mode that the CX must be set for DM
> interoperability?

You have (at least) two options.  CLARiiON Open mode 1 is the 
traditional way where all paths to the passive controller is present but 
will refuse almost all I/O operations.  You'll need to be using the 
"emc" hardware handler for that to work, since a proprietary command 
needs to be sent to a path to the passive controller in order to 
trespass the volume there.

The other option (which is only available in FLARE 26) is ALUA mode 
(CLARiiON Open mode 4), where I/O is accepted on both controllers even 
though only one is the one handling all the I/O.  You can use this 
without a hardware handler (there's one in developement, though), but 
you'll still want to be prioritising the paths to the preferred 
controller for optimal performance.  This is the mode I would prefer, at 
least.  I wrote a bit more about the ALUA mode a few days ago:

http://thread.gmane.org/gmane.linux.kernel.device-mapper.devel/4366/focus=4435

Regards
-- 
Tore Anderson

RE: CLARiiON failover modes and DM.

[Paul Cote]

Tore

I have found the definitions of all CX failover modes. The SAN admin
assigned failover mode 3. Will DM support option 3 assuming that I
follow EMC's guidance wrt the appropriate settings for the CX in
multipath.conf? ... Or should we reassign the failover mode to 1 per
your email? It seems to me that the path state status will differ
significantly based on the failover modes; specifically 1 or 3. We don't
have FLARE 26 so ALUA mode is N/A.

The failover definitions are:
1) Passive not ready; a command failure when I/O issued to non-owning
SP.
2) quiet trespass on I/O to non-owning SP
3) passive always ready; some commands (TUR) return "passive always
ready" status.

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Tore Anderson
Sent: Monday, November 19, 2007 1:51 PM
To: device-mapper development
Subject: Re: [dm-devel] CLARiiON failover modes and DM.


* Paul Cote

> Is there a specific failover mode that the CX must be set for DM 
> interoperability?

You have (at least) two options.  CLARiiON Open mode 1 is the 
traditional way where all paths to the passive controller is present but

will refuse almost all I/O operations.  You'll need to be using the 
"emc" hardware handler for that to work, since a proprietary command 
needs to be sent to a path to the passive controller in order to 
trespass the volume there.

The other option (which is only available in FLARE 26) is ALUA mode 
(CLARiiON Open mode 4), where I/O is accepted on both controllers even 
though only one is the one handling all the I/O.  You can use this 
without a hardware handler (there's one in developement, though), but 
you'll still want to be prioritising the paths to the preferred 
controller for optimal performance.  This is the mode I would prefer, at

least.  I wrote a bit more about the ALUA mode a few days ago:

http://thread.gmane.org/gmane.linux.kernel.device-mapper.devel/4366/focu
s=4435

Regards
-- 
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: CLARiiON failover modes and DM.

[Tore Anderson]

* Paul Cote

> I have found the definitions of all CX failover modes. The SAN admin 
> assigned failover mode 3. Will DM support option 3 assuming that I 
> follow EMC's guidance wrt the appropriate settings for the CX in 
> multipath.conf? ... Or should we reassign the failover mode to 1 per 
> your email? It seems to me that the path state status will differ 
> significantly based on the failover modes; specifically 1 or 3. We
> don't have FLARE 26 so ALUA mode is N/A.
> 
> The failover definitions are:
> 1) Passive not ready; a command failure when I/O issued to non-owning
> SP.
> 2) quiet trespass on I/O to non-owning SP
> 3) passive always ready; some commands (TUR) return "passive always
> ready" status.

Mode 1 is what I've used on my CX so far (no support for ALUA yet).
You'll get lots of I/O failures when something wants to access the
passive paths (to scan for partition tables, PV signatures, and so on),
but they're harmless and can be disregarded.  You'll need path_checker
emc_clariion, hardware_handler emc, prio_callout mpath_prio_emc, and
path_grouping_policy group_by_prio  I think this is the default mode
multipath-tools is set up to handle, so unless you can invite EMC over
for a FLARE 26 upgrade party, I think this is the one you want.

Mode 2 you don't want unless you really know what you're doing.  This
will cause the I/O that failed in mode 1 to cause a volume trespass.  So
if you boot node 2 in a cluster, the data volumes will bounce around
wildly between SPs until it has finished booting, which'll affect I/O
from the production nodes in your cluster.  Same problem if you run
"vgdisplay", "fdisk -l", run HAL (maybe), and so on.

Mode 3 I never tried but it sounds like it's like mode 1, only that the
TEST UNIT READY will succeed.  If that's the case the only difference is
probably that you could use path_checker tur instead of path_checker emc.

Regards
-- 
Tore Anderson

RE: CLARiiON failover modes and DM.

[Paul Cote]

Tore,

Thanks for all the info ... Setting the CX FO mode to 1 appears to have
clear up the mess. One thing I've noticed after doing a dynamic
discovery of newly provisioned CX LUNs in that path state is generally
wrong. It will come up as [active][faulty]and to correct that requires a
reboot of the server. Don't know if that's a bug or not ... But I'm not
opposed to a reboot.

-----Original Message-----
From: dm-devel-bounces@redhat.com [mailto:dm-devel-bounces@redhat.com]
On Behalf Of Tore Anderson
Sent: Tuesday, November 20, 2007 2:47 AM
To: device-mapper development
Subject: Re: [dm-devel] CLARiiON failover modes and DM.


* Paul Cote

> I have found the definitions of all CX failover modes. The SAN admin
> assigned failover mode 3. Will DM support option 3 assuming that I 
> follow EMC's guidance wrt the appropriate settings for the CX in 
> multipath.conf? ... Or should we reassign the failover mode to 1 per 
> your email? It seems to me that the path state status will differ 
> significantly based on the failover modes; specifically 1 or 3. We
> don't have FLARE 26 so ALUA mode is N/A.
> 
> The failover definitions are:
> 1) Passive not ready; a command failure when I/O issued to non-owning 
> SP.
> 2) quiet trespass on I/O to non-owning SP
> 3) passive always ready; some commands (TUR) return "passive always 
> ready" status.

Mode 1 is what I've used on my CX so far (no support for ALUA yet).
You'll get lots of I/O failures when something wants to access the
passive paths (to scan for partition tables, PV signatures, and so on),
but they're harmless and can be disregarded.  You'll need path_checker
emc_clariion, hardware_handler emc, prio_callout mpath_prio_emc, and
path_grouping_policy group_by_prio  I think this is the default mode
multipath-tools is set up to handle, so unless you can invite EMC over
for a FLARE 26 upgrade party, I think this is the one you want.

Mode 2 you don't want unless you really know what you're doing.  This
will cause the I/O that failed in mode 1 to cause a volume trespass.  So
if you boot node 2 in a cluster, the data volumes will bounce around
wildly between SPs until it has finished booting, which'll affect I/O
from the production nodes in your cluster.  Same problem if you run
"vgdisplay", "fdisk -l", run HAL (maybe), and so on.

Mode 3 I never tried but it sounds like it's like mode 1, only that the
TEST UNIT READY will succeed.  If that's the case the only difference is
probably that you could use path_checker tur instead of path_checker
emc.

Regards
-- 
Tore Anderson

--
dm-devel mailing list
dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel

Re: CLARiiON failover modes and DM.

[Tore Anderson]

* Paul Cote

> Thanks for all the info ... Setting the CX FO mode to 1 appears to
> have clear up the mess. One thing I've noticed after doing a dynamic 
> discovery of newly provisioned CX LUNs in that path state is
> generally wrong. It will come up as [active][faulty]and to correct
> that requires a reboot of the server. Don't know if that's a bug or
> not ... But I'm not opposed to a reboot.

That's odd, never had that problem myself.  I rescan like this:

(where hostN is a FC adapter, repeat for all of them but give it some
time between iterations as you'll see paths failing for a short while):

echo 1 > /sys/class/fc_host/hostN/issue_lip
echo - - - > /sys/class/scsi_host/hostN/scan

Usually only these steps are required to discover new volumes properly,
but you can also force a rediscovery of the multipath topology and maybe
also do a restart of multipathd:

multipath -v2
pkill multipathd && multipathd

Regards
-- 
Tore Anderson

Re: CLARiiON failover modes and DM.

[Stephan Austermühle]

Hi Paul,

> Is there a specific failover mode that the CX must be set for DM
> interoperability?

We are using the defaults:

Initiator Type: 3 (Clariion Open)
Array Comm Path: 1 (enabled)
Failover Mode: 1
Unit Serial Number: Array

These settings work reliably for some hundred Linux servers and about two
years.

Kind regards,

Stephan