multipath and san fabric

multipath and san fabric

[John Little]

Hi all

Admittedly this is not the correct forum for this question but I have googled 
this and the only people I have to ask around here are people who won't touch 
it because it is Linux.

My question is this:  I'm using SLES10 with two emulex hbas, one connected to 
fabric a and one to fabric b.  When zoning the switch for the hbas do I put 
in only one path to the fabric per hba or do I put in two, one to each of the 
separate fabrics?

Again I realize this is not exactly the correct forum but since I couldn't 
find an answer anywhere else I figured you guys would be the most 
knowledgeable.  If there are some docs somewhere that you could point me to I 
would appreciate it.

John

Re: multipath and san fabric

[James Smart]

If the hba's are on different fabrics - why would you ever put an hba
that is not in the fabric in a zone in that fabric ?  (e.g. your second
option makes no sense).

If you are stating that the 2hba's are on different fabrics (and in zones
within those fabric) - but they are seeing the same Storage device on both
fabrics - then the dm config (note: I'm not talking about zoning anymore)
would have a path per I_T nexus. I state it this way to account for dual-ported
controllers, which may appear as separate targets, may exist within the fabric.

Example 1:
   Fabric A contains HBA1 and TargetPort1. Typically a zone would exist
     within Fabric A that contains these 2 ports.
   Fabric B contains HBA2 and TargetPort2. Typically a zone would exist
     within Fabric B that contains these 2 ports.
   The storage array contains 2 controllers with 1 port per controller, or
     is a single controller with 2 ports. Either way, the two ports are
     TargetPort1 and TargetPort2, are on the same array, and all storage
     can be seen from any port.
   --
   This would have 2 DM paths.
   The I_T nexus's are:
      HBA1_TargetPort1
      HBA2_TargetPort2

Example 2:
   Fabric A contains HBA1, TargetPort1, and TargetPort3. Typically a zone
     would exist within Fabric A that contains these 3 ports.
   Fabric B contains HBA2, TargetPort2, and TargetPort4. Typically a zone
     would exist within Fabric B that contains these 3 ports.
   The storage array contains 2 controllers, with 2 ports per controller.
     Meaning TargetPort1 and TargetPort2 are on controller A, and TargetPort3
     and TargetPort4 are on controller B. The ports are cross connected on
     the fabrics for redundancy. All storage can be seen from any port.
   --
   This would ahve 4 DM paths
   The I_T nexus's are:
      HBA1_TargetPort1
      HBA1_TargetPort2
      HBA2_TargetPort3
      HBA2_TargetPort4

-- james

John Little wrote:
> Hi all
> 
> Admittedly this is not the correct forum for this question but I have googled 
> this and the only people I have to ask around here are people who won't touch 
> it because it is Linux.
> 
> My question is this:  I'm using SLES10 with two emulex hbas, one connected to 
> fabric a and one to fabric b.  When zoning the switch for the hbas do I put 
> in only one path to the fabric per hba or do I put in two, one to each of the 
> separate fabrics?
> 
> Again I realize this is not exactly the correct forum but since I couldn't 
> find an answer anywhere else I figured you guys would be the most 
> knowledgeable.  If there are some docs somewhere that you could point me to I 
> would appreciate it.
> 
> John
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 

RE: multipath and san fabric

[Edward Goggin]

You may want to also cross connect across the 2 target
controllers (A,B) in case 2.  Doing so provides more
redundancy for asymmetric arrays and better resource
utilization in case of a failure of either an HBA or
target controller which would otherwise prevent
utilization of the other HBA or target controller
for that LU when load sharing I/O across multiple
paths.

So the I_T nexus's for the 4 paths are:

	HBA1_TargetPort1
	HBA1_TargetPort3
	HBA2_TargetPort2
	HBA2_TargetPort4

> -----Original Message-----
> From: dm-devel-bounces@redhat.com 
> [mailto:dm-devel-bounces@redhat.com] On Behalf Of James Smart
> Sent: Monday, November 13, 2006 2:18 PM
> To: jslittl@hendricks.org; device-mapper development
> Subject: Re: [dm-devel] multipath and san fabric
> 
> If the hba's are on different fabrics - why would you ever put an hba
> that is not in the fabric in a zone in that fabric ?  (e.g. 
> your second
> option makes no sense).
> 
> If you are stating that the 2hba's are on different fabrics 
> (and in zones
> within those fabric) - but they are seeing the same Storage 
> device on both
> fabrics - then the dm config (note: I'm not talking about 
> zoning anymore)
> would have a path per I_T nexus. I state it this way to 
> account for dual-ported
> controllers, which may appear as separate targets, may exist 
> within the fabric.
> 
> Example 1:
>    Fabric A contains HBA1 and TargetPort1. Typically a zone 
> would exist
>      within Fabric A that contains these 2 ports.
>    Fabric B contains HBA2 and TargetPort2. Typically a zone 
> would exist
>      within Fabric B that contains these 2 ports.
>    The storage array contains 2 controllers with 1 port per 
> controller, or
>      is a single controller with 2 ports. Either way, the two 
> ports are
>      TargetPort1 and TargetPort2, are on the same array, and 
> all storage
>      can be seen from any port.
>    --
>    This would have 2 DM paths.
>    The I_T nexus's are:
>       HBA1_TargetPort1
>       HBA2_TargetPort2
> 
> Example 2:
>    Fabric A contains HBA1, TargetPort1, and TargetPort3. 
> Typically a zone
>      would exist within Fabric A that contains these 3 ports.
>    Fabric B contains HBA2, TargetPort2, and TargetPort4. 
> Typically a zone
>      would exist within Fabric B that contains these 3 ports.
>    The storage array contains 2 controllers, with 2 ports per 
> controller.
>      Meaning TargetPort1 and TargetPort2 are on controller A, 
> and TargetPort3
>      and TargetPort4 are on controller B. The ports are cross 
> connected on
>      the fabrics for redundancy. All storage can be seen from 
> any port.
>    --
>    This would ahve 4 DM paths
>    The I_T nexus's are:
>       HBA1_TargetPort1
>       HBA1_TargetPort2
>       HBA2_TargetPort3
>       HBA2_TargetPort4
> 
> -- james
> 
> John Little wrote:
> > Hi all
> > 
> > Admittedly this is not the correct forum for this question 
> but I have googled 
> > this and the only people I have to ask around here are 
> people who won't touch 
> > it because it is Linux.
> > 
> > My question is this:  I'm using SLES10 with two emulex 
> hbas, one connected to 
> > fabric a and one to fabric b.  When zoning the switch for 
> the hbas do I put 
> > in only one path to the fabric per hba or do I put in two, 
> one to each of the 
> > separate fabrics?
> > 
> > Again I realize this is not exactly the correct forum but 
> since I couldn't 
> > find an answer anywhere else I figured you guys would be the most 
> > knowledgeable.  If there are some docs somewhere that you 
> could point me to I 
> > would appreciate it.
> > 
> > John
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 
> 

Re: multipath and san fabric

[Philip R. Auld]

Rumor has it that on Mon, Nov 13, 2006 at 02:39:08PM -0500 Edward Goggin said:
> You may want to also cross connect across the 2 target
> controllers (A,B) in case 2.  Doing so provides more
> redundancy for asymmetric arrays and better resource
> utilization in case of a failure of either an HBA or
> target controller which would otherwise prevent
> utilization of the other HBA or target controller
> for that LU when load sharing I/O across multiple
> paths.
> 
> So the I_T nexus's for the 4 paths are:
> 
> 	HBA1_TargetPort1
> 	HBA1_TargetPort3
> 	HBA2_TargetPort2
> 	HBA2_TargetPort4
> 

That's exactly what James's example 2 describes. It's just 
typo'd in the DM paths list. 



> > -----Original Message-----
> > From: dm-devel-bounces@redhat.com 
> > [mailto:dm-devel-bounces@redhat.com] On Behalf Of James Smart
> > Sent: Monday, November 13, 2006 2:18 PM
> > To: jslittl@hendricks.org; device-mapper development
> > Subject: Re: [dm-devel] multipath and san fabric
> > 
> > If the hba's are on different fabrics - why would you ever put an hba
> > that is not in the fabric in a zone in that fabric ?  (e.g. 
> > your second
> > option makes no sense).
> > 
> > If you are stating that the 2hba's are on different fabrics 
> > (and in zones
> > within those fabric) - but they are seeing the same Storage 
> > device on both
> > fabrics - then the dm config (note: I'm not talking about 
> > zoning anymore)
> > would have a path per I_T nexus. I state it this way to 
> > account for dual-ported
> > controllers, which may appear as separate targets, may exist 
> > within the fabric.
> > 
> > Example 1:
> >    Fabric A contains HBA1 and TargetPort1. Typically a zone 
> > would exist
> >      within Fabric A that contains these 2 ports.
> >    Fabric B contains HBA2 and TargetPort2. Typically a zone 
> > would exist
> >      within Fabric B that contains these 2 ports.
> >    The storage array contains 2 controllers with 1 port per 
> > controller, or
> >      is a single controller with 2 ports. Either way, the two 
> > ports are
> >      TargetPort1 and TargetPort2, are on the same array, and 
> > all storage
> >      can be seen from any port.
> >    --
> >    This would have 2 DM paths.
> >    The I_T nexus's are:
> >       HBA1_TargetPort1
> >       HBA2_TargetPort2
> > 
> > Example 2:
> >    Fabric A contains HBA1, TargetPort1, and TargetPort3. 
> > Typically a zone
> >      would exist within Fabric A that contains these 3 ports.
> >    Fabric B contains HBA2, TargetPort2, and TargetPort4. 
> > Typically a zone
> >      would exist within Fabric B that contains these 3 ports.
> >    The storage array contains 2 controllers, with 2 ports per 
> > controller.
> >      Meaning TargetPort1 and TargetPort2 are on controller A, 
> > and TargetPort3
> >      and TargetPort4 are on controller B. The ports are cross 
> > connected on
> >      the fabrics for redundancy. All storage can be seen from 
> > any port.
> >    --
> >    This would ahve 4 DM paths
> >    The I_T nexus's are:
> >       HBA1_TargetPort1
> >       HBA1_TargetPort2
> >       HBA2_TargetPort3
> >       HBA2_TargetPort4
> > 
> > -- james
> > 
> > John Little wrote:
> > > Hi all
> > > 
> > > Admittedly this is not the correct forum for this question 
> > but I have googled 
> > > this and the only people I have to ask around here are 
> > people who won't touch 
> > > it because it is Linux.
> > > 
> > > My question is this:  I'm using SLES10 with two emulex 
> > hbas, one connected to 
> > > fabric a and one to fabric b.  When zoning the switch for 
> > the hbas do I put 
> > > in only one path to the fabric per hba or do I put in two, 
> > one to each of the 
> > > separate fabrics?
> > > 
> > > Again I realize this is not exactly the correct forum but 
> > since I couldn't 
> > > find an answer anywhere else I figured you guys would be the most 
> > > knowledgeable.  If there are some docs somewhere that you 
> > could point me to I 
> > > would appreciate it.
> > > 
> > > John
> > > 
> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> > > 
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > 
> > 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

-- 
Philip R. Auld, Ph.D.  	        	       Egenera, Inc.    
Software Architect                            165 Forest St.
(508) 858-2628                            Marlboro, MA 01752

Re: multipath and san fabric

[James Smart]

Yep - Ed caught my typo....  :)

-- james

Philip R. Auld wrote:
> Rumor has it that on Mon, Nov 13, 2006 at 02:39:08PM -0500 Edward Goggin said:
>> You may want to also cross connect across the 2 target
>> controllers (A,B) in case 2.  Doing so provides more
>> redundancy for asymmetric arrays and better resource
>> utilization in case of a failure of either an HBA or
>> target controller which would otherwise prevent
>> utilization of the other HBA or target controller
>> for that LU when load sharing I/O across multiple
>> paths.
>>
>> So the I_T nexus's for the 4 paths are:
>>
>> 	HBA1_TargetPort1
>> 	HBA1_TargetPort3
>> 	HBA2_TargetPort2
>> 	HBA2_TargetPort4
>>
> 
> That's exactly what James's example 2 describes. It's just 
> typo'd in the DM paths list. 
> 
> 
> 
>>> -----Original Message-----
>>> From: dm-devel-bounces@redhat.com 
>>> [mailto:dm-devel-bounces@redhat.com] On Behalf Of James Smart
>>> Sent: Monday, November 13, 2006 2:18 PM
>>> To: jslittl@hendricks.org; device-mapper development
>>> Subject: Re: [dm-devel] multipath and san fabric
>>>
>>> If the hba's are on different fabrics - why would you ever put an hba
>>> that is not in the fabric in a zone in that fabric ?  (e.g. 
>>> your second
>>> option makes no sense).
>>>
>>> If you are stating that the 2hba's are on different fabrics 
>>> (and in zones
>>> within those fabric) - but they are seeing the same Storage 
>>> device on both
>>> fabrics - then the dm config (note: I'm not talking about 
>>> zoning anymore)
>>> would have a path per I_T nexus. I state it this way to 
>>> account for dual-ported
>>> controllers, which may appear as separate targets, may exist 
>>> within the fabric.
>>>
>>> Example 1:
>>>    Fabric A contains HBA1 and TargetPort1. Typically a zone 
>>> would exist
>>>      within Fabric A that contains these 2 ports.
>>>    Fabric B contains HBA2 and TargetPort2. Typically a zone 
>>> would exist
>>>      within Fabric B that contains these 2 ports.
>>>    The storage array contains 2 controllers with 1 port per 
>>> controller, or
>>>      is a single controller with 2 ports. Either way, the two 
>>> ports are
>>>      TargetPort1 and TargetPort2, are on the same array, and 
>>> all storage
>>>      can be seen from any port.
>>>    --
>>>    This would have 2 DM paths.
>>>    The I_T nexus's are:
>>>       HBA1_TargetPort1
>>>       HBA2_TargetPort2
>>>
>>> Example 2:
>>>    Fabric A contains HBA1, TargetPort1, and TargetPort3. 
>>> Typically a zone
>>>      would exist within Fabric A that contains these 3 ports.
>>>    Fabric B contains HBA2, TargetPort2, and TargetPort4. 
>>> Typically a zone
>>>      would exist within Fabric B that contains these 3 ports.
>>>    The storage array contains 2 controllers, with 2 ports per 
>>> controller.
>>>      Meaning TargetPort1 and TargetPort2 are on controller A, 
>>> and TargetPort3
>>>      and TargetPort4 are on controller B. The ports are cross 
>>> connected on
>>>      the fabrics for redundancy. All storage can be seen from 
>>> any port.
>>>    --
>>>    This would ahve 4 DM paths
>>>    The I_T nexus's are:
>>>       HBA1_TargetPort1
>>>       HBA1_TargetPort2
>>>       HBA2_TargetPort3
>>>       HBA2_TargetPort4
>>>
>>> -- james
>>>
>>> John Little wrote:
>>>> Hi all
>>>>
>>>> Admittedly this is not the correct forum for this question 
>>> but I have googled 
>>>> this and the only people I have to ask around here are 
>>> people who won't touch 
>>>> it because it is Linux.
>>>>
>>>> My question is this:  I'm using SLES10 with two emulex 
>>> hbas, one connected to 
>>>> fabric a and one to fabric b.  When zoning the switch for 
>>> the hbas do I put 
>>>> in only one path to the fabric per hba or do I put in two, 
>>> one to each of the 
>>>> separate fabrics?
>>>>
>>>> Again I realize this is not exactly the correct forum but 
>>> since I couldn't 
>>>> find an answer anywhere else I figured you guys would be the most 
>>>> knowledgeable.  If there are some docs somewhere that you 
>>> could point me to I 
>>>> would appreciate it.
>>>>
>>>> John
>>>>
>>>> --
>>>> dm-devel mailing list
>>>> dm-devel@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>>
>>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
> 

RE: multipath and san fabric

[John Little]

Hi all

Wow.  thanks for the responses...

James Smart <James.Smart@Emulex.Com> wrote on 11/13/2006 02:54:15 PM:

> >>> Example 2:
> >>>    Fabric A contains HBA1, TargetPort1, and TargetPort3. 
> >>> Typically a zone
> >>>      would exist within Fabric A that contains these 3 ports.
> >>>    Fabric B contains HBA2, TargetPort2, and TargetPort4. 
> >>> Typically a zone
> >>>      would exist within Fabric B that contains these 3 ports.
> >>>    The storage array contains 2 controllers, with 2 ports per 
> >>> controller.
> >>>      Meaning TargetPort1 and TargetPort2 are on controller A, 
> >>> and TargetPort3
> >>>      and TargetPort4 are on controller B. The ports are cross 
> >>> connected on
> >>>      the fabrics for redundancy. All storage can be seen from 
> >>> any port.
> >>>    --
> >>>    This would ahve 4 DM paths
> >>>    The I_T nexus's are:
> >>>       HBA1_TargetPort1
> >>>       HBA1_TargetPort3
> >>>       HBA2_TargetPort2
> >>>       HBA2_TargetPort4

Ok I'm new to SAN and multipathing so maybe it's just the way it is worded.  
Here is what we have and the instructions that were given to me.
####
FCSW00 xxx.xxx.xxx.233 Fabric A 
FCSW01 xxx.xxx.xxx.234 Fabric B
FCSW02 xxx.xxx.xxx.235 Fabric A
FCSW03 xxx.xxx.xxx.236 Fabric B

Summary: One HBA is attached to each fabric; two zones per HBA. One is zoned 
to a port on SPA and one is zoned to a port on SPB We have a CX600 which has 
4 ports per SP. So, it could be zoned to SPA0 (0-3)and SPB0(0-3)
####

If I understand correctly what I have is Example 2 (typo corrected).  Then I 
would, for an hba connected to Fabric A on a port on FCSW00, setup a zone on 
FCSW00 and another on FCSW02.  The same would apply for the hba in Fabric B 
using FCSW01 and FCSW03.

Is this correct or am I still lost?

Thanks,
John

RE: multipath and san fabric

[Edward Goggin]

> -----Original Message-----
> From: dm-devel-bounces@redhat.com 
> [mailto:dm-devel-bounces@redhat.com] On Behalf Of John Little
> Sent: Tuesday, November 14, 2006 8:09 AM
> To: dm-devel@redhat.com
> Cc: James.Smart@emulex.com; goggin, edward
> Subject: RE: [dm-devel] multipath and san fabric
> 
> Hi all
> 
> Wow.  thanks for the responses...
> 
> James Smart <James.Smart@Emulex.Com> wrote on 11/13/2006 02:54:15 PM:
> 
> > >>> Example 2:
> > >>>    Fabric A contains HBA1, TargetPort1, and TargetPort3. 
> > >>> Typically a zone
> > >>>      would exist within Fabric A that contains these 3 ports.
> > >>>    Fabric B contains HBA2, TargetPort2, and TargetPort4. 
> > >>> Typically a zone
> > >>>      would exist within Fabric B that contains these 3 ports.
> > >>>    The storage array contains 2 controllers, with 2 ports per 
> > >>> controller.
> > >>>      Meaning TargetPort1 and TargetPort2 are on controller A, 
> > >>> and TargetPort3
> > >>>      and TargetPort4 are on controller B. The ports are cross 
> > >>> connected on
> > >>>      the fabrics for redundancy. All storage can be seen from 
> > >>> any port.
> > >>>    --
> > >>>    This would ahve 4 DM paths
> > >>>    The I_T nexus's are:
> > >>>       HBA1_TargetPort1
> > >>>       HBA1_TargetPort3
> > >>>       HBA2_TargetPort2
> > >>>       HBA2_TargetPort4
> 
> Ok I'm new to SAN and multipathing so maybe it's just the way 
> it is worded.  
> Here is what we have and the instructions that were given to me.
> ####
> FCSW00 xxx.xxx.xxx.233 Fabric A 
> FCSW01 xxx.xxx.xxx.234 Fabric B
> FCSW02 xxx.xxx.xxx.235 Fabric A
> FCSW03 xxx.xxx.xxx.236 Fabric B
> 
> Summary: One HBA is attached to each fabric; two zones per 
> HBA. One is zoned 
> to a port on SPA and one is zoned to a port on SPB We have a 
> CX600 which has 
> 4 ports per SP. So, it could be zoned to SPA0 (0-3)and SPB0(0-3)
> ####
> 
> If I understand correctly what I have is Example 2 (typo 
> corrected).

Yes, you have a modified version of James's "Example 2",
since you have 4 ports per CLARiiON controller instead of 2. 

> Then I 
> would, for an hba connected to Fabric A on a port on FCSW00, 
> setup a zone on 
> FCSW00 and another on FCSW02.  The same would apply for the 
> hba in Fabric B 
> using FCSW01 and FCSW03.

You need only configure a single zone for the active zone set
for each fabric as shown below using your terminology from above
to refer to the CLARiiON SP (A,B) and ports (0,1,2,3).

      Fabric A zone = {HBA1, SPA0, SPB0}
	Fabric B zone = {HBA2, SPA1, SPB1}

> 
> Is this correct or am I still lost?
> 
> Thanks,
> John
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
> 
>