Multipath problem in a SAN enviromnent

Multipath problem in a SAN enviromnent

[Ionut Nistor]

Hello,

I'm using a 2.4.18-ac3 kernel and the multipath module in order to ensure
automatic IO path failover. The systems connect to a FC SAN environment.
New logical disks are added/removed from the SAN periodically.

Unfortunatelly, linux's raid subsystem uses major and minor numbers in the
superblock in order to identify members for all raid types - including
multipath. This causes the multipath module to fail when attempting to
identify the members. When the scsi driver is loaded, the bus is scanned and
all the available devices/targets/luns are added to the scsi subsystem and
are allocated a major and minor number.

Say initially, the following are available

scsi1, bus0, target0, lun0
scsi1, bus0, target0, lun1
scsi1, bus0, target0, lun2
scsi2, bus0, target0, lun0
scsi2, bus0, target0, lun1
scsi2, bus0, target0, lun2

They will be mapped as

/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf

and

8,0
8,16
8,32
8,48
8,64
8,80

majors,minors respectivelly.

The disks with LUNs 0, 1, 2 are the same but are viewed on two paths:

/dev/sda - /dev/sdd
/dev/sdb - /dev/sde
/dev/sdc - /dev/sdf

Multipath is created in this configuration and works properly.

Now LUN1 is removed. The new configuration is:

scsi1, bus0, target0, lun0
scsi1, bus0, target0, lun2
scsi2, bus0, target0, lun0
scsi2, bus0, target0, lun2

Unfortunatelly, majors and minors became
8,0
8,16
8,32
8,48
So scsi1, bus0, target0, lun2 is now 8,16 instead of 8,32

However, the raid superblock still has 8,32 in it - this confuses
multipath - I even managed to get it crashing really ugly at some point.
The problem is partially caused by md identifying devices based on
major,minor and partially caused by the fact the the scsi subsystem has no
way (afaik) of mapping device,channel,target,lun to major,minor pairs.
This mapping is being performed dynamically by devfs for user-space
programs, but kernel modules that need to access devices via major,minor
have this problem.

Is there something I am missing? Is this a lack in the kernel?

Thanks,
Ionut

Re: Multipath problem in a SAN enviromnent

[Dominik Kubla]

Am Mittwoch, 11. Juni 2003 16:09 schrieb Ionut Nistor:

> Is there something I am missing? Is this a lack in the kernel?

Yes. For the second I am not sure.

You need to map your drives WWN's to SCSI-ID's. And no, the stock kernels 
apparently can not do this (at least i found not documentation and no 
indications in the code). However all major FC-HBA vendors offer their own 
drivers for Linux which allow you to do WWN mapping.

Regards,
  Dominik Kubla
-- 
ScioByte GmbH                     | ScioByte Information Technologies AG
Fritz-Erler-Strasse 6             | Innere Güterstrasse 4
55129 Mainz (Germany)             | 6304 Zug (Switzerland)
Phone: +49 700 724 629 83         | Phone: +41 41 710 30 18
Fax:   +49 700 724 629 84         |

GnuPG: 1024D/717F16BB A384 F5F1 F566 5716 5485  27EF 3B00 C007 717F 16BB

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

I can map WWNs to scsi id (target) - this does not help at all cause the
LUNs are changing, not the targets.

Ionut

----- Original Message ----- 
From: "Dominik Kubla" <dominik.kubla@sciobyte.com>
To: "Ionut Nistor" <ionut@modulo.ro>; <linux-raid@vger.kernel.org>
Sent: Wednesday, June 11, 2003 6:49 PM
Subject: Re: Multipath problem in a SAN enviromnent


Am Mittwoch, 11. Juni 2003 16:09 schrieb Ionut Nistor:

> Is there something I am missing? Is this a lack in the kernel?

Yes. For the second I am not sure.

You need to map your drives WWN's to SCSI-ID's. And no, the stock kernels
apparently can not do this (at least i found not documentation and no
indications in the code). However all major FC-HBA vendors offer their own
drivers for Linux which allow you to do WWN mapping.

Regards,
  Dominik Kubla
-- 
ScioByte GmbH                     | ScioByte Information Technologies AG
Fritz-Erler-Strasse 6             | Innere Güterstrasse 4
55129 Mainz (Germany)             | 6304 Zug (Switzerland)
Phone: +49 700 724 629 83         | Phone: +41 41 710 30 18
Fax:   +49 700 724 629 84         |

GnuPG: 1024D/717F16BB A384 F5F1 F566 5716 5485  27EF 3B00 C007 717F 16BB


-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Paul Clements]

Ionut Nistor wrote:

> So scsi1, bus0, target0, lun2 is now 8,16 instead of 8,32
> 
> However, the raid superblock still has 8,32 in it - this confuses
> multipath 

One way to get around this problem is to use non-persistent superblocks
for your arrays and just re-create them at bootup time by calling mkraid
(or mdadm) after you've correctly identified (using WWID or similar)
which devices belong to each particular raid set.

--
Paul

Moving Disks [Re: Multipath problem in a SAN enviromnent]

[Gordon Henderson]

On Wed, 11 Jun 2003, Paul Clements wrote:

> Ionut Nistor wrote:
>
> > So scsi1, bus0, target0, lun2 is now 8,16 instead of 8,32
> >
> > However, the raid superblock still has 8,32 in it - this confuses
> > multipath
>
> One way to get around this problem is to use non-persistent superblocks
> for your arrays and just re-create them at bootup time by calling mkraid
> (or mdadm) after you've correctly identified (using WWID or similar)
> which devices belong to each particular raid set.

Hm. This is vaguely intersting to me, as soon I'm going to be moving 8
drives off one server (arranged as 2 md RAID5's, 4 drives each) into a
server which already has 4 drives fitted... (configured as various md
RAID1 and RAID5 devices)

Should I expect any problems? Can I degrade an existing running set into
non persistent superblock mode to facilitate with this move?

Gordon

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Yes, you would get around it. But the main point of the superblock (i.e.
disk membership/status/array uuid) is to be able to identify the disks in an
array if devices allocation changes.

There must be a *safe* way of identifying the devices.

-
Ionut


----- Original Message ----- 
From: "Paul Clements" <Paul.Clements@SteelEye.com>
To: "Ionut Nistor" <ionut@modulo.ro>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, June 11, 2003 7:27 PM
Subject: Re: Multipath problem in a SAN enviromnent


> Ionut Nistor wrote:
>
> > So scsi1, bus0, target0, lun2 is now 8,16 instead of 8,32
> >
> > However, the raid superblock still has 8,32 in it - this confuses
> > multipath
>
> One way to get around this problem is to use non-persistent superblocks
> for your arrays and just re-create them at bootup time by calling mkraid
> (or mdadm) after you've correctly identified (using WWID or similar)
> which devices belong to each particular raid set.
>
> --
> Paul
>

Re: Multipath problem in a SAN enviromnent

[Paul Clements]

Ionut Nistor wrote:
> 
> Yes, you would get around it. But the main point of the superblock (i.e.
> disk membership/status/array uuid) is to be able to identify the disks in an
> array if devices allocation changes.
> 
> There must be a *safe* way of identifying the devices.

I agree that for most raid levels this is the case, but for multipath
there's nothing unsafe about probing the paths at boot time and
assembling the multipath md devices by matching up SCSI IDs (or FC
WWNs). This is how the QLogic (and other) HBA drivers do their multipath
setup.

But getting back to the original problem, it does seem like we should be
looking for matching superblock UUID (only) on the different paths
rather than matching major,minor since those can obviously change. 

Could you maybe post your boot messages (from md) so we can see what md
thinks is going on? I assume your partitions are IDed as RAID (0xfd) and
you're using the md autorun feature to start up your arrays?

--
Paul

 
> ----- Original Message -----
> From: "Paul Clements" <Paul.Clements@SteelEye.com>
> To: "Ionut Nistor" <ionut@modulo.ro>
> Cc: <linux-raid@vger.kernel.org>
> Sent: Wednesday, June 11, 2003 7:27 PM
> Subject: Re: Multipath problem in a SAN enviromnent
> 
> > Ionut Nistor wrote:
> >
> > > So scsi1, bus0, target0, lun2 is now 8,16 instead of 8,32
> > >
> > > However, the raid superblock still has 8,32 in it - this confuses
> > > multipath
> >
> > One way to get around this problem is to use non-persistent superblocks
> > for your arrays and just re-create them at bootup time by calling mkraid
> > (or mdadm) after you've correctly identified (using WWID or similar)
> > which devices belong to each particular raid set.
> >
> > --
> > Paul
> >
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Hi again,

When you're talking about mapping, you only reffer to SCSI IDs (i.e.
targets). I found no way of mapping the LUN IDs to specific kernel devices.
When you're using controllers with multiple disks and you map SCSI IDs to
devices all is fine. The problem appears when you are dealing with shifting
LUNs.

Matching *could* be done by raid superblock UUID (mdadm does such a thing)
but, if the major an minor in the superblock (the ones that reffer to 'this'
disk) are not the 'real' ones, md goes bezerk.

More specific, as in my initial example,

scsi1, bus0, target0, lun2 was 8,32
when scsi1, bus0, target0, lun1 is removed and you reload the FC HBA module,
scsi1, bus0, target0, lun2 becomes 8,16

When MD starts (the first disk in raidtab is /dev/sdb - cause i've changed
it to match the configuration - if using devfsd no change necessary at this
level) it looks in the superblock and finds
8,32 and 8,64 - obviously they are not the same and multipath crashes.

Ionut


----- Original Message ----- 
From: "Paul Clements" <Paul.Clements@SteelEye.com>
To: "Ionut Nistor" <ionut@modulo.ro>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, June 11, 2003 8:59 PM
Subject: Re: Multipath problem in a SAN enviromnent


> Ionut Nistor wrote:
> >
> > Yes, you would get around it. But the main point of the superblock (i.e.
> > disk membership/status/array uuid) is to be able to identify the disks
in an
> > array if devices allocation changes.
> >
> > There must be a *safe* way of identifying the devices.
>
> I agree that for most raid levels this is the case, but for multipath
> there's nothing unsafe about probing the paths at boot time and
> assembling the multipath md devices by matching up SCSI IDs (or FC
> WWNs). This is how the QLogic (and other) HBA drivers do their multipath
> setup.
>
> But getting back to the original problem, it does seem like we should be
> looking for matching superblock UUID (only) on the different paths
> rather than matching major,minor since those can obviously change.
>
> Could you maybe post your boot messages (from md) so we can see what md
> thinks is going on? I assume your partitions are IDed as RAID (0xfd) and
> you're using the md autorun feature to start up your arrays?
>
> --
> Paul

Re: Multipath problem in a SAN enviromnent

[Lars Marowsky-Bree]

On 2003-06-11T12:27:10,
   Paul Clements <Paul.Clements@SteelEye.com> said:

> One way to get around this problem is to use non-persistent superblocks
> for your arrays and just re-create them at bootup time by calling mkraid
> (or mdadm) after you've correctly identified (using WWID or similar)
> which devices belong to each particular raid set.

The patches I did to multipath in 2.4 went even further: As the md
superblock - this_disk etc - has little meaning for multipath, as they
are all identical, I made the assembly based entirely on the UUID,
ignoring the device nodes completely.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Hello Lars,

Are these changes documented?
Is there a changelog?
In which official kernel has the patch been merged?
Is there a separate patch I can apply to 2.4.18 vanilla?

Thanks a lot,
Ionut

----- Original Message ----- 
From: "Lars Marowsky-Bree" <lmb@suse.de>
To: "Paul Clements" <Paul.Clements@SteelEye.com>; "Ionut Nistor"
<ionut@modulo.ro>
Cc: <linux-raid@vger.kernel.org>
Sent: Thursday, June 12, 2003 1:04 PM
Subject: Re: Multipath problem in a SAN enviromnent


> On 2003-06-11T12:27:10,
>    Paul Clements <Paul.Clements@SteelEye.com> said:
>
> > One way to get around this problem is to use non-persistent superblocks
> > for your arrays and just re-create them at bootup time by calling mkraid
> > (or mdadm) after you've correctly identified (using WWID or similar)
> > which devices belong to each particular raid set.
>
> The patches I did to multipath in 2.4 went even further: As the md
> superblock - this_disk etc - has little meaning for multipath, as they
> are all identical, I made the assembly based entirely on the UUID,
> ignoring the device nodes completely.
>
>
> Sincerely,
>     Lars Marowsky-Brée <lmb@suse.de>
>
> -- 
> SuSE Labs - Research & Development, SuSE Linux AG
>
> "If anything can go wrong, it will." "Chance favors the prepared (mind)."
>   -- Capt. Edward A. Murphy            -- Louis Pasteur
>

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Paul Clements]

Ionut Nistor wrote:

> In which official kernel has the patch been merged?

It's been in the SuSE kernel for a while (2.4.19 and higher). It does
not appear to be in mainline yet (I checked 2.4.21-rc4).

 
> Is there a separate patch I can apply to 2.4.18 vanilla?

The patch is at the URL I posted in the other message yesterday:

http://lwn.net/Articles/8485/

or you can surely get it from SuSE's website.

--
Paul

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Agreed, but Lars mentioned 'the patch in 2.4 went even further' -  hence I
assume what's at the link you posted is obsolete - am I correct?
While I'm patching it, I want to make sure it's the latest one.

Ionut

----- Original Message ----- 
From: "Paul Clements" <Paul.Clements@SteelEye.com>
To: "Ionut Nistor" <ionut@modulo.ro>
Cc: "Lars Marowsky-Bree" <lmb@suse.de>; <linux-raid@vger.kernel.org>
Sent: Thursday, June 12, 2003 5:52 PM
Subject: Re: Multipath problem in a SAN enviromnent


> Ionut Nistor wrote:
>
> > In which official kernel has the patch been merged?
>
> It's been in the SuSE kernel for a while (2.4.19 and higher). It does
> not appear to be in mainline yet (I checked 2.4.21-rc4).
>
>
> > Is there a separate patch I can apply to 2.4.18 vanilla?
>
> The patch is at the URL I posted in the other message yesterday:
>
> http://lwn.net/Articles/8485/
>
> or you can surely get it from SuSE's website.
>
> --
> Paul
>

Re: Multipath problem in a SAN enviromnent

[Lars Marowsky-Bree]

On 2003-06-13T11:12:15,
   Ionut Nistor <ionut@modulo.ro> said:

> Agreed, but Lars mentioned 'the patch in 2.4 went even further' -  hence I
> assume what's at the link you posted is obsolete - am I correct?
> While I'm patching it, I want to make sure it's the latest one.

I've pulled out the latest patch set against our kernel and made it
available at ftp://ftp.suse.com/pub/people/lmb/md-mp/; however, it
probably will not apply cleanly against 2.4.21 yet.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Thanks Lars,

I'll try this and see how it goes.
I've seen that you also included a mdadm patch (probably to support the
other ioctls clean, active, inactive).

Have a nice day,
Ionut

----- Original Message ----- 
From: "Lars Marowsky-Bree" <lmb@suse.de>
To: "Ionut Nistor" <ionut@modulo.ro>; "Paul Clements"
<Paul.Clements@SteelEye.com>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, June 18, 2003 1:21 PM
Subject: Re: Multipath problem in a SAN enviromnent


> On 2003-06-13T11:12:15,
>    Ionut Nistor <ionut@modulo.ro> said:
>
> > Agreed, but Lars mentioned 'the patch in 2.4 went even further' -  hence
I
> > assume what's at the link you posted is obsolete - am I correct?
> > While I'm patching it, I want to make sure it's the latest one.
>
> I've pulled out the latest patch set against our kernel and made it
> available at ftp://ftp.suse.com/pub/people/lmb/md-mp/; however, it
> probably will not apply cleanly against 2.4.21 yet.
>
>
> Sincerely,
>     Lars Marowsky-Brée <lmb@suse.de>
>
> -- 
> SuSE Labs - Research & Development, SuSE Linux AG
>
> "If anything can go wrong, it will." "Chance favors the prepared (mind)."
>   -- Capt. Edward A. Murphy            -- Louis Pasteur
>

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Multipath problem in a SAN enviromnent

[Luca Berra]

Ionut Nistor wrote:
> Hello,
> 
> I'm using a 2.4.18-ac3 kernel and the multipath module in order to ensure
> automatic IO path failover. The systems connect to a FC SAN environment.
> New logical disks are added/removed from the SAN periodically.
> 
> Unfortunatelly, linux's raid subsystem uses major and minor numbers in the
> superblock in order to identify members for all raid types - including
> multipath. This causes the multipath module to fail when attempting to
> identify the members. When the scsi driver is loaded, the bus is scanned and
> all the available devices/targets/luns are added to the scsi subsystem and
> are allocated a major and minor number.
> 
you mean autodetection fails? don't use autodetection.

regards,
Luca.

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Hi,

What do you mean by autodetection?

Ionut
----- Original Message ----- 
From: "Luca Berra" <bluca@comedia.it>
To: "Ionut Nistor" <ionut@modulo.ro>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, June 11, 2003 10:24 PM
Subject: Re: Multipath problem in a SAN enviromnent


> Ionut Nistor wrote:
> > Hello,
> >
> > I'm using a 2.4.18-ac3 kernel and the multipath module in order to
ensure
> > automatic IO path failover. The systems connect to a FC SAN environment.
> > New logical disks are added/removed from the SAN periodically.
> >
> > Unfortunatelly, linux's raid subsystem uses major and minor numbers in
the
> > superblock in order to identify members for all raid types - including
> > multipath. This causes the multipath module to fail when attempting to
> > identify the members. When the scsi driver is loaded, the bus is scanned
and
> > all the available devices/targets/luns are added to the scsi subsystem
and
> > are allocated a major and minor number.
> >
> you mean autodetection fails? don't use autodetection.
>
> regards,
> Luca.
>
>

Re: Multipath problem in a SAN enviromnent

[Paul Clements]

Ionut Nistor wrote:

> I'm using a 2.4.18-ac3 kernel 

It just dawned on me that this kernel probably does not have the
following patch applied:

http://lwn.net/Articles/8485/

If that's the case, you may want to move to a later kernel or apply that
patch. I believe the patch will fix your issues with the shifting minor
numbers causing problems. 

I did a brief test on SuSE's 2.4.19 kernel, which has this patch
applied, and everything worked, even after shifting devices around.

--
Paul

Re: Multipath problem in a SAN enviromnent

[Ionut Nistor]

Great,

Do you happen to know if this patch was included in the kernel? If so, at
what version?

I need to know how reliable it is.

Thanks a lot,
Ionut

----- Original Message ----- 
From: "Paul Clements" <Paul.Clements@SteelEye.com>
To: "Ionut Nistor" <ionut@modulo.ro>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, June 11, 2003 10:34 PM
Subject: Re: Multipath problem in a SAN enviromnent


> Ionut Nistor wrote:
>
> > I'm using a 2.4.18-ac3 kernel
>
> It just dawned on me that this kernel probably does not have the
> following patch applied:
>
> http://lwn.net/Articles/8485/
>
> If that's the case, you may want to move to a later kernel or apply that
> patch. I believe the patch will fix your issues with the shifting minor
> numbers causing problems.
>
> I did a brief test on SuSE's 2.4.19 kernel, which has this patch
> applied, and everything worked, even after shifting devices around.
>
> --
> Paul
>