EMC Clariion ghost devices

EMC Clariion ghost devices

[Joel Becker]

Christophe,
	I've just tried using the multipath-tools (SLES9 SP2,
multipath-tools-0.4.4-0.22) with my EMC clariion.  Now, this clariion is
an active/passive system, and the second path is showing a ghost LUN 0
(this is expected).
	During normal boot, the device (/dev/sdap) returns an I/O error
when the kernel tries to get the partition table.  No problem.  If later
someone scans the disk, they get an I/O error.  No problem.
	Then I fire up multipathd and multipath.  Multipath creates a dm
device for the ghost, and tries to run kpartx on it.  That kpartx hangs.
In fact, I don't know how, but it ends up in a frozen I/O state.  It's
unkillable.  I'd expect an I/O error, but no, it just plain doesn't
exit.  Ditto any other access to the associated dm.
	Now, is this a dm problem (the device not seeing the error and
returning it)?  Is it a multipath problem (multipath should notice the
ghost and not try to dm it)?  Is it a configuration problem (I should
somehow know which name the ghost got, and configure multipath to ignore
it)?

Joel

-- 

Life's Little Instruction Book #182

	"Be romantic."

Joel Becker
Senior Member of Technical Staff
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: EMC Clariion ghost devices

[christophe varoqui]

On ven, 2005-07-29 at 15:28 -0700, Joel Becker wrote:
> Christophe,
> 	I've just tried using the multipath-tools (SLES9 SP2,
> multipath-tools-0.4.4-0.22) with my EMC clariion.  Now, this clariion is
> an active/passive system, and the second path is showing a ghost LUN 0
> (this is expected).
> 	During normal boot, the device (/dev/sdap) returns an I/O error
> when the kernel tries to get the partition table.  No problem.  If later
> someone scans the disk, they get an I/O error.  No problem.
> 	Then I fire up multipathd and multipath.  Multipath creates a dm
> device for the ghost, and tries to run kpartx on it.  That kpartx hangs.
> In fact, I don't know how, but it ends up in a frozen I/O state.  It's
> unkillable.  I'd expect an I/O error, but no, it just plain doesn't
> exit.  Ditto any other access to the associated dm.
> 	Now, is this a dm problem (the device not seeing the error and
> returning it)?  Is it a multipath problem (multipath should notice the
> ghost and not try to dm it)?  Is it a configuration problem (I should
> somehow know which name the ghost got, and configure multipath to ignore
> it)?

/sbin/multipath sure shouldn't create a multipath map with just ghost
paths in it. If it is the case, some paths' wwid must be wrong. May be
due to scsi_id inconsistencies ?

You can try and verify that with 'multipath -v4', reading the "all
paths" section.

I hope Lars will react on this one as he has the hardware at hand ...

Regards,
-- 
christophe varoqui <christophe.varoqui@free.fr>

Re: EMC Clariion ghost devices

[Lars Marowsky-Bree]

On 2005-07-29T15:28:03, Joel Becker <Joel.Becker@oracle.com> wrote:

> 	I've just tried using the multipath-tools (SLES9 SP2,
> multipath-tools-0.4.4-0.22) with my EMC clariion.  Now, this clariion is
> an active/passive system, and the second path is showing a ghost LUN 0
> (this is expected).

"WFM". I actually tested the SP2 code on a CLARiiON, it's my development
box ;-) The version of multipath-tools on SLES9 SP2 is not quite 0.4.4
from upstream, though, but with some fixes pulled in.

Have you seen
http://portal.suse.com/sdb/en/2005/04/sles_multipathing.html ?

> 	During normal boot, the device (/dev/sdap) returns an I/O error
> when the kernel tries to get the partition table.  No problem.  If later
> someone scans the disk, they get an I/O error.  No problem.

Yeah, this is ugly and annoying, but harmless.

> 	Then I fire up multipathd and multipath.  Multipath creates a dm
> device for the ghost, and tries to run kpartx on it.  That kpartx hangs.
> In fact, I don't know how, but it ends up in a frozen I/O state.  It's
> unkillable.

That should not happen(tm). Where does it hang according to sysrq-t?

(kpartx supposedly is run against the dm-multipath table and not against
sdap, so it should already benefit from the multipath protection...)

"sg_inq -P /dev/sdap" might also be worth a try, what does it report for
the various paths?

Is that a LUN 0 backed by actual storage or a LUN_Z?


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: EMC Clariion ghost devices

[Joel Becker]

On Mon, Aug 01, 2005 at 12:01:54AM +0200, Lars Marowsky-Bree wrote:
> That should not happen(tm). Where does it hang according to sysrq-t?

	Not in front of it right now, I'll let you know when I find out
:-)
>
> (kpartx supposedly is run against the dm-multipath table and not against
> sdap, so it should already benefit from the multipath protection...)

	This kpartx is run against dm-16 (on the particular boot I
remember), which was the dm on top of the ghost device.
 
> "sg_inq -P /dev/sdap" might also be worth a try, what does it report for
> the various paths?

	Again, have to check tomorrow.

> Is that a LUN 0 backed by actual storage or a LUN_Z?

	I don't quite know what you mean by LUN_Z.  My EMC guy says it
is a "ghost" projected because the second SP owns no disks, but SCSI
requires a LUN 0 to be scannable.

Joel

-- 

"A narcissist is someone better looking than you are."  
         - Gore Vidal

Joel Becker
Senior Member of Technical Staff
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

RE: EMC Clariion ghost devices

[Gaál Norbert]

Hi!

>> Is that a LUN 0 backed by actual storage or a LUN_Z?

>	I don't quite know what you mean by LUN_Z.  My EMC guy says it
>is a "ghost" projected because the second SP owns no disks, but SCSI
>requires a LUN 0 to be scannable.
>
>Joel

If no LUN 0 is present (or no LUN is in the Storage Group), and arraycommpath is enabled the Clariion 'emulates' a LUN this is a LUNZ.

From a EMC primus article:

A SCSI-3 (SCC-2) term defined as "the logical unit number that an application client uses to communicate with, configure and determine information about an SCSI storage array and the logical units attached to it. The LUN_Z value shall be zero." In the CLARiiON context, LUNz refers to a fake logical unit zero presented to the host to provide a path for host software to send configuration commands to the array when no physical logical unit zero is available to the host. When Access Logix is used on a CLARiiON array, an agent runs on the host and communicates with the storage system through either LUNz or a storage device. On a CLARiiON array, the LUNZ device is replaced when a valid LUN is assigned to the HLU LUN0 by the Storage Group. The agent then communicates through the storage device. The user will continue, however, to see DGC LUNz in the Device Manager. 
LUNz has been implemented on CLARiiON arrays to make arrays visible to the host OS and PowerPath when no LUNs are bound on that array.  When using a direct connect configuration, and there is no Navisphere Management station to talk directly to the array over IP, the LUNZ can be used as a pathway for Navisphere CLI to send Bind commands to the array. 

LUNz also makes arrays visible to the host OS and PowerPath when the host·s initiators have not yet ·logged in to the Storage Group created for the host.  Without LUNz, there would be no device on the host for Navisphere Agent to push the initiator record through to the array. This is mandatory for the host to log in to the Storage Group. Once this initiator push is done, the host will be displayed as an available host to add to the Storage Group in Navisphere Manager (Navisphere Express).

LUNz should disappear once a LUN zero is bound, or when Storage Group access has been attained.

To turn on the LUNz behavior on CLARiiON arrays, you must configure the "arraycommpath." 

Regards:
 Norbert Gaal

Re: EMC Clariion ghost devices

[Joel Becker]

On Mon, Aug 01, 2005 at 12:01:54AM +0200, Lars Marowsky-Bree wrote:
> That should not happen(tm). Where does it hang according to sysrq-t?
> 
> (kpartx supposedly is run against the dm-multipath table and not against
> sdap, so it should already benefit from the multipath protection...)
> 
> "sg_inq -P /dev/sdap" might also be worth a try, what does it report for
> the various paths?

	This reports a LUN_Z, which is what I expected after I
was explained a LUN_Z is the same as a ghost report.
	I get this in dmesg at multipath(8):

device-mapper: dm-multipath: 66:144: Error trying to initialize PG, failing path
device-mapper: dm-multipath: Failing path 66:144

66:144 is, indeed, /dev/sdap.  Kpartx is again hanging on:

# ps -ef | grep kpartx
root     26698 25674  0 10:53 ?        00:00:00 /sbin/kpartx -m -a /dev/dm-16

# ls -l /dev/disk/by-name | grep dm-16
/dev/disk/by-name/3500601609020ea1a500601609020ea1a -> ../../dm-16

# dmsetup deps 3500601609020ea1a500601609020ea1a
1 dependencies  : (66, 144)

# echo t > /proc/sysrq-trigger
kpartx        D 00000100bea0f680     0 26698  25674               25755 (NOTLB)
00000100c1511bc8 0000000000000006 00000100c05e00a0 000001d09566ce68
       0000000000000246 0000010081b54580 0000007300000001 000001007b608b10
       0000000100000001 0000010081b55660
Call Trace:<ffffffff802859a4>{generic_unplug_device+36} <ffffffff8013953f>{io_schedule+63}
       <ffffffff80163b39>{__lock_page_wq+233} <ffffffff8013ce40>{autoremove_wake_function+0}
       <ffffffff8016c518>{page_cache_readahead+328} <ffffffff8013ce40>{autoremove_wake_function+0}
       <ffffffff8016518c>{do_generic_mapping_read+876} <ffffffff801637a0>{file_read_actor+0}
       <ffffffff80166024>{__generic_file_aio_read+420} <ffffffff801661eb>{generic_file_read+187}
       <ffffffff8019592e>{do_open+318} <ffffffff8019712a>{cp_new_stat+234}
       <ffffffff8013ce40>{autoremove_wake_function+0} <ffffffff8018d184>{vfs_read+244}
       <ffffffff8018d3dd>{sys_read+157} <ffffffff80110794>{system_call+124}

-- 

"Can any of you seriously say the Bill of Rights could get through
 Congress today?  It wouldn't even get out of committee."
	- F. Lee Bailey

Joel Becker
Senior Member of Technical Staff
Oracle
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127