different LUN numbers under the same dm device

different LUN numbers under the same dm device

[Brian Bunker]

Our company produces a multiple port Fibre Channel storage array. We are continually plagued by this problem. We get a dm device which combines paths for different LUN's. We would like to understand why this is happening. Wouldn't this problem almost certainly lead to a data corruption?

Thanks,
Brian

3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
size=500G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:0:12 sde  8:64   failed faulty running
  |- 0:0:0:12 sdd  8:48   failed faulty running
  |- 0:0:0:10 sdar 66:176 active ready  running
  `- 1:0:0:10 sdba 67:64  active ready  running

Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the other two are for LUN 12. We have 24 other dm devices which have only the expected 2 paths. 

Brian Bunker
brian@purestorage.com

Re: different LUN numbers under the same dm device

[Mike Snitzer]

On Wed, Jun 06 2012 at  3:27pm -0400,
Brian Bunker <brian@purestorage.com> wrote:

> Our company produces a multiple port Fibre Channel storage array. We
> are continually plagued by this problem. We get a dm device which
> combines paths for different LUN's. We would like to understand why
> this is happening. Wouldn't this problem almost certainly lead to a
> data corruption?
> 
> Thanks,
> Brian
> 
> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
> size=500G features='0' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=1 status=active
>   |- 1:0:0:12 sde  8:64   failed faulty running
>   |- 0:0:0:12 sdd  8:48   failed faulty running
>   |- 0:0:0:10 sdar 66:176 active ready  running
>   `- 1:0:0:10 sdba 67:64  active ready  running
> 
> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the
> other two are for LUN 12. We have 24 other dm devices which have only
> the expected 2 paths.

Multipath considers all LUNs with the same UUID to be the same LUN.

So you should first try to understand why all of these paths were held
to have the same UUID (3624a93700a14254d729923840001000b).

Re: different LUN numbers under the same dm device

[Brian Bunker]

Mike,

The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this:

[root@r13init32 ~]# sg_inq /dev/sde
standard INQUIRY: [qualifier indicates no connected LU]
  PQual=1  Device_type=31  RMB=0  version=0x06  [SPC-4]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
  EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=96 (0x60)   Peripheral device type: no physical device on this lu
 Vendor identification: PURE    
 Product identification: FlashArray      
 Product revision level: 100 

There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. 

The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty:

  3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
  size=500G features='0' hwhandler='0' wp=rw
   `-+- policy='round-robin 0' prio=1 status=active
   |- 1:0:0:12 sde  8:64   failed faulty running
   |- 0:0:0:12 sdd  8:48   failed faulty running

Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect:

 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
 size=500G features='0' hwhandler='0' wp=rw
 `-+- policy='round-robin 0' prio=1 status=active
   |- 0:0:0:10 sdar 66:176 active ready  running
   !- 1:0:0:10 sdba 67:64  active ready  running

Thanks,
Brian

On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote:

> On Wed, Jun 06 2012 at  3:27pm -0400,
> Brian Bunker <brian@purestorage.com> wrote:
> 
>> Our company produces a multiple port Fibre Channel storage array. We
>> are continually plagued by this problem. We get a dm device which
>> combines paths for different LUN's. We would like to understand why
>> this is happening. Wouldn't this problem almost certainly lead to a
>> data corruption?
>> 
>> Thanks,
>> Brian
>> 
>> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>> size=500G features='0' hwhandler='0' wp=rw
>> `-+- policy='round-robin 0' prio=1 status=active
>>  |- 1:0:0:12 sde  8:64   failed faulty running
>>  |- 0:0:0:12 sdd  8:48   failed faulty running
>>  |- 0:0:0:10 sdar 66:176 active ready  running
>>  `- 1:0:0:10 sdba 67:64  active ready  running
>> 
>> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the
>> other two are for LUN 12. We have 24 other dm devices which have only
>> the expected 2 paths.
> 
> Multipath considers all LUNs with the same UUID to be the same LUN.
> 
> So you should first try to understand why all of these paths were held
> to have the same UUID (3624a93700a14254d729923840001000b).

Brian Bunker
brian@purestorage.com

Re: different LUN numbers under the same dm device

[Benjamin Marzinski]

On Wed, Jun 06, 2012 at 01:59:02PM -0700, Brian Bunker wrote:
> Mike,
> 
> The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this:
> 
> [root@r13init32 ~]# sg_inq /dev/sde
> standard INQUIRY: [qualifier indicates no connected LU]
>   PQual=1  Device_type=31  RMB=0  version=0x06  [SPC-4]
>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
>   SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
>   EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>   [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
>   [SPI: Clocking=0x0  QAS=0  IUS=0]
>     length=96 (0x60)   Peripheral device type: no physical device on this lu
>  Vendor identification: PURE    
>  Product identification: FlashArray      
>  Product revision level: 100 
> 
> There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. 
> 
> The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty:
> 
>   3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>   size=500G features='0' hwhandler='0' wp=rw
>    `-+- policy='round-robin 0' prio=1 status=active
>    |- 1:0:0:12 sde  8:64   failed faulty running
>    |- 0:0:0:12 sdd  8:48   failed faulty running
> 
> Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect:
> 
>  3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>  size=500G features='0' hwhandler='0' wp=rw
>  `-+- policy='round-robin 0' prio=1 status=active
>    |- 0:0:0:10 sdar 66:176 active ready  running
>    !- 1:0:0:10 sdba 67:64  active ready  running
> 
> Thanks,
> Brian

So, did sde and sdd (the paths for the multipath device for LUN 12)
originally have the wwid of 3624a93700a14254d729923840001000b.  Do
sdar and sdba actually have the wwid of
3624a93700a14254d729923840001000b.  You can check this by running

# scsi_id --whitelisted --device=<devname>

for example

# scsi_id --whitelisted --device=/dev/sde

If all of these scsi devices return the same value, then multipath
has no way of knowing that they don't belong together. If not, I'd
like to know which devices don't really have that UUID.

-Ben

> 
> On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote:
> 
> > On Wed, Jun 06 2012 at  3:27pm -0400,
> > Brian Bunker <brian@purestorage.com> wrote:
> > 
> >> Our company produces a multiple port Fibre Channel storage array. We
> >> are continually plagued by this problem. We get a dm device which
> >> combines paths for different LUN's. We would like to understand why
> >> this is happening. Wouldn't this problem almost certainly lead to a
> >> data corruption?
> >> 
> >> Thanks,
> >> Brian
> >> 
> >> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
> >> size=500G features='0' hwhandler='0' wp=rw
> >> `-+- policy='round-robin 0' prio=1 status=active
> >>  |- 1:0:0:12 sde  8:64   failed faulty running
> >>  |- 0:0:0:12 sdd  8:48   failed faulty running
> >>  |- 0:0:0:10 sdar 66:176 active ready  running
> >>  `- 1:0:0:10 sdba 67:64  active ready  running
> >> 
> >> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the
> >> other two are for LUN 12. We have 24 other dm devices which have only
> >> the expected 2 paths.
> > 
> > Multipath considers all LUNs with the same UUID to be the same LUN.
> > 
> > So you should first try to understand why all of these paths were held
> > to have the same UUID (3624a93700a14254d729923840001000b).
> 
> Brian Bunker
> brian@purestorage.com
> 
> 
> 
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: different LUN numbers under the same dm device

[Brian Bunker]

The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry. Then the LUN was unmasked from that initiator. That initiator is holding on to those device names in multipath. If you query them when they are in the state that I show in the multipath -ll result, they will not return an NAA number at all in page code 0x83 or a serial number in page code 0x80. They will instead return a PQ of 1 meaning that the LUN is capable of supporting a peripheral device but is not currently.

I understand about LUN's needing different NAA numbers and ours do, and we also have different LUN serial numbers for each LUN on the target. An initiator doesn't alway have to access to all LUN's that it once did. It is the re-use of dm devices that seems to cause this result.

Thanks,
Brian

On Jun 7, 2012, at 3:39 PM, Benjamin Marzinski wrote:

> On Wed, Jun 06, 2012 at 01:59:02PM -0700, Brian Bunker wrote:
>> Mike,
>> 
>> The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this:
>> 
>> [root@r13init32 ~]# sg_inq /dev/sde
>> standard INQUIRY: [qualifier indicates no connected LU]
>>  PQual=1  Device_type=31  RMB=0  version=0x06  [SPC-4]
>>  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
>>  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
>>  EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>>  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
>>  [SPI: Clocking=0x0  QAS=0  IUS=0]
>>    length=96 (0x60)   Peripheral device type: no physical device on this lu
>> Vendor identification: PURE    
>> Product identification: FlashArray      
>> Product revision level: 100 
>> 
>> There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. 
>> 
>> The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty:
>> 
>>  3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>>  size=500G features='0' hwhandler='0' wp=rw
>>   `-+- policy='round-robin 0' prio=1 status=active
>>   |- 1:0:0:12 sde  8:64   failed faulty running
>>   |- 0:0:0:12 sdd  8:48   failed faulty running
>> 
>> Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect:
>> 
>> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>> size=500G features='0' hwhandler='0' wp=rw
>> `-+- policy='round-robin 0' prio=1 status=active
>>   |- 0:0:0:10 sdar 66:176 active ready  running
>>   !- 1:0:0:10 sdba 67:64  active ready  running
>> 
>> Thanks,
>> Brian
> 
> So, did sde and sdd (the paths for the multipath device for LUN 12)
> originally have the wwid of 3624a93700a14254d729923840001000b.  Do
> sdar and sdba actually have the wwid of
> 3624a93700a14254d729923840001000b.  You can check this by running
> 
> # scsi_id --whitelisted --device=<devname>
> 
> for example
> 
> # scsi_id --whitelisted --device=/dev/sde
> 
> If all of these scsi devices return the same value, then multipath
> has no way of knowing that they don't belong together. If not, I'd
> like to know which devices don't really have that UUID.
> 
> -Ben
> 
>> 
>> On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote:
>> 
>>> On Wed, Jun 06 2012 at  3:27pm -0400,
>>> Brian Bunker <brian@purestorage.com> wrote:
>>> 
>>>> Our company produces a multiple port Fibre Channel storage array. We
>>>> are continually plagued by this problem. We get a dm device which
>>>> combines paths for different LUN's. We would like to understand why
>>>> this is happening. Wouldn't this problem almost certainly lead to a
>>>> data corruption?
>>>> 
>>>> Thanks,
>>>> Brian
>>>> 
>>>> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>>>> size=500G features='0' hwhandler='0' wp=rw
>>>> `-+- policy='round-robin 0' prio=1 status=active
>>>> |- 1:0:0:12 sde  8:64   failed faulty running
>>>> |- 0:0:0:12 sdd  8:48   failed faulty running
>>>> |- 0:0:0:10 sdar 66:176 active ready  running
>>>> `- 1:0:0:10 sdba 67:64  active ready  running
>>>> 
>>>> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the
>>>> other two are for LUN 12. We have 24 other dm devices which have only
>>>> the expected 2 paths.
>>> 
>>> Multipath considers all LUNs with the same UUID to be the same LUN.
>>> 
>>> So you should first try to understand why all of these paths were held
>>> to have the same UUID (3624a93700a14254d729923840001000b).
>> 
>> Brian Bunker
>> brian@purestorage.com
>> 
>> 
>> 
>> 
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Brian Bunker
brian@purestorage.com

Re: different LUN numbers under the same dm device

[Hannes Reinecke]

On 06/06/2012 09:27 PM, Brian Bunker wrote:
> Our company produces a multiple port Fibre Channel storage array. We are continually plagued
> by this problem. We get a dm device which combines paths for
different LUN's. We would like
> to understand why this is happening. Wouldn't this problem almost
certainly lead to a data
> corruption?
> 
Most likely; given that the two LUNs indeed refer to different disks.

> Thanks,
> Brian
> 
> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
> size=500G features='0' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=1 status=active
>   |- 1:0:0:12 sde  8:64   failed faulty running
>   |- 0:0:0:12 sdd  8:48   failed faulty running
>   |- 0:0:0:10 sdar 66:176 active ready  running
>   `- 1:0:0:10 sdba 67:64  active ready  running
> 
> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the other two are for LUN 12.
> We have 24 other dm devices which have only the expected 2 paths.
> 
Check the output of 'sg_vpd -p di /dev/sd(e|d|ar|ba)'.

'sde' and 'sdd' should be identical (well, the bits for 'Addressed
logical unit' should be), as should 'sdar' and 'sdab'.
'sde' and 'sdar', on the other hand, should _not_.

Given the above, however, I strongly suspect they are.
Looks like your LUN WWID generation code is buggered.

But I guess you can always blame Nic Bellinger :-)

Greetings to Joern Engel.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: different LUN numbers under the same dm device

[Hannes Reinecke]

On 06/06/2012 10:59 PM, Brian Bunker wrote:
> Mike,
> 
> The devices for LUN 12 are failed and correspond to LUN's not currently shared
> to the initiator at all. They were at one point and were likely
used by dm-11
> for its underlying paths. The inquiry data of those LUN's when the
problem happened was like this:
> 
> [root@r13init32 ~]# sg_inq /dev/sde
> standard INQUIRY: [qualifier indicates no connected LU]
>   PQual=1  Device_type=31  RMB=0  version=0x06  [SPC-4]
>   [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
>   SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
>   EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
>   [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
>   [SPI: Clocking=0x0  QAS=0  IUS=0]
>     length=96 (0x60)   Peripheral device type: no physical device on this lu
>  Vendor identification: PURE    
>  Product identification: FlashArray      
>  Product revision level: 100 
> 
> There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80
> since there is no LUN 12 attached as a disk device at the time
multipath -ll was run.
> Different LUN's from our array would ever have the same NAA value,
what I think you are calling UUID.
> 
Yep. Hmm. So the devices are unmapped from the storage, but still
visible from the initiator?
Have you run 'rescan-scsi-bus.sh -r' here?
That should clean up these devices.

> The sequence is something like share a LUN from the array with two paths to
> the initiator, a dm device gets created presumably like this at
first (except
> that the status would be active and ready and not failed and faulty:
> 
>   3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>   size=500G features='0' hwhandler='0' wp=rw
>    `-+- policy='round-robin 0' prio=1 status=active
>    |- 1:0:0:12 sde  8:64   failed faulty running
>    |- 0:0:0:12 sdd  8:48   failed faulty running
> 
> Then that LUN 12 is taken away from the initiator and the dm device dm-11 is
> reused later by LUN 10 when it is shared to the initiator, but the
LUN 12
> devices still remain as part of the dm device. Then I would expect:
> 
>  3624a93700a14254d729923840001000b dm-11 PURE,FlashArray
>  size=500G features='0' hwhandler='0' wp=rw
>  `-+- policy='round-robin 0' prio=1 status=active
>    |- 0:0:0:10 sdar 66:176 active ready  running
>    !- 1:0:0:10 sdba 67:64  active ready  running
> 

Yeah, but still: it means that at one point LUN 12 had the same NAA
value than LUN 10, correct?
It _might_ happen that multipath created a dm-device for LUN12, set
them to 'faulty' during unsharing, and then added the then-new LUN10
to the same device, given that the NAA number is identical.

So the point still stands: LUN10 must have had the same NAA value
than LUN12 now has.
So unless the original LUN10 referred to the same storage entity as
LUN12 now does, this is a definite no-no.
And if it does, we're pretty much in the clear, as then LUN10 would
now refer to a stale device (with status 'failed faulty'), and
should be cleared up with 'rescan-scsi-bus.sh -r'.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: different LUN numbers under the same dm device

[Hannes Reinecke]

(Third mail on this topic .. I really should've read the entire
thread before answering. But there you go.)

On 06/08/2012 01:26 AM, Brian Bunker wrote:
> The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry.
> Then the LUN was unmasked from that initiator. That initiator is
holding on to those device
> names in multipath. If you query them when they are in the state
that I show in the
> multipath -ll result, they will not return an NAA number at all in
page code 0x83 or
> a serial number in page code 0x80. They will instead return a PQ
of 1 meaning that the
> LUN is capable of supporting a peripheral device but is not currently.
> 
Hehe.

> I understand about LUN's needing different NAA numbers and ours do, and we also have
> different LUN serial numbers for each LUN on the target. An
initiator doesn't always
> have to access to all LUN's that it once did. It is the re-use of
dm devices that
> seems to cause this result.
> 
No, rather a problem with the SCSI stack. multipath normally relies
on udev to keep track of any device events, like LUNs coming and
going. But this only works reliably if these events are triggered
via the underlying transport, like FC RSCN et al.

'Real' scsi events which will get transmitted via Sense codes are
not evaluated further, sadly.

So short-term you are supposed to call 'rescan-scsi-bus.sh -r'
whenever you made any LUN assigment changes on the array.
Mid-term we already agreed on implementing some proper sense code
handling in the SCSI midlayer.
However, as usual in these cases, real life interfered and I've been
busy with other things.
But it's definitely on the agenda.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: different LUN numbers under the same dm device

[Mike Christie]

On 06/08/2012 01:59 AM, Hannes Reinecke wrote:
> (Third mail on this topic .. I really should've read the entire
> thread before answering. But there you go.)
> 
> On 06/08/2012 01:26 AM, Brian Bunker wrote:
>> The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry.
>> Then the LUN was unmasked from that initiator. That initiator is
> holding on to those device
>> names in multipath. If you query them when they are in the state
> that I show in the
>> multipath -ll result, they will not return an NAA number at all in
> page code 0x83 or
>> a serial number in page code 0x80. They will instead return a PQ
> of 1 meaning that the
>> LUN is capable of supporting a peripheral device but is not currently.
>>
> Hehe.
> 
>> I understand about LUN's needing different NAA numbers and ours do, and we also have
>> different LUN serial numbers for each LUN on the target. An
> initiator doesn't always
>> have to access to all LUN's that it once did. It is the re-use of
> dm devices that
>> seems to cause this result.
>>
> No, rather a problem with the SCSI stack. multipath normally relies
> on udev to keep track of any device events, like LUNs coming and
> going. But this only works reliably if these events are triggered
> via the underlying transport, like FC RSCN et al.
> 
> 'Real' scsi events which will get transmitted via Sense codes are
> not evaluated further, sadly.
> 
> So short-term you are supposed to call 'rescan-scsi-bus.sh -r'
> whenever you made any LUN assigment changes on the array.
> Mid-term we already agreed on implementing some proper sense code
> handling in the SCSI midlayer.
> However, as usual in these cases, real life interfered and I've been
> busy with other things.
> But it's definitely on the agenda.
> 

I think this is a bug in multipath though. Even though we are not
handling sense that would indicate the path/device is not longer valid
(PQ changed values) if it was sent, it would make sense that when
multipath is assembling devices it would not create a device with
different NAA/UUID values. So multipath should not make a device that
includes sde, sdd, sdar and sdba but where sde and sdd have a different
UUID than sdar and sdba, right?

Brian, what is the device returning in this case for sde and sdd? Does
it return a error, or is it returning invalid data thinking the OS would
check the PQ value first? If it returns a error what is the sense, asc
and ascq?

Re: different LUN numbers under the same dm device

[Brian Bunker]

Mike,

Here is what is returned by the device when it is available meaning that the LUN is available to the initiator. I am giving you the VPD page code 0x83 since I believe that is what multipath is using. If you want the output of something else let me know. It is trivial to get:

[root@vminirhl61 ~]# sg_inq /dev/sg0
standard INQUIRY:
  PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
  SCCS=1  ACC=0  TPGS=1  3PC=0  Protect=0  BQue=0
  EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=96 (0x60)   Peripheral device type: disk
 Vendor identification: PURE    
 Product identification: FlashArray      
 Product revision level: 100 
 Unit serial number: 436FC2124DA7883E00010003

[root@vminirhl61 ~]# sg_inq -p 0x83 /dev/sg0
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 20
    designator_type: NAA,  code_set: Binary
    associated with the addressed logical unit
      NAA 6, IEEE Company_id: 0x24a937
      Vendor Specific Identifier: 0x436fc212
      Vendor Specific Identifier Extension: 0x4da7883e00010003
      [0x624a9370436fc2124da7883e00010003]
  Designation descriptor number 2, descriptor length: 47
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: PURE    
      vendor specific: FlashArray:436FC2124DA7883E00010003
  Designation descriptor number 3, descriptor length: 8
    transport: Fibre Channel (FCP-2)
    designator_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x3
  Designation descriptor number 4, descriptor length: 8
    transport: Fibre Channel (FCP-2)
    designator_type: Target port group,  code_set: Binary
    associated with the target port
      Target port group: 0x0
  Designation descriptor number 5, descriptor length: 8
    designator_type: Logical unit group,  code_set: Binary
    associated with the addressed logical unit
      Logical unit group: 0x0
  Designation descriptor number 6, descriptor length: 36
    transport: Fibre Channel (FCP-2)
    designator_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      naa.21000024ff229d7a,t,0x0001

When I go and take away access to this LUN from the initiator the inquiry becomes:
[root@vminirhl61 ~]# sg_inq /dev/sg0
standard INQUIRY: [qualifier indicates no connected lu]
  PQual=1  Device_type=31  RMB=0  version=0x06  [SPC-4]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=0  BQue=0
  EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=96 (0x60)   Peripheral device type: no physical device on this lu
 Vendor identification: PURE    
 Product identification: FlashArray      
 Product revision level: 100

[root@vminirhl61 ~]# sg_inq -p 0x83 /dev/sg0
VPD INQUIRY: Device Identification page
  Designation descriptor number 1, descriptor length: 12
    designator_type: T10 vendor identification,  code_set: ASCII
    associated with the addressed logical unit
      vendor id: PURE    
  Designation descriptor number 2, descriptor length: 8
    transport: Fibre Channel (FCP-2)
    designator_type: Relative target port,  code_set: Binary
    associated with the target port
      Relative target port: 0x4
  Designation descriptor number 3, descriptor length: 36
    transport: Fibre Channel (FCP-2)
    designator_type: SCSI name string,  code_set: UTF-8
    associated with the target port
      SCSI name string:
      naa.21000024ff229d7a,t,0x0001

If you try to access this LUN when it doesn't have access permissions you will get the following SCSI error:

sd 3:0:0:2: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 3:0:0:2: [sdf] Sense Key : Illegal Request [current] 
sd 3:0:0:2: [sdf] Add. Sense: Logical unit not supported
sd 3:0:0:2: [sdf] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
end_request: I/O error, dev sdf, sector 0

As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with.

Hopefully that helps.

Thanks,
Brian

On Jun 8, 2012, at 12:52 AM, Mike Christie wrote:

> On 06/08/2012 01:59 AM, Hannes Reinecke wrote:
>> (Third mail on this topic .. I really should've read the entire
>> thread before answering. But there you go.)
>> 
>> On 06/08/2012 01:26 AM, Brian Bunker wrote:
>>> The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry.
>>> Then the LUN was unmasked from that initiator. That initiator is
>> holding on to those device
>>> names in multipath. If you query them when they are in the state
>> that I show in the
>>> multipath -ll result, they will not return an NAA number at all in
>> page code 0x83 or
>>> a serial number in page code 0x80. They will instead return a PQ
>> of 1 meaning that the
>>> LUN is capable of supporting a peripheral device but is not currently.
>>> 
>> Hehe.
>> 
>>> I understand about LUN's needing different NAA numbers and ours do, and we also have
>>> different LUN serial numbers for each LUN on the target. An
>> initiator doesn't always
>>> have to access to all LUN's that it once did. It is the re-use of
>> dm devices that
>>> seems to cause this result.
>>> 
>> No, rather a problem with the SCSI stack. multipath normally relies
>> on udev to keep track of any device events, like LUNs coming and
>> going. But this only works reliably if these events are triggered
>> via the underlying transport, like FC RSCN et al.
>> 
>> 'Real' scsi events which will get transmitted via Sense codes are
>> not evaluated further, sadly.
>> 
>> So short-term you are supposed to call 'rescan-scsi-bus.sh -r'
>> whenever you made any LUN assigment changes on the array.
>> Mid-term we already agreed on implementing some proper sense code
>> handling in the SCSI midlayer.
>> However, as usual in these cases, real life interfered and I've been
>> busy with other things.
>> But it's definitely on the agenda.
>> 
> 
> I think this is a bug in multipath though. Even though we are not
> handling sense that would indicate the path/device is not longer valid
> (PQ changed values) if it was sent, it would make sense that when
> multipath is assembling devices it would not create a device with
> different NAA/UUID values. So multipath should not make a device that
> includes sde, sdd, sdar and sdba but where sde and sdd have a different
> UUID than sdar and sdba, right?
> 
> Brian, what is the device returning in this case for sde and sdd? Does
> it return a error, or is it returning invalid data thinking the OS would
> check the PQ value first? If it returns a error what is the sense, asc
> and ascq?
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Brian Bunker
brian@purestorage.com

Re: different LUN numbers under the same dm device

[Mike Christie]

On 06/08/2012 11:35 AM, Brian Bunker wrote:
> As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with.
> 

This is different then what I thought you were talking about first. That
is the bug Hannes is talking about. As Hannes was saying we do not
handle that case, and in the scsi layer you get a message like:

"Warning! Received an indication that the "
"LUN assignments on this target have "
"changed. The Linux SCSI layer does not "
"automatically remap LUN assignments.\n");

logged in /var/log/messages.

This issue will definitely cause corruption if you are using the scsi
devices directly. For example if /dev/sda is LUN10, you then remap it,
and on the target it is now LUN11. The scsi layer just logs the message
above and the OS continues to use /dev/sda like it was LUN10, but the
data gets written to what is now LUN11's storage.

At the multipath layer if you do not rescan at the scsi layer level
after the remap, then there will be corruption because dm-0 is accessing
/dev/sda which it thinks is LUN10 but is now LUN 11. Because we do not
handle the sense indicating the LUNS changed, dm-0 will not change.

If you rescan at the scsi layer with just sysfs or procfs  and you get
new paths (/dev/sdXs) for the newly remapped LUNs, I was saying that I
think multipath should at least not add paths with different UUIDs to
the same dm deivce. So it should kick out the old paths before adding
the new ones or create a new device with the new paths, but it looks
like it doesn't.

So right now you should do what Hannes suggests and use
'rescan-scsi-bus.sh -r'. This will delete old mappings and setup
/dev/sdXs for new ones. multipath will get events and then should do the
right thing dm device assembly wise (I mean you should not end up with 4
paths where 2 have different UUIDs).

Re: different LUN numbers under the same dm device

[Brian Bunker]

Mike and all,

Thanks for the information. I think that everyone is on the same page now. The problem comes up for us because we have mostly automated tools processing this output and they choked when they saw 4 paths even though as Hannes pointed out 2 are faulty. We changed the automated scripts to look at the state as well so we will get past this. We were mostly curious why we only see this occasionally and on some dm devices. I will also change the rescanning mechanism to use 'rescan-scsi-bus.sh'. I think that we should use both -r and -i right so that we send a LIP to the FC target? I think that our current rescan behavior was just to go the /sys/class/fc_host/hostX and echo 1 to issue_lip. To answer all the questions, yes LUN 10 and LUN 12 did point to the same data LUN on the array not two differ
 ent ones. If we ever shared NAA numbers for different LUN's on the array itself we would have a big problem and would see corruption everywhere.

Thanks,
Brian

On Jun 8, 2012, at 10:05 AM, Mike Christie wrote:

> On 06/08/2012 11:35 AM, Brian Bunker wrote:
>> As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with.
>> 
> 
> This is different then what I thought you were talking about first. That
> is the bug Hannes is talking about. As Hannes was saying we do not
> handle that case, and in the scsi layer you get a message like:
> 
> "Warning! Received an indication that the "
> "LUN assignments on this target have "
> "changed. The Linux SCSI layer does not "
> "automatically remap LUN assignments.\n");
> 
> logged in /var/log/messages.
> 
> This issue will definitely cause corruption if you are using the scsi
> devices directly. For example if /dev/sda is LUN10, you then remap it,
> and on the target it is now LUN11. The scsi layer just logs the message
> above and the OS continues to use /dev/sda like it was LUN10, but the
> data gets written to what is now LUN11's storage.
> 
> At the multipath layer if you do not rescan at the scsi layer level
> after the remap, then there will be corruption because dm-0 is accessing
> /dev/sda which it thinks is LUN10 but is now LUN 11. Because we do not
> handle the sense indicating the LUNS changed, dm-0 will not change.
> 
> If you rescan at the scsi layer with just sysfs or procfs  and you get
> new paths (/dev/sdXs) for the newly remapped LUNs, I was saying that I
> think multipath should at least not add paths with different UUIDs to
> the same dm deivce. So it should kick out the old paths before adding
> the new ones or create a new device with the new paths, but it looks
> like it doesn't.
> 
> So right now you should do what Hannes suggests and use
> 'rescan-scsi-bus.sh -r'. This will delete old mappings and setup
> /dev/sdXs for new ones. multipath will get events and then should do the
> right thing dm device assembly wise (I mean you should not end up with 4
> paths where 2 have different UUIDs).

Brian Bunker
brian@purestorage.com

Re: different LUN numbers under the same dm device

[Hannes Reinecke]

On 06/08/2012 07:18 PM, Brian Bunker wrote:
> Mike and all,
> 
> Thanks for the information. I think that everyone is on the same page now.
> The problem comes up for us because we have mostly automated tools
processing
> this output and they choked when they saw 4 paths even though as
Hannes
> pointed out 2 are faulty. We changed the automated scripts to look
at the
> state as well so we will get past this. We were mostly curious why
we only
> see this occasionally and on some dm devices. I will also change the
> rescanning mechanism to use 'rescan-scsi-bus.sh'. I think that we
should
> use both -r and -i right so that we send a LIP to the FC target?
>
Gnaa. I should have never implemented the '-i' option in the first
place. '-i' is just a horrible hack for misbehaving SANs.
FC arrays should detect any change in the port mapping themselves,
either via RSCNs or if a LIP reset occured.
During normal operation LIP should _not_ be necessary; in fact, I
would consider it a bug if you do.

> I think that our current rescan behavior was just to go the 
> /sys/class/fc_host/hostX and echo 1 to issue_lip. To answer all the
> questions, yes LUN 10 and LUN 12 did point to the same data LUN on the
> array not two different ones. If we ever shared NAA numbers for
different
> LUN's on the array itself we would have a big problem and would see
> corruption everywhere.
> 
Close, but not banana. 'issue_lip' is in fact a nasty method of
invoking a scan, it basically amounts to a card reset.

The 'correct' way would be to do a

echo '- - -' > /sys/class/fc_host/hostX/scan

and then remove all devices for which sg_tur / sg_inq returns an
error code.
Which is what rescan-scsi-bus.sh -r does.

But occasionally you're indeed better off by coding it by hand.
I know rescan-scsi-bus.sh far too well to recommend it to each and
sundry :-)

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)