* different LUN numbers under the same dm device @ 2012-06-06 19:27 Brian Bunker 2012-06-06 20:35 ` Mike Snitzer 2012-06-08 6:29 ` Hannes Reinecke 0 siblings, 2 replies; 13+ messages in thread From: Brian Bunker @ 2012-06-06 19:27 UTC (permalink / raw) To: dm-devel Our company produces a multiple port Fibre Channel storage array. We are continually plagued by this problem. We get a dm device which combines paths for different LUN's. We would like to understand why this is happening. Wouldn't this problem almost certainly lead to a data corruption? Thanks, Brian 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray size=500G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:12 sde 8:64 failed faulty running |- 0:0:0:12 sdd 8:48 failed faulty running |- 0:0:0:10 sdar 66:176 active ready running `- 1:0:0:10 sdba 67:64 active ready running Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the other two are for LUN 12. We have 24 other dm devices which have only the expected 2 paths. Brian Bunker brian@purestorage.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-06 19:27 different LUN numbers under the same dm device Brian Bunker @ 2012-06-06 20:35 ` Mike Snitzer 2012-06-06 20:59 ` Brian Bunker 2012-06-08 6:29 ` Hannes Reinecke 1 sibling, 1 reply; 13+ messages in thread From: Mike Snitzer @ 2012-06-06 20:35 UTC (permalink / raw) To: Brian Bunker; +Cc: device-mapper development On Wed, Jun 06 2012 at 3:27pm -0400, Brian Bunker <brian@purestorage.com> wrote: > Our company produces a multiple port Fibre Channel storage array. We > are continually plagued by this problem. We get a dm device which > combines paths for different LUN's. We would like to understand why > this is happening. Wouldn't this problem almost certainly lead to a > data corruption? > > Thanks, > Brian > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 1:0:0:12 sde 8:64 failed faulty running > |- 0:0:0:12 sdd 8:48 failed faulty running > |- 0:0:0:10 sdar 66:176 active ready running > `- 1:0:0:10 sdba 67:64 active ready running > > Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the > other two are for LUN 12. We have 24 other dm devices which have only > the expected 2 paths. Multipath considers all LUNs with the same UUID to be the same LUN. So you should first try to understand why all of these paths were held to have the same UUID (3624a93700a14254d729923840001000b). ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-06 20:35 ` Mike Snitzer @ 2012-06-06 20:59 ` Brian Bunker 2012-06-07 22:39 ` Benjamin Marzinski 2012-06-08 6:37 ` Hannes Reinecke 0 siblings, 2 replies; 13+ messages in thread From: Brian Bunker @ 2012-06-06 20:59 UTC (permalink / raw) To: Mike Snitzer; +Cc: device-mapper development Mike, The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this: [root@r13init32 ~]# sg_inq /dev/sde standard INQUIRY: [qualifier indicates no connected LU] PQual=1 Device_type=31 RMB=0 version=0x06 [SPC-4] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 BQue=0 EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 [SPI: Clocking=0x0 QAS=0 IUS=0] length=96 (0x60) Peripheral device type: no physical device on this lu Vendor identification: PURE Product identification: FlashArray Product revision level: 100 There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty: 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray size=500G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 1:0:0:12 sde 8:64 failed faulty running |- 0:0:0:12 sdd 8:48 failed faulty running Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect: 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray size=500G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 0:0:0:10 sdar 66:176 active ready running !- 1:0:0:10 sdba 67:64 active ready running Thanks, Brian On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote: > On Wed, Jun 06 2012 at 3:27pm -0400, > Brian Bunker <brian@purestorage.com> wrote: > >> Our company produces a multiple port Fibre Channel storage array. We >> are continually plagued by this problem. We get a dm device which >> combines paths for different LUN's. We would like to understand why >> this is happening. Wouldn't this problem almost certainly lead to a >> data corruption? >> >> Thanks, >> Brian >> >> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray >> size=500G features='0' hwhandler='0' wp=rw >> `-+- policy='round-robin 0' prio=1 status=active >> |- 1:0:0:12 sde 8:64 failed faulty running >> |- 0:0:0:12 sdd 8:48 failed faulty running >> |- 0:0:0:10 sdar 66:176 active ready running >> `- 1:0:0:10 sdba 67:64 active ready running >> >> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the >> other two are for LUN 12. We have 24 other dm devices which have only >> the expected 2 paths. > > Multipath considers all LUNs with the same UUID to be the same LUN. > > So you should first try to understand why all of these paths were held > to have the same UUID (3624a93700a14254d729923840001000b). Brian Bunker brian@purestorage.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-06 20:59 ` Brian Bunker @ 2012-06-07 22:39 ` Benjamin Marzinski 2012-06-07 23:26 ` Brian Bunker 2012-06-08 6:37 ` Hannes Reinecke 1 sibling, 1 reply; 13+ messages in thread From: Benjamin Marzinski @ 2012-06-07 22:39 UTC (permalink / raw) To: device-mapper development On Wed, Jun 06, 2012 at 01:59:02PM -0700, Brian Bunker wrote: > Mike, > > The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this: > > [root@r13init32 ~]# sg_inq /dev/sde > standard INQUIRY: [qualifier indicates no connected LU] > PQual=1 Device_type=31 RMB=0 version=0x06 [SPC-4] > [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 > SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 BQue=0 > EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 > [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 > [SPI: Clocking=0x0 QAS=0 IUS=0] > length=96 (0x60) Peripheral device type: no physical device on this lu > Vendor identification: PURE > Product identification: FlashArray > Product revision level: 100 > > There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. > > The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty: > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 1:0:0:12 sde 8:64 failed faulty running > |- 0:0:0:12 sdd 8:48 failed faulty running > > Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect: > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 0:0:0:10 sdar 66:176 active ready running > !- 1:0:0:10 sdba 67:64 active ready running > > Thanks, > Brian So, did sde and sdd (the paths for the multipath device for LUN 12) originally have the wwid of 3624a93700a14254d729923840001000b. Do sdar and sdba actually have the wwid of 3624a93700a14254d729923840001000b. You can check this by running # scsi_id --whitelisted --device=<devname> for example # scsi_id --whitelisted --device=/dev/sde If all of these scsi devices return the same value, then multipath has no way of knowing that they don't belong together. If not, I'd like to know which devices don't really have that UUID. -Ben > > On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote: > > > On Wed, Jun 06 2012 at 3:27pm -0400, > > Brian Bunker <brian@purestorage.com> wrote: > > > >> Our company produces a multiple port Fibre Channel storage array. We > >> are continually plagued by this problem. We get a dm device which > >> combines paths for different LUN's. We would like to understand why > >> this is happening. Wouldn't this problem almost certainly lead to a > >> data corruption? > >> > >> Thanks, > >> Brian > >> > >> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > >> size=500G features='0' hwhandler='0' wp=rw > >> `-+- policy='round-robin 0' prio=1 status=active > >> |- 1:0:0:12 sde 8:64 failed faulty running > >> |- 0:0:0:12 sdd 8:48 failed faulty running > >> |- 0:0:0:10 sdar 66:176 active ready running > >> `- 1:0:0:10 sdba 67:64 active ready running > >> > >> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the > >> other two are for LUN 12. We have 24 other dm devices which have only > >> the expected 2 paths. > > > > Multipath considers all LUNs with the same UUID to be the same LUN. > > > > So you should first try to understand why all of these paths were held > > to have the same UUID (3624a93700a14254d729923840001000b). > > Brian Bunker > brian@purestorage.com > > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-07 22:39 ` Benjamin Marzinski @ 2012-06-07 23:26 ` Brian Bunker 2012-06-08 6:59 ` Hannes Reinecke 0 siblings, 1 reply; 13+ messages in thread From: Brian Bunker @ 2012-06-07 23:26 UTC (permalink / raw) To: device-mapper development The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry. Then the LUN was unmasked from that initiator. That initiator is holding on to those device names in multipath. If you query them when they are in the state that I show in the multipath -ll result, they will not return an NAA number at all in page code 0x83 or a serial number in page code 0x80. They will instead return a PQ of 1 meaning that the LUN is capable of supporting a peripheral device but is not currently. I understand about LUN's needing different NAA numbers and ours do, and we also have different LUN serial numbers for each LUN on the target. An initiator doesn't alway have to access to all LUN's that it once did. It is the re-use of dm devices that seems to cause this result. Thanks, Brian On Jun 7, 2012, at 3:39 PM, Benjamin Marzinski wrote: > On Wed, Jun 06, 2012 at 01:59:02PM -0700, Brian Bunker wrote: >> Mike, >> >> The devices for LUN 12 are failed and correspond to LUN's not currently shared to the initiator at all. They were at one point and were likely used by dm-11 for its underlying paths. The inquiry data of those LUN's when the problem happened was like this: >> >> [root@r13init32 ~]# sg_inq /dev/sde >> standard INQUIRY: [qualifier indicates no connected LU] >> PQual=1 Device_type=31 RMB=0 version=0x06 [SPC-4] >> [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 >> SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 BQue=0 >> EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 >> [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 >> [SPI: Clocking=0x0 QAS=0 IUS=0] >> length=96 (0x60) Peripheral device type: no physical device on this lu >> Vendor identification: PURE >> Product identification: FlashArray >> Product revision level: 100 >> >> There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 since there is no LUN 12 attached as a disk device at the time multipath -ll was run. Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. >> >> The sequence is something like share a LUN from the array with two paths to the initiator, a dm device gets created presumably like this at first (except that the status would be active and ready and not failed and faulty: >> >> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray >> size=500G features='0' hwhandler='0' wp=rw >> `-+- policy='round-robin 0' prio=1 status=active >> |- 1:0:0:12 sde 8:64 failed faulty running >> |- 0:0:0:12 sdd 8:48 failed faulty running >> >> Then that LUN 12 is taken away from the initiator and the dm device dm-11 is reused later by LUN 10 when it is shared to the initiator, but the LUN 12 devices still remain as part of the dm device. Then I would expect: >> >> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray >> size=500G features='0' hwhandler='0' wp=rw >> `-+- policy='round-robin 0' prio=1 status=active >> |- 0:0:0:10 sdar 66:176 active ready running >> !- 1:0:0:10 sdba 67:64 active ready running >> >> Thanks, >> Brian > > So, did sde and sdd (the paths for the multipath device for LUN 12) > originally have the wwid of 3624a93700a14254d729923840001000b. Do > sdar and sdba actually have the wwid of > 3624a93700a14254d729923840001000b. You can check this by running > > # scsi_id --whitelisted --device=<devname> > > for example > > # scsi_id --whitelisted --device=/dev/sde > > If all of these scsi devices return the same value, then multipath > has no way of knowing that they don't belong together. If not, I'd > like to know which devices don't really have that UUID. > > -Ben > >> >> On Jun 6, 2012, at 1:35 PM, Mike Snitzer wrote: >> >>> On Wed, Jun 06 2012 at 3:27pm -0400, >>> Brian Bunker <brian@purestorage.com> wrote: >>> >>>> Our company produces a multiple port Fibre Channel storage array. We >>>> are continually plagued by this problem. We get a dm device which >>>> combines paths for different LUN's. We would like to understand why >>>> this is happening. Wouldn't this problem almost certainly lead to a >>>> data corruption? >>>> >>>> Thanks, >>>> Brian >>>> >>>> 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray >>>> size=500G features='0' hwhandler='0' wp=rw >>>> `-+- policy='round-robin 0' prio=1 status=active >>>> |- 1:0:0:12 sde 8:64 failed faulty running >>>> |- 0:0:0:12 sdd 8:48 failed faulty running >>>> |- 0:0:0:10 sdar 66:176 active ready running >>>> `- 1:0:0:10 sdba 67:64 active ready running >>>> >>>> Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the >>>> other two are for LUN 12. We have 24 other dm devices which have only >>>> the expected 2 paths. >>> >>> Multipath considers all LUNs with the same UUID to be the same LUN. >>> >>> So you should first try to understand why all of these paths were held >>> to have the same UUID (3624a93700a14254d729923840001000b). >> >> Brian Bunker >> brian@purestorage.com >> >> >> >> >> -- >> dm-devel mailing list >> dm-devel@redhat.com >> https://www.redhat.com/mailman/listinfo/dm-devel > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel Brian Bunker brian@purestorage.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-07 23:26 ` Brian Bunker @ 2012-06-08 6:59 ` Hannes Reinecke 2012-06-08 7:52 ` Mike Christie 0 siblings, 1 reply; 13+ messages in thread From: Hannes Reinecke @ 2012-06-08 6:59 UTC (permalink / raw) To: dm-devel (Third mail on this topic .. I really should've read the entire thread before answering. But there you go.) On 06/08/2012 01:26 AM, Brian Bunker wrote: > The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry. > Then the LUN was unmasked from that initiator. That initiator is holding on to those device > names in multipath. If you query them when they are in the state that I show in the > multipath -ll result, they will not return an NAA number at all in page code 0x83 or > a serial number in page code 0x80. They will instead return a PQ of 1 meaning that the > LUN is capable of supporting a peripheral device but is not currently. > Hehe. > I understand about LUN's needing different NAA numbers and ours do, and we also have > different LUN serial numbers for each LUN on the target. An initiator doesn't always > have to access to all LUN's that it once did. It is the re-use of dm devices that > seems to cause this result. > No, rather a problem with the SCSI stack. multipath normally relies on udev to keep track of any device events, like LUNs coming and going. But this only works reliably if these events are triggered via the underlying transport, like FC RSCN et al. 'Real' scsi events which will get transmitted via Sense codes are not evaluated further, sadly. So short-term you are supposed to call 'rescan-scsi-bus.sh -r' whenever you made any LUN assigment changes on the array. Mid-term we already agreed on implementing some proper sense code handling in the SCSI midlayer. However, as usual in these cases, real life interfered and I've been busy with other things. But it's definitely on the agenda. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-08 6:59 ` Hannes Reinecke @ 2012-06-08 7:52 ` Mike Christie 2012-06-08 16:35 ` Brian Bunker 0 siblings, 1 reply; 13+ messages in thread From: Mike Christie @ 2012-06-08 7:52 UTC (permalink / raw) To: device-mapper development On 06/08/2012 01:59 AM, Hannes Reinecke wrote: > (Third mail on this topic .. I really should've read the entire > thread before answering. But there you go.) > > On 06/08/2012 01:26 AM, Brian Bunker wrote: >> The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry. >> Then the LUN was unmasked from that initiator. That initiator is > holding on to those device >> names in multipath. If you query them when they are in the state > that I show in the >> multipath -ll result, they will not return an NAA number at all in > page code 0x83 or >> a serial number in page code 0x80. They will instead return a PQ > of 1 meaning that the >> LUN is capable of supporting a peripheral device but is not currently. >> > Hehe. > >> I understand about LUN's needing different NAA numbers and ours do, and we also have >> different LUN serial numbers for each LUN on the target. An > initiator doesn't always >> have to access to all LUN's that it once did. It is the re-use of > dm devices that >> seems to cause this result. >> > No, rather a problem with the SCSI stack. multipath normally relies > on udev to keep track of any device events, like LUNs coming and > going. But this only works reliably if these events are triggered > via the underlying transport, like FC RSCN et al. > > 'Real' scsi events which will get transmitted via Sense codes are > not evaluated further, sadly. > > So short-term you are supposed to call 'rescan-scsi-bus.sh -r' > whenever you made any LUN assigment changes on the array. > Mid-term we already agreed on implementing some proper sense code > handling in the SCSI midlayer. > However, as usual in these cases, real life interfered and I've been > busy with other things. > But it's definitely on the agenda. > I think this is a bug in multipath though. Even though we are not handling sense that would indicate the path/device is not longer valid (PQ changed values) if it was sent, it would make sense that when multipath is assembling devices it would not create a device with different NAA/UUID values. So multipath should not make a device that includes sde, sdd, sdar and sdba but where sde and sdd have a different UUID than sdar and sdba, right? Brian, what is the device returning in this case for sde and sdd? Does it return a error, or is it returning invalid data thinking the OS would check the PQ value first? If it returns a error what is the sense, asc and ascq? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-08 7:52 ` Mike Christie @ 2012-06-08 16:35 ` Brian Bunker 2012-06-08 17:05 ` Mike Christie 0 siblings, 1 reply; 13+ messages in thread From: Brian Bunker @ 2012-06-08 16:35 UTC (permalink / raw) To: device-mapper development Mike, Here is what is returned by the device when it is available meaning that the LUN is available to the initiator. I am giving you the VPD page code 0x83 since I believe that is what multipath is using. If you want the output of something else let me know. It is trivial to get: [root@vminirhl61 ~]# sg_inq /dev/sg0 standard INQUIRY: PQual=0 Device_type=0 RMB=0 version=0x06 [SPC-4] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 SCCS=1 ACC=0 TPGS=1 3PC=0 Protect=0 BQue=0 EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 [SPI: Clocking=0x0 QAS=0 IUS=0] length=96 (0x60) Peripheral device type: disk Vendor identification: PURE Product identification: FlashArray Product revision level: 100 Unit serial number: 436FC2124DA7883E00010003 [root@vminirhl61 ~]# sg_inq -p 0x83 /dev/sg0 VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 designator_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x24a937 Vendor Specific Identifier: 0x436fc212 Vendor Specific Identifier Extension: 0x4da7883e00010003 [0x624a9370436fc2124da7883e00010003] Designation descriptor number 2, descriptor length: 47 designator_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: PURE vendor specific: FlashArray:436FC2124DA7883E00010003 Designation descriptor number 3, descriptor length: 8 transport: Fibre Channel (FCP-2) designator_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x3 Designation descriptor number 4, descriptor length: 8 transport: Fibre Channel (FCP-2) designator_type: Target port group, code_set: Binary associated with the target port Target port group: 0x0 Designation descriptor number 5, descriptor length: 8 designator_type: Logical unit group, code_set: Binary associated with the addressed logical unit Logical unit group: 0x0 Designation descriptor number 6, descriptor length: 36 transport: Fibre Channel (FCP-2) designator_type: SCSI name string, code_set: UTF-8 associated with the target port SCSI name string: naa.21000024ff229d7a,t,0x0001 When I go and take away access to this LUN from the initiator the inquiry becomes: [root@vminirhl61 ~]# sg_inq /dev/sg0 standard INQUIRY: [qualifier indicates no connected lu] PQual=1 Device_type=31 RMB=0 version=0x06 [SPC-4] [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 BQue=0 EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 [SPI: Clocking=0x0 QAS=0 IUS=0] length=96 (0x60) Peripheral device type: no physical device on this lu Vendor identification: PURE Product identification: FlashArray Product revision level: 100 [root@vminirhl61 ~]# sg_inq -p 0x83 /dev/sg0 VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 12 designator_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: PURE Designation descriptor number 2, descriptor length: 8 transport: Fibre Channel (FCP-2) designator_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x4 Designation descriptor number 3, descriptor length: 36 transport: Fibre Channel (FCP-2) designator_type: SCSI name string, code_set: UTF-8 associated with the target port SCSI name string: naa.21000024ff229d7a,t,0x0001 If you try to access this LUN when it doesn't have access permissions you will get the following SCSI error: sd 3:0:0:2: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE sd 3:0:0:2: [sdf] Sense Key : Illegal Request [current] sd 3:0:0:2: [sdf] Add. Sense: Logical unit not supported sd 3:0:0:2: [sdf] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00 end_request: I/O error, dev sdf, sector 0 As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with. Hopefully that helps. Thanks, Brian On Jun 8, 2012, at 12:52 AM, Mike Christie wrote: > On 06/08/2012 01:59 AM, Hannes Reinecke wrote: >> (Third mail on this topic .. I really should've read the entire >> thread before answering. But there you go.) >> >> On 06/08/2012 01:26 AM, Brian Bunker wrote: >>> The answer is yes they did have that LUN NAA value that comes from page code 0x83 in inquiry. >>> Then the LUN was unmasked from that initiator. That initiator is >> holding on to those device >>> names in multipath. If you query them when they are in the state >> that I show in the >>> multipath -ll result, they will not return an NAA number at all in >> page code 0x83 or >>> a serial number in page code 0x80. They will instead return a PQ >> of 1 meaning that the >>> LUN is capable of supporting a peripheral device but is not currently. >>> >> Hehe. >> >>> I understand about LUN's needing different NAA numbers and ours do, and we also have >>> different LUN serial numbers for each LUN on the target. An >> initiator doesn't always >>> have to access to all LUN's that it once did. It is the re-use of >> dm devices that >>> seems to cause this result. >>> >> No, rather a problem with the SCSI stack. multipath normally relies >> on udev to keep track of any device events, like LUNs coming and >> going. But this only works reliably if these events are triggered >> via the underlying transport, like FC RSCN et al. >> >> 'Real' scsi events which will get transmitted via Sense codes are >> not evaluated further, sadly. >> >> So short-term you are supposed to call 'rescan-scsi-bus.sh -r' >> whenever you made any LUN assigment changes on the array. >> Mid-term we already agreed on implementing some proper sense code >> handling in the SCSI midlayer. >> However, as usual in these cases, real life interfered and I've been >> busy with other things. >> But it's definitely on the agenda. >> > > I think this is a bug in multipath though. Even though we are not > handling sense that would indicate the path/device is not longer valid > (PQ changed values) if it was sent, it would make sense that when > multipath is assembling devices it would not create a device with > different NAA/UUID values. So multipath should not make a device that > includes sde, sdd, sdar and sdba but where sde and sdd have a different > UUID than sdar and sdba, right? > > Brian, what is the device returning in this case for sde and sdd? Does > it return a error, or is it returning invalid data thinking the OS would > check the PQ value first? If it returns a error what is the sense, asc > and ascq? > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel Brian Bunker brian@purestorage.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-08 16:35 ` Brian Bunker @ 2012-06-08 17:05 ` Mike Christie 2012-06-08 17:18 ` Brian Bunker 0 siblings, 1 reply; 13+ messages in thread From: Mike Christie @ 2012-06-08 17:05 UTC (permalink / raw) To: device-mapper development; +Cc: Brian Bunker On 06/08/2012 11:35 AM, Brian Bunker wrote: > As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with. > This is different then what I thought you were talking about first. That is the bug Hannes is talking about. As Hannes was saying we do not handle that case, and in the scsi layer you get a message like: "Warning! Received an indication that the " "LUN assignments on this target have " "changed. The Linux SCSI layer does not " "automatically remap LUN assignments.\n"); logged in /var/log/messages. This issue will definitely cause corruption if you are using the scsi devices directly. For example if /dev/sda is LUN10, you then remap it, and on the target it is now LUN11. The scsi layer just logs the message above and the OS continues to use /dev/sda like it was LUN10, but the data gets written to what is now LUN11's storage. At the multipath layer if you do not rescan at the scsi layer level after the remap, then there will be corruption because dm-0 is accessing /dev/sda which it thinks is LUN10 but is now LUN 11. Because we do not handle the sense indicating the LUNS changed, dm-0 will not change. If you rescan at the scsi layer with just sysfs or procfs and you get new paths (/dev/sdXs) for the newly remapped LUNs, I was saying that I think multipath should at least not add paths with different UUIDs to the same dm deivce. So it should kick out the old paths before adding the new ones or create a new device with the new paths, but it looks like it doesn't. So right now you should do what Hannes suggests and use 'rescan-scsi-bus.sh -r'. This will delete old mappings and setup /dev/sdXs for new ones. multipath will get events and then should do the right thing dm device assembly wise (I mean you should not end up with 4 paths where 2 have different UUIDs). ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-08 17:05 ` Mike Christie @ 2012-06-08 17:18 ` Brian Bunker 2012-06-11 6:38 ` Hannes Reinecke 0 siblings, 1 reply; 13+ messages in thread From: Brian Bunker @ 2012-06-08 17:18 UTC (permalink / raw) To: Mike Christie; +Cc: device-mapper development Mike and all, Thanks for the information. I think that everyone is on the same page now. The problem comes up for us because we have mostly automated tools processing this output and they choked when they saw 4 paths even though as Hannes pointed out 2 are faulty. We changed the automated scripts to look at the state as well so we will get past this. We were mostly curious why we only see this occasionally and on some dm devices. I will also change the rescanning mechanism to use 'rescan-scsi-bus.sh'. I think that we should use both -r and -i right so that we send a LIP to the FC target? I think that our current rescan behavior was just to go the /sys/class/fc_host/hostX and echo 1 to issue_lip. To answer all the questions, yes LUN 10 and LUN 12 did point to the same data LUN on the array not two differ ent ones. If we ever shared NAA numbers for different LUN's on the array itself we would have a big problem and would see corruption everywhere. Thanks, Brian On Jun 8, 2012, at 10:05 AM, Mike Christie wrote: > On 06/08/2012 11:35 AM, Brian Bunker wrote: >> As far as the NAA number for different LUN's, we have the requirement that each LUN on our array has a different NAA number as you would expect. How those LUN's are presented to the host however there is no such restriction. For example, you could present the array's LUN 1 masked as LUN 10 to the initiator and then later masked that same LUN to the initiator as LUN 12 or any other number. In that case LUN 10 and later LUN 12 would have the same NAA number since they are in fact talking to the same actual LUN on the array. This is what I think multipath is having trouble with. >> > > This is different then what I thought you were talking about first. That > is the bug Hannes is talking about. As Hannes was saying we do not > handle that case, and in the scsi layer you get a message like: > > "Warning! Received an indication that the " > "LUN assignments on this target have " > "changed. The Linux SCSI layer does not " > "automatically remap LUN assignments.\n"); > > logged in /var/log/messages. > > This issue will definitely cause corruption if you are using the scsi > devices directly. For example if /dev/sda is LUN10, you then remap it, > and on the target it is now LUN11. The scsi layer just logs the message > above and the OS continues to use /dev/sda like it was LUN10, but the > data gets written to what is now LUN11's storage. > > At the multipath layer if you do not rescan at the scsi layer level > after the remap, then there will be corruption because dm-0 is accessing > /dev/sda which it thinks is LUN10 but is now LUN 11. Because we do not > handle the sense indicating the LUNS changed, dm-0 will not change. > > If you rescan at the scsi layer with just sysfs or procfs and you get > new paths (/dev/sdXs) for the newly remapped LUNs, I was saying that I > think multipath should at least not add paths with different UUIDs to > the same dm deivce. So it should kick out the old paths before adding > the new ones or create a new device with the new paths, but it looks > like it doesn't. > > So right now you should do what Hannes suggests and use > 'rescan-scsi-bus.sh -r'. This will delete old mappings and setup > /dev/sdXs for new ones. multipath will get events and then should do the > right thing dm device assembly wise (I mean you should not end up with 4 > paths where 2 have different UUIDs). Brian Bunker brian@purestorage.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-08 17:18 ` Brian Bunker @ 2012-06-11 6:38 ` Hannes Reinecke 0 siblings, 0 replies; 13+ messages in thread From: Hannes Reinecke @ 2012-06-11 6:38 UTC (permalink / raw) To: Brian Bunker; +Cc: dm-devel On 06/08/2012 07:18 PM, Brian Bunker wrote: > Mike and all, > > Thanks for the information. I think that everyone is on the same page now. > The problem comes up for us because we have mostly automated tools processing > this output and they choked when they saw 4 paths even though as Hannes > pointed out 2 are faulty. We changed the automated scripts to look at the > state as well so we will get past this. We were mostly curious why we only > see this occasionally and on some dm devices. I will also change the > rescanning mechanism to use 'rescan-scsi-bus.sh'. I think that we should > use both -r and -i right so that we send a LIP to the FC target? > Gnaa. I should have never implemented the '-i' option in the first place. '-i' is just a horrible hack for misbehaving SANs. FC arrays should detect any change in the port mapping themselves, either via RSCNs or if a LIP reset occured. During normal operation LIP should _not_ be necessary; in fact, I would consider it a bug if you do. > I think that our current rescan behavior was just to go the > /sys/class/fc_host/hostX and echo 1 to issue_lip. To answer all the > questions, yes LUN 10 and LUN 12 did point to the same data LUN on the > array not two different ones. If we ever shared NAA numbers for different > LUN's on the array itself we would have a big problem and would see > corruption everywhere. > Close, but not banana. 'issue_lip' is in fact a nasty method of invoking a scan, it basically amounts to a card reset. The 'correct' way would be to do a echo '- - -' > /sys/class/fc_host/hostX/scan and then remove all devices for which sg_tur / sg_inq returns an error code. Which is what rescan-scsi-bus.sh -r does. But occasionally you're indeed better off by coding it by hand. I know rescan-scsi-bus.sh far too well to recommend it to each and sundry :-) Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-06 20:59 ` Brian Bunker 2012-06-07 22:39 ` Benjamin Marzinski @ 2012-06-08 6:37 ` Hannes Reinecke 1 sibling, 0 replies; 13+ messages in thread From: Hannes Reinecke @ 2012-06-08 6:37 UTC (permalink / raw) To: device-mapper development; +Cc: Brian Bunker, Mike Snitzer On 06/06/2012 10:59 PM, Brian Bunker wrote: > Mike, > > The devices for LUN 12 are failed and correspond to LUN's not currently shared > to the initiator at all. They were at one point and were likely used by dm-11 > for its underlying paths. The inquiry data of those LUN's when the problem happened was like this: > > [root@r13init32 ~]# sg_inq /dev/sde > standard INQUIRY: [qualifier indicates no connected LU] > PQual=1 Device_type=31 RMB=0 version=0x06 [SPC-4] > [AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2 > SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=0 BQue=0 > EncServ=0 MultiP=1 (VS=0) [MChngr=0] [ACKREQQ=0] Addr16=0 > [RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1 > [SPI: Clocking=0x0 QAS=0 IUS=0] > length=96 (0x60) Peripheral device type: no physical device on this lu > Vendor identification: PURE > Product identification: FlashArray > Product revision level: 100 > > There is no NAA number, page code 0x83 or LUN serial number available, page code 0x80 > since there is no LUN 12 attached as a disk device at the time multipath -ll was run. > Different LUN's from our array would ever have the same NAA value, what I think you are calling UUID. > Yep. Hmm. So the devices are unmapped from the storage, but still visible from the initiator? Have you run 'rescan-scsi-bus.sh -r' here? That should clean up these devices. > The sequence is something like share a LUN from the array with two paths to > the initiator, a dm device gets created presumably like this at first (except > that the status would be active and ready and not failed and faulty: > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 1:0:0:12 sde 8:64 failed faulty running > |- 0:0:0:12 sdd 8:48 failed faulty running > > Then that LUN 12 is taken away from the initiator and the dm device dm-11 is > reused later by LUN 10 when it is shared to the initiator, but the LUN 12 > devices still remain as part of the dm device. Then I would expect: > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 0:0:0:10 sdar 66:176 active ready running > !- 1:0:0:10 sdba 67:64 active ready running > Yeah, but still: it means that at one point LUN 12 had the same NAA value than LUN 10, correct? It _might_ happen that multipath created a dm-device for LUN12, set them to 'faulty' during unsharing, and then added the then-new LUN10 to the same device, given that the NAA number is identical. So the point still stands: LUN10 must have had the same NAA value than LUN12 now has. So unless the original LUN10 referred to the same storage entity as LUN12 now does, this is a definite no-no. And if it does, we're pretty much in the clear, as then LUN10 would now refer to a stale device (with status 'failed faulty'), and should be cleared up with 'rescan-scsi-bus.sh -r'. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: different LUN numbers under the same dm device 2012-06-06 19:27 different LUN numbers under the same dm device Brian Bunker 2012-06-06 20:35 ` Mike Snitzer @ 2012-06-08 6:29 ` Hannes Reinecke 1 sibling, 0 replies; 13+ messages in thread From: Hannes Reinecke @ 2012-06-08 6:29 UTC (permalink / raw) To: Brian Bunker; +Cc: dm-devel On 06/06/2012 09:27 PM, Brian Bunker wrote: > Our company produces a multiple port Fibre Channel storage array. We are continually plagued > by this problem. We get a dm device which combines paths for different LUN's. We would like > to understand why this is happening. Wouldn't this problem almost certainly lead to a data > corruption? > Most likely; given that the two LUNs indeed refer to different disks. > Thanks, > Brian > > 3624a93700a14254d729923840001000b dm-11 PURE,FlashArray > size=500G features='0' hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=1 status=active > |- 1:0:0:12 sde 8:64 failed faulty running > |- 0:0:0:12 sdd 8:48 failed faulty running > |- 0:0:0:10 sdar 66:176 active ready running > `- 1:0:0:10 sdba 67:64 active ready running > > Of the 4 paths to dm-11, we can see two paths are for LUN 10 and the other two are for LUN 12. > We have 24 other dm devices which have only the expected 2 paths. > Check the output of 'sg_vpd -p di /dev/sd(e|d|ar|ba)'. 'sde' and 'sdd' should be identical (well, the bits for 'Addressed logical unit' should be), as should 'sdar' and 'sdab'. 'sde' and 'sdar', on the other hand, should _not_. Given the above, however, I strongly suspect they are. Looks like your LUN WWID generation code is buggered. But I guess you can always blame Nic Bellinger :-) Greetings to Joern Engel. Cheers, Hannes -- Dr. Hannes Reinecke zSeries & Storage hare@suse.de +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg) ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-06-11 6:38 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-06-06 19:27 different LUN numbers under the same dm device Brian Bunker 2012-06-06 20:35 ` Mike Snitzer 2012-06-06 20:59 ` Brian Bunker 2012-06-07 22:39 ` Benjamin Marzinski 2012-06-07 23:26 ` Brian Bunker 2012-06-08 6:59 ` Hannes Reinecke 2012-06-08 7:52 ` Mike Christie 2012-06-08 16:35 ` Brian Bunker 2012-06-08 17:05 ` Mike Christie 2012-06-08 17:18 ` Brian Bunker 2012-06-11 6:38 ` Hannes Reinecke 2012-06-08 6:37 ` Hannes Reinecke 2012-06-08 6:29 ` Hannes Reinecke
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.