The PQ=1 saga

The PQ=1 saga

[Martin K. Petersen]

I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
PDT=other values to SCSI_SCAN_TARGET_PRESENT").

I have been spending quite a bit of time digging through old SCSI and
controller specs. As far as I can tell the original Linux behavior was
correct. Recent SPC specs are very abstract in this department resulting
in unfortunate ambiguity. But originally PQ=1 meant "LUN supports this
peripheral device type but no physical device is currently connected".

Based on this original definition, PQ=1 has been widely used throughout
the industry as a means to avoid associating an ULD driver with a
device. The LUN is accessible (primary commands, etc.) but no media is
present (no physical device connected).

Our original algorithm, which I would like to reinstate, is essentially
the following (in slightly unrolled form):

        if (PQ == 3)
            /* Don't expose device */
        else if ((PQ == 1 || sdev->pdt_1f_for_no_lun) && PDT == 0x1f)
            /* Don't expose device */
        else if (PQ == 1) {
            /* Expose device, don't bind ULD */
        } else /* PQ == 0 */
            /* Expose device, bind ULD if PDT is supported */

I would like to understand why -- in the case of the IBM 2145 --
exposing the sg device caused problems. Li: Can you shed some light on
the problems caused by 2145 LUNs reporting PQ=1?

Thanks!

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: The PQ=1 saga

[Brian Bunker]

For a completely separate reason I would like to see PQ=1 expose the sd device.

ALUA state transitions from unavailable back to another state does not
work depending on what state devices are in when they are initially discovered.
In the ALUA unavailable state the peripheral qualifier of the device should also
be set to 001b.

If the device is not in the unavailable state when it is initially discovered
(PQ not 001b), it can move to and from the unavailable state with the expected
result like this:

/dev/sg1  7 0 1 1  0  /dev/sdb  PURE      FlashArray        8888

root@init106-1 class]# sg_inq /dev/sdb
standard INQUIRY: [qualifier indicates no connected LU]
PQual=1  Device_type=0  RMB=0  version=0x06  [SPC-4]
[AERC=0]  [TrmTsk=0]  NormACA=1  HiSUP=1  Resp_data_format=2
SCCS=1  ACC=0  TPGS=1  3PC=1  Protect=0  [BQue=0]
EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
[SPI: Clocking=0x0  QAS=0  IUS=0]
length=96 (0x60)   Peripheral device type: disk
Vendor identification: PURE
Product identification: FlashArray
Product revision level: 8888
Unit serial number: 1D6DB146171D4E32000113E6

If, however, the device is already in this state when it is initially
discovered, no sd device is created like this:

/dev/sg2  7 0 1 1  0  PURE      FlashArray        8888

Since no sd device is ever created, when the ALUA state changes and the
peripheral qualifier is set back to 0, manual intervention is required.
The devices peripheral qualifier is correct after another rescan,
but no sd device is created.

[root@init106-1 ~]# sg_inq /dev/sg2
standard INQUIRY:
PQual=0  Device_type=0  RMB=0  version=0x06  [SPC-4]
[AERC=0]  [TrmTsk=0]  NormACA=1  HiSUP=1  Resp_data_format=2
SCCS=1  ACC=0  TPGS=1  3PC=1  Protect=0  [BQue=0]
EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
[RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
[SPI: Clocking=0x0  QAS=0  IUS=0]
length=96 (0x60)   Peripheral device type: disk
Vendor identification: PURE
Product identification: FlashArray
Product revision level: 8888
Unit serial number: 1D6DB146171D4E32000113E6

The existing device must first be removed for another initial
rediscovery to correct the issue.

This hole makes the unavailable ALUA state unattractive. Allowing
the peripheral qualifier set to 001b to still create an sd device
on discovery corrects this hole.

Thanks,
Brian

On Tue, Jan 24, 2023 at 4:02 PM Martin K. Petersen <martin.petersen@oracle.com> wrote:
> 
> 
> I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
> PDT=other values to SCSI_SCAN_TARGET_PRESENT").
> 
> I have been spending quite a bit of time digging through old SCSI and
> controller specs. As far as I can tell the original Linux behavior was
> correct. Recent SPC specs are very abstract in this department resulting
> in unfortunate ambiguity. But originally PQ=1 meant "LUN supports this
> peripheral device type but no physical device is currently connected".
> 
> Based on this original definition, PQ=1 has been widely used throughout
> the industry as a means to avoid associating an ULD driver with a
> device. The LUN is accessible (primary commands, etc.) but no media is
> present (no physical device connected).
> 
> Our original algorithm, which I would like to reinstate, is essentially
> the following (in slightly unrolled form):
> 
>        if (PQ == 3)
>            /* Don't expose device */
>        else if ((PQ == 1 || sdev->pdt_1f_for_no_lun) && PDT == 0x1f)
>            /* Don't expose device */
>        else if (PQ == 1) {
>            /* Expose device, don't bind ULD */
>        } else /* PQ == 0 */
>            /* Expose device, bind ULD if PDT is supported */
> 
> I would like to understand why -- in the case of the IBM 2145 --
> exposing the sg device caused problems. Li: Can you shed some light on
> the problems caused by 2145 LUNs reporting PQ=1?
> 
> Thanks!
> 
> --
> Martin K. Petersen      Oracle Linux Engineering

Re: The PQ=1 saga

[Martin K. Petersen]

Brian,

> For a completely separate reason I would like to see PQ=1 expose the
> sd device.

The host RAID controller case we could probably cover without relying on
PQ=1 at all (we kind-of already do). But there are also storage arrays
out there that rely on PQ=1 to inhibit devices being claimed.
Historically they did this because some other operating systems couldn't
handle a processor device type. So I suspect that keying off of TPGS
alone is probably not sufficient to determine whether PQ=1 should cause
us to attach a ULD or not in your scenario.

> ALUA state transitions from unavailable back to another state does not
> work depending on what state devices are in when they are initially
> discovered.  In the ALUA unavailable state the peripheral qualifier of
> the device should also be set to 001b.

Yep, an unfortunate wrinkle in the spec (although it makes sense).

> This hole makes the unavailable ALUA state unattractive. Allowing the
> peripheral qualifier set to 001b to still create an sd device on
> discovery corrects this hole.

Does your implementation actually support READ CAPACITY etc. in
unavailable state? Otherwise we'd end up with zero-length, read-only
block devices with no logical block size. And we've been down that path
before and that is no fun.

I suspect it would be better to trigger a re-probe of the device when
transitioning out of unavailable state. Most of the logic is already in
place and we reread VPD pages, etc. I believe there are only a few
pieces missing from being able to do a full in-place update.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: The PQ=1 saga

[Brian Bunker]

> On Jan 24, 2023, at 8:04 PM, Martin K. Petersen <martin.petersen@oracle.com> wrote:
> 
> 
> Brian,
> 
>> For a completely separate reason I would like to see PQ=1 expose the
>> sd device.
> 
> The host RAID controller case we could probably cover without relying on
> PQ=1 at all (we kind-of already do). But there are also storage arrays
> out there that rely on PQ=1 to inhibit devices being claimed.
> Historically they did this because some other operating systems couldn't
> handle a processor device type. So I suspect that keying off of TPGS
> alone is probably not sufficient to determine whether PQ=1 should cause
> us to attach a ULD or not in your scenario.
I had the idea to change the check in scsi_sysfs.c from:
-       return (sdp->inq_periph_qual == SCSI_INQ_PQ_CON)? 1: 0;
+       return (sdp->inq_periph_qual != SCSI_INQ_PQ_NOT_CAP)? 1: 0;

This would allow PQ=1 but not PQ=3 which I think is the right thing to do.
> 
>> ALUA state transitions from unavailable back to another state does not
>> work depending on what state devices are in when they are initially
>> discovered.  In the ALUA unavailable state the peripheral qualifier of
>> the device should also be set to 001b.
> 
> Yep, an unfortunate wrinkle in the spec (although it makes sense).
> 
>> This hole makes the unavailable ALUA state unattractive. Allowing the
>> peripheral qualifier set to 001b to still create an sd device on
>> discovery corrects this hole.
> 
> Does your implementation actually support READ CAPACITY etc. in
> unavailable state? Otherwise we'd end up with zero-length, read-only
> block devices with no logical block size. And we've been down that path
> before and that is no fun.
Yes we can support read capacity when in the unavailable state. For
us the unavailable state means that one controller or array can not
reach the other controller or array on the backend but the front end
ports are still connected. They are up from an initiator transport
perspective.
> 
> I suspect it would be better to trigger a re-probe of the device when
> transitioning out of unavailable state. Most of the logic is already in
> place and we reread VPD pages, etc. I believe there are only a few
> pieces missing from being able to do a full in-place update.
Unfortunately this doesn’t work. This does work in other OS’s where
I can logout the connection, and when it comes back it will discover
that the LUN no longer has the PQ set and will come online fine. But
in Linux this results in (after the PLOGI and PRLI):

Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: INQUIRY pass 1 length 36
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: INQUIRY successful with code 0x0
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: INQUIRY pass 2 length 96
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: INQUIRY successful with code 0x0
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: peripheral device type of 31, no device added
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: Sending REPORT LUNS to (try 0)
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: REPORT LUNS successful (try 0) result 0x0
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:0: scsi scan: REPORT LUN scan
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:1: scsi scan: device exists on 7:0:0:1
Jan 25 11:42:28 init72-5 kernel: scsi 7:0:0:2: scsi scan: device exists on 7:0:0:2

So unless those devices are removed before the rescan, which I
cannot control from the target, an sd device will not be created on 
the rescanning after the logout.

/dev/sg5  7 0 0 1  0  PURE      FlashArray        8888
/dev/sg6  7 0 0 2  0  PURE      FlashArray        8888

Thanks,
Brian
                
> 
> -- 
> Martin K. Petersen Oracle Linux Engineering

Re: The PQ=1 saga

[Bart Van Assche]

On 1/25/23 11:56, Brian Bunker wrote:
>> On Jan 24, 2023, at 8:04 PM, Martin K. Petersen <martin.petersen@oracle.com> wrote:
>> I suspect it would be better to trigger a re-probe of the device when
>> transitioning out of unavailable state. Most of the logic is already in
>> place and we reread VPD pages, etc. I believe there are only a few
>> pieces missing from being able to do a full in-place update.
 >
> Unfortunately this doesn’t work.

If that doesn't work today, how about implementing this functionality?
How about letting the ALUA device handler do a SCSI rescan if the ALUA state
changes from a state in which READ CAPACITY does not have to be supported
(standby, unavailable) to a state in which READ CAPACITY must be supported
(active/optimized, active/non-optimized)?

Thanks,

Bart.

Re: The PQ=1 saga

[Bart Van Assche]

On 1/24/23 16:01, Martin K. Petersen wrote:
> I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
> PDT=other values to SCSI_SCAN_TARGET_PRESENT").

That sounds good to me.

Bart.

Re: The PQ=1 saga

[Martin Wilck]

On Tue, 2023-01-24 at 17:41 -0800, Bart Van Assche wrote:
> On 1/24/23 16:01, Martin K. Petersen wrote:
> > I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
> > PDT=other values to SCSI_SCAN_TARGET_PRESENT").
> 
> That sounds good to me.
> 
> Bart.
> 

I agree.

Martin

Re: The PQ=1 saga

[Hannes Reinecke]

On 1/25/23 09:33, Martin Wilck wrote:
> On Tue, 2023-01-24 at 17:41 -0800, Bart Van Assche wrote:
>> On 1/24/23 16:01, Martin K. Petersen wrote:
>>> I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
>>> PDT=other values to SCSI_SCAN_TARGET_PRESENT").
>>
>> That sounds good to me.
>>
>> Bart.
>>
> 
> I agree.
> 
Yep.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                Kernel Storage Architect
hare@suse.de                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman

Re: The PQ=1 saga

[Brian Bunker]

I was doing some more testing of this since it has been a while since I 
ran these tests. It looks like reverting this will make the particular situation
that I am worried about even worse. I will put the detail in.

With this in place (before you revert it). When SCSI devices are discovered
and some have a PQ=1 because they are in an unavailable ALUA state:

Jan 27 12:05:29 localhost kernel: scsi 7:0:0:1: scsi scan: peripheral device type of 31, no device added

I don’t know if this intentional with the patch or not but any devices with PQ=1
will not create SCSI devices. The logging is deceptive too since the device type
Is 0 and not 31. In my case I have two paths to LUN 1. One is ALUA AO and the 
other in ALUA unavailable.

With this patch in I only get an sd device and an sg device for the AO path. 
The other path to LUN 1 gets no devices created because it is caught in the
If condition logged above.

Because there are no SCSI devices created, when the ALUA state returns
to an active state, a SCSI rescan, which I can trigger from the target will result
in the devices getting created since the initial scan never created devices.

Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 1 length 36
Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 2 length 96
Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6

Things are good with both paths to LUN 1 showing up. It is not optimal since the
target has to trigger a LUN scan on the initiator affecting all paths to those target
ports.

With the revert of this, things are a little different, but the way they had been in
the past.

Jan 27 13:41:19 localhost kernel: sd 7:0:1:1: Asymmetric access state changed
Jan 27 13:41:56 localhost kernel: scsi 7:0:1:1: alua: Detached
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 1 length 36
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 2 length 96
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Direct-Access     PURE     FlashArray       8888 PQ: 1 ANSI: 6
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: supports implicit TPGS
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: device naa.624a9370acc31b042de141460001141c port group 0 rel port a
Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Attached scsi generic sg7 type 0

Now an sg device is created but not an sd device. This means that there will be
no way for this device to get an sd device created once the ALUA state goes into
an active state.

The same thing done on the target that worked above no longer does:

Jan 27 13:47:48 localhost kernel: scsi 7:0:1:1: scsi scan: device exists on 7:0:1:1

To get around this, the existing disk must be deleted so it is not caught in the rescan
check. This cannot be controlled on the target, but it will require manual intervention
on the initiator.

So the question becomes how should initial scan work when a LUN has a PQ=1 set.
It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
handled. Why allow an sg device but not an sd one on initial scan in this case? There
are probably many ways to fix this. I think the simplest is to allow sd device creation
on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
targets. The other approach which will no longer work after the revert is to trigger a
rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
the ALUA device handler will not help since there is no device to transition if it is
discovered with PQ=1.

Thanks,
Brian


> On Jan 26, 2023, at 1:01 AM, Hannes Reinecke <hare@suse.de> wrote:
> 
> On 1/25/23 09:33, Martin Wilck wrote:
>> On Tue, 2023-01-24 at 17:41 -0800, Bart Van Assche wrote:
>>> On 1/24/23 16:01, Martin K. Petersen wrote:
>>>> I would like to revert commit 948e922fc446 ("scsi: core: map PQ=1,
>>>> PDT=other values to SCSI_SCAN_TARGET_PRESENT").
>>> 
>>> That sounds good to me.
>>> 
>>> Bart.
>>> 
>> I agree.
> Yep.
> 
> Cheers,
> 
> Hannes
> -- 
> Dr. Hannes Reinecke                Kernel Storage Architect
> hare@suse.de                              +49 911 74053 688
> SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
> HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
> Myers, Andrew McDonald, Martje Boudien Moerman
> 

Re: The PQ=1 saga

[Bart Van Assche]

On 1/27/23 11:57, Brian Bunker wrote:
> So the question becomes how should initial scan work when a LUN has a PQ=1 set.
> It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
> handled. Why allow an sg device but not an sd one on initial scan in this case? There
> are probably many ways to fix this. I think the simplest is to allow sd device creation
> on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
> targets. The other approach which will no longer work after the revert is to trigger a
> rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
> the ALUA device handler will not help since there is no device to transition if it is
> discovered with PQ=1.

When Mike Christie and I looked into the ALUA unavailable state many 
years ago we concluded that using this state is so troublesome that it's 
better not to use this state. How about using active/optimized and 
active/non-optimized instead?

Thanks,

Bart.

Re: The PQ=1 saga

[Brian Bunker]

> On Jan 27, 2023, at 12:43 PM, Bart Van Assche <bvanassche@acm.org> wrote:
> 
> On 1/27/23 11:57, Brian Bunker wrote:
>> So the question becomes how should initial scan work when a LUN has a PQ=1 set.
>> It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
>> handled. Why allow an sg device but not an sd one on initial scan in this case? There
>> are probably many ways to fix this. I think the simplest is to allow sd device creation
>> on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
>> targets. The other approach which will no longer work after the revert is to trigger a
>> rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
>> the ALUA device handler will not help since there is no device to transition if it is
>> discovered with PQ=1.
> 
> When Mike Christie and I looked into the ALUA unavailable state many years ago we concluded that using this state is so troublesome that it's better not to use this state. How about using active/optimized and active/non-optimized instead?
I can’t do that because active/non-optimized is just advisory. If a request comes down that path
it must succeed. In our case, we can’t allow that. Our only choice is between non-active
ALUA states. We can use standby but that invites SCSI commands we would rather
not have to deal with, most notably persistent reservations. ALUA unavailable is the
attractive one but it comes with the PQ=1 baggage.

Thanks,
Brian
> Thanks,
> 
> Bart.
> 

Re: The PQ=1 saga

[Hannes Reinecke]

On 1/27/23 20:57, Brian Bunker wrote:
> I was doing some more testing of this since it has been a while since I
> ran these tests. It looks like reverting this will make the particular situation
> that I am worried about even worse. I will put the detail in.
> 
> With this in place (before you revert it). When SCSI devices are discovered
> and some have a PQ=1 because they are in an unavailable ALUA state:
> 
> Jan 27 12:05:29 localhost kernel: scsi 7:0:0:1: scsi scan: peripheral device type of 31, no device added
> 
> I don’t know if this intentional with the patch or not but any devices with PQ=1
> will not create SCSI devices. The logging is deceptive too since the device type
> Is 0 and not 31. In my case I have two paths to LUN 1. One is ALUA AO and the
> other in ALUA unavailable.
> 
> With this patch in I only get an sd device and an sg device for the AO path.
> The other path to LUN 1 gets no devices created because it is caught in the
> If condition logged above.
> 
> Because there are no SCSI devices created, when the ALUA state returns
> to an active state, a SCSI rescan, which I can trigger from the target will result
> in the devices getting created since the initial scan never created devices.
> 
> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 1 length 36
> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 2 length 96
> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
> 
> Things are good with both paths to LUN 1 showing up. It is not optimal since the
> target has to trigger a LUN scan on the initiator affecting all paths to those target
> ports.
> 
> With the revert of this, things are a little different, but the way they had been in
> the past.
> 
> Jan 27 13:41:19 localhost kernel: sd 7:0:1:1: Asymmetric access state changed
> Jan 27 13:41:56 localhost kernel: scsi 7:0:1:1: alua: Detached
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 1 length 36
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 2 length 96
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Direct-Access     PURE     FlashArray       8888 PQ: 1 ANSI: 6
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: supports implicit TPGS
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: device naa.624a9370acc31b042de141460001141c port group 0 rel port a
> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Attached scsi generic sg7 type 0
> 
> Now an sg device is created but not an sd device. This means that there will be
> no way for this device to get an sd device created once the ALUA state goes into
> an active state.
> 
> The same thing done on the target that worked above no longer does:
> 
> Jan 27 13:47:48 localhost kernel: scsi 7:0:1:1: scsi scan: device exists on 7:0:1:1
> 
> To get around this, the existing disk must be deleted so it is not caught in the rescan
> check. This cannot be controlled on the target, but it will require manual intervention
> on the initiator.
> 
> So the question becomes how should initial scan work when a LUN has a PQ=1 set.
> It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
> handled. Why allow an sg device but not an sd one on initial scan in this case? There
> are probably many ways to fix this. I think the simplest is to allow sd device creation
> on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
> targets. The other approach which will no longer work after the revert is to trigger a
> rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
> the ALUA device handler will not help since there is no device to transition if it is
> discovered with PQ=1.
> 
Sheesh.

There _is_ an easy solution for this, and that is to not use PQ=1 in 
conjunction with ALUA unavailable :-)

Hiding PQ=1 devices did serve the purpose for linux as we still cannot 
to a 'real' rescan of a SCSI device; the 'vendor' and 'model' string is 
pretty much fixed for the lifetime of the device, alongside with the 
entire standard inquiry data. So if anything changes here we have to 
delete the device before we can properly read it.

(which also means that I'll have to retract my earlier comment about 
this being a good idea ...)

And in the absence of that hiding PQ=1 devices is the best we can do.
The alternative would be to implement a 'real' device rescan; but that 
was too daunting a challenge to be undertaken until now.
Things did change in the meantime, so maybe it's time to revisit that.

But really, we should ask vendors to _not_ use PQ=1 when using ALUA. I 
fail to see the benefit of this as both have roughly the same meaning; 
if you have ALUA unavailable you can't access the device, hence it's 
completely irrelevant what PQ says. And same for the other way round: if 
PQ=1 is set really the only ALUA state which makes sense is 'unavailable'.

Sadly it's not so easy to fix things up in the SCSI stack, as the PQ 
setting is evaluated during scanning, and the ALUA state way back later.

Cheers,

Hannes

Re: The PQ=1 saga

[Brian Bunker]

> On Jan 30, 2023, at 5:35 AM, Hannes Reinecke <hare@suse.de> wrote:
> 
> On 1/27/23 20:57, Brian Bunker wrote:
>> I was doing some more testing of this since it has been a while since I
>> ran these tests. It looks like reverting this will make the particular situation
>> that I am worried about even worse. I will put the detail in.
>> With this in place (before you revert it). When SCSI devices are discovered
>> and some have a PQ=1 because they are in an unavailable ALUA state:
>> Jan 27 12:05:29 localhost kernel: scsi 7:0:0:1: scsi scan: peripheral device type of 31, no device added
>> I don’t know if this intentional with the patch or not but any devices with PQ=1
>> will not create SCSI devices. The logging is deceptive too since the device type
>> Is 0 and not 31. In my case I have two paths to LUN 1. One is ALUA AO and the
>> other in ALUA unavailable.
>> With this patch in I only get an sd device and an sg device for the AO path.
>> The other path to LUN 1 gets no devices created because it is caught in the
>> If condition logged above.
>> Because there are no SCSI devices created, when the ALUA state returns
>> to an active state, a SCSI rescan, which I can trigger from the target will result
>> in the devices getting created since the initial scan never created devices.
>> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 1 length 36
>> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
>> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 2 length 96
>> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
>> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
>> Things are good with both paths to LUN 1 showing up. It is not optimal since the
>> target has to trigger a LUN scan on the initiator affecting all paths to those target
>> ports.
>> With the revert of this, things are a little different, but the way they had been in
>> the past.
>> Jan 27 13:41:19 localhost kernel: sd 7:0:1:1: Asymmetric access state changed
>> Jan 27 13:41:56 localhost kernel: scsi 7:0:1:1: alua: Detached
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 1 length 36
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 2 length 96
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Direct-Access     PURE     FlashArray       8888 PQ: 1 ANSI: 6
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: supports implicit TPGS
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: device naa.624a9370acc31b042de141460001141c port group 0 rel port a
>> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Attached scsi generic sg7 type 0
>> Now an sg device is created but not an sd device. This means that there will be
>> no way for this device to get an sd device created once the ALUA state goes into
>> an active state.
>> The same thing done on the target that worked above no longer does:
>> Jan 27 13:47:48 localhost kernel: scsi 7:0:1:1: scsi scan: device exists on 7:0:1:1
>> To get around this, the existing disk must be deleted so it is not caught in the rescan
>> check. This cannot be controlled on the target, but it will require manual intervention
>> on the initiator.
>> So the question becomes how should initial scan work when a LUN has a PQ=1 set.
>> It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
>> handled. Why allow an sg device but not an sd one on initial scan in this case? There
>> are probably many ways to fix this. I think the simplest is to allow sd device creation
>> on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
>> targets. The other approach which will no longer work after the revert is to trigger a
>> rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
>> the ALUA device handler will not help since there is no device to transition if it is
>> discovered with PQ=1.
> Sheesh.
> 
> There _is_ an easy solution for this, and that is to not use PQ=1 in conjunction with ALUA unavailable :-)
> 
> Hiding PQ=1 devices did serve the purpose for linux as we still cannot to a 'real' rescan of a SCSI device; the 'vendor' and 'model' string is pretty much fixed for the lifetime of the device, alongside with the entire standard inquiry data. So if anything changes here we have to delete the device before we can properly read it.
> 
> (which also means that I'll have to retract my earlier comment about this being a good idea ...)
> 
> And in the absence of that hiding PQ=1 devices is the best we can do.
> The alternative would be to implement a 'real' device rescan; but that was too daunting a challenge to be undertaken until now.
> Things did change in the meantime, so maybe it's time to revisit that.
> 
> But really, we should ask vendors to _not_ use PQ=1 when using ALUA. I fail to see the benefit of this as both have roughly the same meaning; if you have ALUA unavailable you can't access the device, hence it's completely irrelevant what PQ says. And same for the other way round: if PQ=1 is set really the only ALUA state which makes sense is 'unavailable'.
> 
> Sadly it's not so easy to fix things up in the SCSI stack, as the PQ setting is evaluated during scanning, and the ALUA state way back later.
> 
> Cheers,
> 
> Hannes
What about something like this? This will remove the device if the PQ=1 and re-discover it. If the TPG
remains unavailable, it will just be created in the same way. If the TPG has moved to an active state
the newly created device will be an available sd device. This way at least target vendors can cause
the initiator to rescan and get the devices from unavailable to an active state without the manual
Intervention on each host having to remove the devices with the PQ=1 set and rescan manually.
If the manual removal of devices is required, it does make ALUA unavailable unrealistic.

diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index f9b18fdc7b3c..9ff9ca1b963e 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -1123,6 +1123,36 @@ static unsigned char *scsi_inq_str(unsigned char *buf, unsigned char *inq,
 }
 #endif
 
+/**
+ * scsi_remove_offline_device - remove the device if the criteria met
+ * * @sdev:    scsi_device to check
+ *
+ * Description:
+ * A SCSI device which is part of a TPG in the unavailable state will
+ * have the PQ=1. If the device is discovered this way, there is no
+ * way for it to transition to an active state. The device must be
+ * removed and rediscovered during rescan in the event that the TPG
+ * has transitioned to an active state.
+ *
+ * Return:
+ * true: the conditions are met for device removal
+ * false: the conditions are not met
+ **/
+static bool scsi_remove_offline_device(struct scsi_device *sdev)
+{
+       if (sdev == NULL || sdev->handler == NULL)
+               return false;
+
+       if (sdev->inq_periph_qual == SCSI_INQ_PQ_NOT_CON &&
+           (strncmp(sdev->handler->name, "alua", 4) == 0)) {
+               SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
+                                 "scsi scan: discovered not accessible %s\n",
+                                 dev_name(&sdev->sdev_gendev)));
+               return true;
+       }
+       return false;
+}
+
 /**
  * scsi_probe_and_add_lun - probe a LUN, if a LUN is found add it
  * @starget:   pointer to target device structure
@@ -1161,6 +1191,10 @@ static int scsi_probe_and_add_lun(struct scsi_target *starget,
         * host adapter calls into here with rescan == 0.
         */
        sdev = scsi_device_lookup_by_target(starget, lun);
+       if (scsi_remove_offline_device(sdev)) {
+               __scsi_remove_device(sdev);
+               sdev = NULL;
+       }
        if (sdev) {
                if (rescan != SCSI_SCAN_INITIAL || !scsi_device_created(sdev)) {
                        SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,

Thanks,
Brian

Re: The PQ=1 saga

[Brian Bunker]

Hello all,

I took another shot at this one. This one I think addresses any
concerns that were brought up:

https://lore.kernel.org/linux-scsi/20251114160213.21243-1-brian@purestorage.com/

Thanks,
Brian


On Tue, Jan 31, 2023 at 1:00 PM Brian Bunker <brian@purestorage.com> wrote:
>
>
> > On Jan 30, 2023, at 5:35 AM, Hannes Reinecke <hare@suse.de> wrote:
> >
> > On 1/27/23 20:57, Brian Bunker wrote:
> >> I was doing some more testing of this since it has been a while since I
> >> ran these tests. It looks like reverting this will make the particular situation
> >> that I am worried about even worse. I will put the detail in.
> >> With this in place (before you revert it). When SCSI devices are discovered
> >> and some have a PQ=1 because they are in an unavailable ALUA state:
> >> Jan 27 12:05:29 localhost kernel: scsi 7:0:0:1: scsi scan: peripheral device type of 31, no device added
> >> I don’t know if this intentional with the patch or not but any devices with PQ=1
> >> will not create SCSI devices. The logging is deceptive too since the device type
> >> Is 0 and not 31. In my case I have two paths to LUN 1. One is ALUA AO and the
> >> other in ALUA unavailable.
> >> With this patch in I only get an sd device and an sg device for the AO path.
> >> The other path to LUN 1 gets no devices created because it is caught in the
> >> If condition logged above.
> >> Because there are no SCSI devices created, when the ALUA state returns
> >> to an active state, a SCSI rescan, which I can trigger from the target will result
> >> in the devices getting created since the initial scan never created devices.
> >> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 1 length 36
> >> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
> >> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY pass 2 length 96
> >> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: scsi scan: INQUIRY successful with code 0x0
> >> Jan 27 12:26:04 localhost kernel: scsi 7:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
> >> Things are good with both paths to LUN 1 showing up. It is not optimal since the
> >> target has to trigger a LUN scan on the initiator affecting all paths to those target
> >> ports.
> >> With the revert of this, things are a little different, but the way they had been in
> >> the past.
> >> Jan 27 13:41:19 localhost kernel: sd 7:0:1:1: Asymmetric access state changed
> >> Jan 27 13:41:56 localhost kernel: scsi 7:0:1:1: alua: Detached
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 1 length 36
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY pass 2 length 96
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: scsi scan: INQUIRY successful with code 0x0
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Direct-Access     PURE     FlashArray       8888 PQ: 1 ANSI: 6
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: supports implicit TPGS
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: alua: device naa.624a9370acc31b042de141460001141c port group 0 rel port a
> >> Jan 27 13:42:22 localhost kernel: scsi 7:0:1:1: Attached scsi generic sg7 type 0
> >> Now an sg device is created but not an sd device. This means that there will be
> >> no way for this device to get an sd device created once the ALUA state goes into
> >> an active state.
> >> The same thing done on the target that worked above no longer does:
> >> Jan 27 13:47:48 localhost kernel: scsi 7:0:1:1: scsi scan: device exists on 7:0:1:1
> >> To get around this, the existing disk must be deleted so it is not caught in the rescan
> >> check. This cannot be controlled on the target, but it will require manual intervention
> >> on the initiator.
> >> So the question becomes how should initial scan work when a LUN has a PQ=1 set.
> >> It is a valid, by spec with ALUA state unavailable but doesn’t seem to be
> >> handled. Why allow an sg device but not an sd one on initial scan in this case? There
> >> are probably many ways to fix this. I think the simplest is to allow sd device creation
> >> on LUNs were PQ=1, and only restrict PQ=3. I am not sure the side effect of this on other
> >> targets. The other approach which will no longer work after the revert is to trigger a
> >> rescan from the target. This is sub-optimal since it is disruptive. Any approach involving
> >> the ALUA device handler will not help since there is no device to transition if it is
> >> discovered with PQ=1.
> > Sheesh.
> >
> > There _is_ an easy solution for this, and that is to not use PQ=1 in conjunction with ALUA unavailable :-)
> >
> > Hiding PQ=1 devices did serve the purpose for linux as we still cannot to a 'real' rescan of a SCSI device; the 'vendor' and 'model' string is pretty much fixed for the lifetime of the device, alongside with the entire standard inquiry data. So if anything changes here we have to delete the device before we can properly read it.
> >
> > (which also means that I'll have to retract my earlier comment about this being a good idea ...)
> >
> > And in the absence of that hiding PQ=1 devices is the best we can do.
> > The alternative would be to implement a 'real' device rescan; but that was too daunting a challenge to be undertaken until now.
> > Things did change in the meantime, so maybe it's time to revisit that.
> >
> > But really, we should ask vendors to _not_ use PQ=1 when using ALUA. I fail to see the benefit of this as both have roughly the same meaning; if you have ALUA unavailable you can't access the device, hence it's completely irrelevant what PQ says. And same for the other way round: if PQ=1 is set really the only ALUA state which makes sense is 'unavailable'.
> >
> > Sadly it's not so easy to fix things up in the SCSI stack, as the PQ setting is evaluated during scanning, and the ALUA state way back later.
> >
> > Cheers,
> >
> > Hannes
> What about something like this? This will remove the device if the PQ=1 and re-discover it. If the TPG
> remains unavailable, it will just be created in the same way. If the TPG has moved to an active state
> the newly created device will be an available sd device. This way at least target vendors can cause
> the initiator to rescan and get the devices from unavailable to an active state without the manual
> Intervention on each host having to remove the devices with the PQ=1 set and rescan manually.
> If the manual removal of devices is required, it does make ALUA unavailable unrealistic.
>
> diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> index f9b18fdc7b3c..9ff9ca1b963e 100644
> --- a/drivers/scsi/scsi_scan.c
> +++ b/drivers/scsi/scsi_scan.c
> @@ -1123,6 +1123,36 @@ static unsigned char *scsi_inq_str(unsigned char *buf, unsigned char *inq,
>  }
>  #endif
>
> +/**
> + * scsi_remove_offline_device - remove the device if the criteria met
> + * * @sdev:    scsi_device to check
> + *
> + * Description:
> + * A SCSI device which is part of a TPG in the unavailable state will
> + * have the PQ=1. If the device is discovered this way, there is no
> + * way for it to transition to an active state. The device must be
> + * removed and rediscovered during rescan in the event that the TPG
> + * has transitioned to an active state.
> + *
> + * Return:
> + * true: the conditions are met for device removal
> + * false: the conditions are not met
> + **/
> +static bool scsi_remove_offline_device(struct scsi_device *sdev)
> +{
> +       if (sdev == NULL || sdev->handler == NULL)
> +               return false;
> +
> +       if (sdev->inq_periph_qual == SCSI_INQ_PQ_NOT_CON &&
> +           (strncmp(sdev->handler->name, "alua", 4) == 0)) {
> +               SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
> +                                 "scsi scan: discovered not accessible %s\n",
> +                                 dev_name(&sdev->sdev_gendev)));
> +               return true;
> +       }
> +       return false;
> +}
> +
>  /**
>   * scsi_probe_and_add_lun - probe a LUN, if a LUN is found add it
>   * @starget:   pointer to target device structure
> @@ -1161,6 +1191,10 @@ static int scsi_probe_and_add_lun(struct scsi_target *starget,
>          * host adapter calls into here with rescan == 0.
>          */
>         sdev = scsi_device_lookup_by_target(starget, lun);
> +       if (scsi_remove_offline_device(sdev)) {
> +               __scsi_remove_device(sdev);
> +               sdev = NULL;
> +       }
>         if (sdev) {
>                 if (rescan != SCSI_SCAN_INITIAL || !scsi_device_created(sdev)) {
>                         SCSI_LOG_SCAN_BUS(3, sdev_printk(KERN_INFO, sdev,
>
> Thanks,
> Brian
>
>


--
Brian Bunker
PURE Storage, Inc.
brian@purestorage.com