Apple Xserve RAID and qlogic ISP2312 (qla2300)

Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

	Hi,

	I have the hardware from the subject, the problem i have is that the
scsi layer doesn't see any luns i made on the raid except LUN 0. here is an
example:

[root@zerg-b root]# cd /proc/scsi/qla2xxx/  
[root@zerg-b qla2xxx]# cat 0
QLogic PCI to Fibre Channel Host Adapter for IBM HS20:
        Firmware version 3.02.30 IPX, Driver version 8.00.00b15-k
Entry address = f88f1000
ISP: ISP2312, Serial# N02631
Request Queue = 0x35780000, Response Queue = 0x35750000
Request Queue count = 2048, Response Queue count = 512
Total number of active commands = 0
Total number of interrupts = 345631
    Device queue depth = 0x10
Number of free request entries = 2047
Number of mailbox timeouts = 0
Number of ISP aborts = 0
Number of loop resyncs = 0
Number of retries for empty slots = 0
Number of reqs in pending_q= 0, retry_q= 0, done_q= 0, scsi_retry_q= 0
Host adapter:loop state = <READY>, flags = 0x1803
Dpc flags = 0x0
MBX flags = 0x0
Link down Timeout = 030
Port down retry = 030
Login retry count = 030
Commands retried with dropped frame(s) = 0
Product ID = 4953 5020 2020 0002


SCSI Device Information:
scsi-qla0-adapter-node=2000000d60d367e0;
scsi-qla0-adapter-port=2100000d60d367e0;
scsi-qla0-target-0=60003930000026b4;

SCSI LUN Information:
(Id:Lun)  * - indicates lun is not registered with the OS.
( 0: 5): Total reqs 0, Pending reqs 0, flags 0x0*, 0:0:83 00
[root@zerg-b qla2xxx]# 

	you can see that the driver has found the 5th LUN but scsi_scan
doesn't se it when scanning, scsi_probe_and_add_lun for LUN 0 returns
SCSI_SCAN_NO_RESPONSE instead of SCSI_SCAN_TARGET_PRESENT. I have added
{"APPLE", "Xserve RAID", NULL, BLIST_SPARSELUN}, in scsi_devinfo.c so
scsi_scan will try and perform a sparse scan on the LUNs but that doesn't
help because scanning LUN 0 fails.

Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 1 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 2 lun 0
Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000

	details on hardware: IBM HS20 blade with Qlogic FC module, 

02:02.0 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02)
02:02.1 Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter (rev 02)

	storage is APPLE Xserve RAID, latest firmware with lun masking activated, for this node
the 5th LUN is allowed. using add-single-device does the trick but i need a
kernel to boot from FC. here is the output after 

# echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi

Oct 28 02:31:38 zerg-b kernel: scsi_scan_host_selected: <0:0:0:5>
Oct 28 02:31:38 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 5
Oct 28 02:31:38 zerg-b kernel: scsi scan: 1st INQUIRY successful with code 0x0
Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID       Rev: 1.21
Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
Oct 28 02:31:38 zerg-b kernel: qla2300 0000:02:02.0: scsi(0:0:0:5): Enabled tagged queuing, queue depth 32.

	any patches, suggestions, questions etc are appreciated.
	please CC me as I'm not on the list.

-- 
Kat

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Patrick Mansfield]

On Thu, Oct 28, 2004 at 02:33:21AM +0300, Catalin Muresan wrote:
> 
> 	Hi,
> 
> 	I have the hardware from the subject, the problem i have is that the
> scsi layer doesn't see any luns i made on the raid except LUN 0. here is an
> example:

LUN 0 didn't show up according to your log output and comments below.

> [root@zerg-b root]# cd /proc/scsi/qla2xxx/  
> [root@zerg-b qla2xxx]# cat 0
> QLogic PCI to Fibre Channel Host Adapter for IBM HS20:
>         Firmware version 3.02.30 IPX, Driver version 8.00.00b15-k

> SCSI Device Information:
> scsi-qla0-adapter-node=2000000d60d367e0;
> scsi-qla0-adapter-port=2100000d60d367e0;
> scsi-qla0-target-0=60003930000026b4;
> 
> SCSI LUN Information:
> (Id:Lun)  * - indicates lun is not registered with the OS.
> ( 0: 5): Total reqs 0, Pending reqs 0, flags 0x0*, 0:0:83 00

The adapter seems to think there is a target and lun 0.

Did anything show up on host 1?

> 	you can see that the driver has found the 5th LUN but scsi_scan
> doesn't se it when scanning, scsi_probe_and_add_lun for LUN 0 returns
> SCSI_SCAN_NO_RESPONSE instead of SCSI_SCAN_TARGET_PRESENT. I have added
> {"APPLE", "Xserve RAID", NULL, BLIST_SPARSELUN}, in scsi_devinfo.c so
> scsi_scan will try and perform a sparse scan on the LUNs but that doesn't
> help because scanning LUN 0 fails.

Yep.

> Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
> Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
> Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000

The 0x10000 is a DID_NO_CONNECT, generally the adapter can't talk to
the target at all.

It seem like the storage was seen, then went away and then came back.

Do you always get the same behaviour?

Are there other messages in your log? Just post the entire log without
the scsi logging on, from adapter load time to scan completion.

> 	storage is APPLE Xserve RAID, latest firmware with lun masking activated, for this node
> the 5th LUN is allowed. using add-single-device does the trick but i need a
> kernel to boot from FC. here is the output after 

> # echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi

You can also use:

	echo "- - -" > /sys/class/scsi_host/host0/scan

-- Patrick Mansfield

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

	Hi,

On Thu, Oct 28, 2004 at 07:37:34AM -0700, Patrick Mansfield wrote:
 > On Thu, Oct 28, 2004 at 02:33:21AM +0300, Catalin Muresan wrote:
 > > 
 > > 	Hi,
 > > 
 > > 	I have the hardware from the subject, the problem i have is that the
 > > scsi layer doesn't see any luns i made on the raid except LUN 0. here is an
 > > example:
 > 
 > LUN 0 didn't show up according to your log output and comments below.

	the storage performs LUN masking so there's only LUN 5 available on
ID 0.

 > > [root@zerg-b root]# cd /proc/scsi/qla2xxx/  
 > > [root@zerg-b qla2xxx]# cat 0
 > > QLogic PCI to Fibre Channel Host Adapter for IBM HS20:
 > >         Firmware version 3.02.30 IPX, Driver version 8.00.00b15-k
 > 
 > > SCSI Device Information:
 > > scsi-qla0-adapter-node=2000000d60d367e0;
 > > scsi-qla0-adapter-port=2100000d60d367e0;
 > > scsi-qla0-target-0=60003930000026b4;
 > > 
 > > SCSI LUN Information:
 > > (Id:Lun)  * - indicates lun is not registered with the OS.
 > > ( 0: 5): Total reqs 0, Pending reqs 0, flags 0x0*, 0:0:83 00
 > 
 > The adapter seems to think there is a target and lun 0.

	the adapter sees id 0 lun 5 ( 0: 5) because it blindly scans every lun

 > Did anything show up on host 1?

	host 1 is not connected to the FC fabric because i have only 1 FC
switch module installed, that's why i didn't include that info - short no.

 > > 	you can see that the driver has found the 5th LUN but scsi_scan
 > > doesn't se it when scanning, scsi_probe_and_add_lun for LUN 0 returns
 > > SCSI_SCAN_NO_RESPONSE instead of SCSI_SCAN_TARGET_PRESENT. I have added
 > > {"APPLE", "Xserve RAID", NULL, BLIST_SPARSELUN}, in scsi_devinfo.c so
 > > scsi_scan will try and perform a sparse scan on the LUNs but that doesn't
 > > help because scanning LUN 0 fails.
 > 
 > Yep.
 > 
 > > Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
 > > Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
 > > Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
 > 
 > The 0x10000 is a DID_NO_CONNECT, generally the adapter can't talk to
 > the target at all.

	I don't know what the SCSI spec has to say about this, but by
looking at the code it should respond with SCSI_SCAN_TARGET_PRESENT.

 > It seem like the storage was seen, then went away and then came back.

	no, it's just that the storage or the controller don't report
anything for masked luns. It looks more and more a storage problem, I'm
gonna try and bug Apple about this. What does the SCSI spec says about this?

 > Do you always get the same behaviour?

	yes.

 > Are there other messages in your log? Just post the entire log without
 > the scsi logging on, from adapter load time to scan completion.

	ok but is like there's no devices attached (it's an older log
because i've rebooted with another kernel that scans every lun on every
target, thus finding the storage)

Oct 27 22:23:22 zerg-b kernel: QLogic Fibre Channel HBA Driver (c02ad498)
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:02:02.0[A] -> GSI 18 (level, low) -> IRQ 177
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Found an ISP2312, irq 177, iobase 0xf8802000
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Configuring PCI space...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Configure NVRAM parameters...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Verifying loaded RISC code...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Waiting for LIP to complete...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: LOOP UP detected (2 Gbps).
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: Topology - (F_Port), Host Loop address 0xffff
Oct 27 22:23:22 zerg-b kernel: scsi0 : qla2xxx
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.0: 
Oct 27 22:23:22 zerg-b kernel:  QLogic Fibre Channel HBA Driver: 8.00.00b15-k
Oct 27 22:23:22 zerg-b kernel:   QLogic IBM HS20 - 
Oct 27 22:23:22 zerg-b kernel:   ISP2312: PCI-X (133 MHz) @ 0000:02:02.0 hdma-, host#=0, fw=3.02.30 IPX
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:02:02.1[B] -> GSI 19 (level, low) -> IRQ 185
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Found an ISP2312, irq 185, iobase 0xf8808000
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Configuring PCI space...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Configure NVRAM parameters...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Verifying loaded RISC code...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Waiting for LIP to complete...
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: Cable is unplugged...
Oct 27 22:23:22 zerg-b kernel: scsi1 : qla2xxx
Oct 27 22:23:22 zerg-b kernel: qla2300 0000:02:02.1: 
Oct 27 22:23:22 zerg-b kernel:  QLogic Fibre Channel HBA Driver: 8.00.00b15-k
Oct 27 22:23:22 zerg-b kernel:   QLogic IBM HS20 - 
Oct 27 22:23:22 zerg-b kernel:   ISP2312: PCI-X (133 MHz) @ 0000:02:02.1 hdma-, host#=1, fw=3.02.30 IPX
Oct 27 22:23:22 zerg-b kernel: ACPI: PCI interrupt 0000:00:0f.2[A] -> GSI 7 (level, low) -> IRQ 7

 > > 	storage is APPLE Xserve RAID, latest firmware with lun masking activated, for this node
 > > the 5th LUN is allowed. using add-single-device does the trick but i need a
 > > kernel to boot from FC. here is the output after 
 > 
 > > # echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi
 > 
 > You can also use:
 > 
 > 	echo "- - -" > /sys/class/scsi_host/host0/scan

	my goal is to have a kernel with which I can boot from FC, but i see
that can be done only with an ugly patch(for now) or with tricks from
initrd.

 > -- Patrick Mansfield

-- 
Kat

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Patrick Mansfield]

On Thu, Oct 28, 2004 at 06:35:22PM +0300, Catalin Muresan wrote:

Ok, that should have been lun 5.

>  > > Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
>  > > Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
>  > > Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
>  > 
>  > The 0x10000 is a DID_NO_CONNECT, generally the adapter can't talk to
>  > the target at all.
> 
> 	I don't know what the SCSI spec has to say about this, but by
> looking at the code it should respond with SCSI_SCAN_TARGET_PRESENT.

Hmmm ... my interpretation of DID_NO_CONNECT was that the adapter cannot
talk to the target at all.

You should be able to send an INQUIRY to any LUN on a target, the HBA
really shouldn't block it.

Given that, it is correct for the scan to give SCSI_SCAN_TARGET_PRESENT
on a DID_NO_CONNECT.

So, it looks like the qlogic is giving a DID_NO_CONNECT when it should
not. We could hack around this in scsi_scan.c but that might be bad.

James or Andrew what do you think?

Catalin - does the device support REPORT LUNS? Do you have that
configured? It would avoid this problem.

> 	my goal is to have a kernel with which I can boot from FC, but i see
> that can be done only with an ugly patch(for now) or with tricks from
> initrd.

What tricks and ugly patch? I haven't tried boot via qlogic on an HS20.

-- Patrick Mansfield

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Patrick Mansfield]

On Thu, Oct 28, 2004 at 09:42:10AM -0700, Patrick Mansfield wrote:

> Hmmm ... my interpretation of DID_NO_CONNECT was that the adapter cannot
> talk to the target at all.
> 
> You should be able to send an INQUIRY to any LUN on a target, the HBA
> really shouldn't block it.
> 
> Given that, it is correct for the scan to give SCSI_SCAN_TARGET_PRESENT
> on a DID_NO_CONNECT.

Ooops, I meant to say SCSI_SCAN_NO_RESPONSE, that is:

Given that, it is correct for the scan to give SCSI_SCAN_NO_RESPONSE
on a DID_NO_CONNECT.

> 
> So, it looks like the qlogic is giving a DID_NO_CONNECT when it should
> not. We could hack around this in scsi_scan.c but that might be bad.
> 
> James or Andrew what do you think?

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Andrew Vasquez]

On Thu, 28 Oct 2004, Patrick Mansfield wrote:

> On Thu, Oct 28, 2004 at 06:35:22PM +0300, Catalin Muresan wrote:
> 
> Ok, that should have been lun 5.
> 
> >  > > Oct 28 01:59:37 zerg-b kernel: scsi_scan_host_selected: <0:4294967295:4294967295:4294967295>
> >  > > Oct 28 01:59:37 zerg-b kernel: scsi scan: INQUIRY to host 0 channel 0 id 0 lun 0
> >  > > Oct 28 01:59:37 zerg-b kernel: scsi scan: 1st INQUIRY failed with code 0x10000
> >  > 
> >  > The 0x10000 is a DID_NO_CONNECT, generally the adapter can't talk to
> >  > the target at all.
> > 
> > 	I don't know what the SCSI spec has to say about this, but by
> > looking at the code it should respond with SCSI_SCAN_TARGET_PRESENT.
> 
> Hmmm ... my interpretation of DID_NO_CONNECT was that the adapter cannot
> talk to the target at all.
> 
> You should be able to send an INQUIRY to any LUN on a target, the HBA
> really shouldn't block it.
> 
> Given that, it is correct for the scan to give SCSI_SCAN_TARGET_PRESENT
> on a DID_NO_CONNECT.
> 

Not wanting to go into the history of the qla2xxx driver's internal
port/lun structures, let me just say that the driver should always
create an internal 'lun 0' object so that the mid-layer can perform
its scan.

> So, it looks like the qlogic is giving a DID_NO_CONNECT when it should
> not. We could hack around this in scsi_scan.c but that might be bad.
> 
> James or Andrew what do you think?
> 

Yuck, the midlayer should not have to change.

> Catalin - does the device support REPORT LUNS? Do you have that
> configured? It would avoid this problem.
> 

Yes, Catalin, could you find that out?  I've only had a small amount
of exposure with the XRAID box and know that there were some quirks
about the device not performing an RFT_ID with the SNS, not sure if
that was addressed.

> > 	my goal is to have a kernel with which I can boot from FC, but i see
> > that can be done only with an ugly patch(for now) or with tricks from
> > initrd.
> 
> What tricks and ugly patch? I haven't tried boot via qlogic on an HS20.
> 

Catalin,  could you enable some debug settings:

In qla_dbg.h, modify the following line:

	/* #define QL_DEBUG_LEVEL_2  */ /* Output error msgs to COM1 */

to read as:

	#define QL_DEBUG_LEVEL_2   /* Output error msgs to COM1 */

In qla_settings.h, modify the following line:

	#define DEBUG_QLA2100           0       /* For Debug of qla2x00 */

to read as:

	#define DEBUG_QLA2100           1       /* For Debug of qla2x00 */

recompile the driver, load, and forward over the relevent
var/log/messages snippet to me.


BTW: Regarding the internal queues and port/lun objects -- most of our
'experimental' work with 2.6 is begin done with our iSCSI driver.
This includes removal of the driver's internal queuing mechanism and
dependencies on these port/lun objects being built internally during
the driver scan.  Porting (actually mostly pruning) to the FC driver
will follow shortly.

--
Andrew Vasquez

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

On Thu, Oct 28, 2004 at 10:21:43AM -0700, Andrew Vasquez wrote:
 > Not wanting to go into the history of the qla2xxx driver's internal
 > port/lun structures, let me just say that the driver should always
 > create an internal 'lun 0' object so that the mid-layer can perform
 > its scan.

	that sound like fixing hardware bugs in software ;)

 > > Catalin - does the device support REPORT LUNS? Do you have that
 > > configured? It would avoid this problem.
 > > 
 > 
 > Yes, Catalin, could you find that out?  I've only had a small amount
 > of exposure with the XRAID box and know that there were some quirks
 > about the device not performing an RFT_ID with the SNS, not sure if
 > that was addressed.

	i don't know how to find that, i'm gonna go and look at the source
and try to figure it out untill you respond.

 > > > 	my goal is to have a kernel with which I can boot from FC, but i see
 > > > that can be done only with an ugly patch(for now) or with tricks from
 > > > initrd.
 > > 
 > > What tricks and ugly patch? I haven't tried boot via qlogic on an HS20.
 > > 
 > 
 > Catalin,  could you enable some debug settings:
 > 
 > In qla_dbg.h, modify the following line:
 > 
 > 	/* #define QL_DEBUG_LEVEL_2  */ /* Output error msgs to COM1 */
 > 
 > to read as:
 > 
 > 	#define QL_DEBUG_LEVEL_2   /* Output error msgs to COM1 */
 > 
 > In qla_settings.h, modify the following line:
 > 
 > 	#define DEBUG_QLA2100           0       /* For Debug of qla2x00 */
 > 
 > to read as:
 > 
 > 	#define DEBUG_QLA2100           1       /* For Debug of qla2x00 */
 > 
 > recompile the driver, load, and forward over the relevent
 > var/log/messages snippet to me.

	will do that asap.

 > BTW: Regarding the internal queues and port/lun objects -- most of our
 > 'experimental' work with 2.6 is begin done with our iSCSI driver.
 > This includes removal of the driver's internal queuing mechanism and
 > dependencies on these port/lun objects being built internally during
 > the driver scan.  Porting (actually mostly pruning) to the FC driver
 > will follow shortly.
 > 
 > --
 > Andrew Vasquez

-- 
Kat

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Patrick Mansfield]

On Fri, Oct 29, 2004 at 11:58:00AM +0300, Catalin Muresan wrote:

>  > > Catalin - does the device support REPORT LUNS? Do you have that
>  > > configured? It would avoid this problem.
>  > > 
>  > 
>  > Yes, Catalin, could you find that out?  I've only had a small amount
>  > of exposure with the XRAID box and know that there were some quirks
>  > about the device not performing an RFT_ID with the SNS, not sure if
>  > that was addressed.
> 
> 	i don't know how to find that, i'm gonna go and look at the source
> and try to figure it out untill you respond.

I forgot a few things:

1) The REPORT LUNS is always on in current 2.6.x kernels, I do not remember
what kernel version first had the change.

2) If LUN 0 gets a DID_NO_CONNECT (and we do return the
SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via
a sequential scan.

3) We have a problem for REPORT LUN with sparse lun devices, in short,
you won't get a REPORT LUN scan if no LUN 0 is configured. I thought Kurt
was working on a fix for that, this post:

http://marc.theaimsgroup.com/?t=109545967900002&r=1&w=2

4) You posted as part of your logs:

Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID Rev: 1.21
Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access ANSI SCSI revision: 02

For SCSI 2 devices by default the REPORT LUN scan won't be used. Some
storage arrays have an option as to what SCSI level to report, try that or
use the BLIST_REPORTLUN2 devinfo flag. 

If you had configured in only LUNs 0 1 and 5, the sequential scan (with no
sparse lun) will stop scanning after LUN 2 is not seen.

In short you need to first figure out why LUN 0 is getting a
DID_NO_CONNECT, and after that the sequential scan should work OK for your
*current* configuration (only LUN 5 configured), though the REPORT LUN
scan or sparse lun scan will be needed to avoid sparse LUNs as noted
above.

-- Patrick Mansfield

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

On Fri, Oct 29, 2004 at 11:06:33AM -0700, Patrick Mansfield wrote:
 > I forgot a few things:
 > 
 > 1) The REPORT LUNS is always on in current 2.6.x kernels, I do not remember
 > what kernel version first had the change.

	from a quick glance at the source it's tried if LUN 0 returns
TARGET_PRESENT (meaning target is there but no lun), so some changes need to
be made to have REPORT LUN (also fake the device as scsi-3). 

 > 2) If LUN 0 gets a DID_NO_CONNECT (and we do return the
 > SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via
 > a sequential scan.

	yep. what i don't know is how the REPORT LUN command is issued: to a
LUN or to the target ? if it's to the LUN it's hopeless and only changes in
the driver can help.

 > 3) We have a problem for REPORT LUN with sparse lun devices, in short,
 > you won't get a REPORT LUN scan if no LUN 0 is configured. I thought Kurt
 > was working on a fix for that, this post:
 > 
 > http://marc.theaimsgroup.com/?t=109545967900002&r=1&w=2

	even if not configured, I think the target should respond with
TARGET_PRESENT but this storage doesn't communicate at all.

 > 4) You posted as part of your logs:
 > 
 > Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID Rev: 1.21
 > Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access ANSI SCSI revision: 02
 > 
 > For SCSI 2 devices by default the REPORT LUN scan won't be used. Some
 > storage arrays have an option as to what SCSI level to report, try that or
 > use the BLIST_REPORTLUN2 devinfo flag. 
 > 
 > If you had configured in only LUNs 0 1 and 5, the sequential scan (with no
 > sparse lun) will stop scanning after LUN 2 is not seen.

	if I would have LUN 0 configured and any other combinations it would
have worked because i've already whitelisted the storage as BLIST_SPARSELUN

 > In short you need to first figure out why LUN 0 is getting a
 > DID_NO_CONNECT, and after that the sequential scan should work OK for your

	I blame it on the storage for not following the spec now.

 > *current* configuration (only LUN 5 configured), though the REPORT LUN

	REPORT LUN is sent to LUN 0 ? if yes it doesn't help because the
storage is willomg to cdommunicate only with unmasked LUNs. 

 > scan or sparse lun scan will be needed to avoid sparse LUNs as noted
 > above.

	I'll continue on this monday when I'll be at work and near the
equipments.

 > -- Patrick Mansfield

-- 
Kat

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

	I have the attached patch from Andrew which fixes the qla2300 part
of the problem (attached) and a patch is still needed for the case where
there are LUN0,3 to blacklist the device as BLIST_SPARSELUN.

On Fri, Oct 29, 2004 at 11:06:33AM -0700, Patrick Mansfield wrote:
 > On Fri, Oct 29, 2004 at 11:58:00AM +0300, Catalin Muresan wrote:
 > 
 > >  > > Catalin - does the device support REPORT LUNS? Do you have that
 > >  > > configured? It would avoid this problem.
 > >  > > 
 > >  > 
 > >  > Yes, Catalin, could you find that out?  I've only had a small amount
 > >  > of exposure with the XRAID box and know that there were some quirks
 > >  > about the device not performing an RFT_ID with the SNS, not sure if
 > >  > that was addressed.
 > > 
 > > 	i don't know how to find that, i'm gonna go and look at the source
 > > and try to figure it out untill you respond.
 > 
 > I forgot a few things:
 > 
 > 1) The REPORT LUNS is always on in current 2.6.x kernels, I do not remember
 > what kernel version first had the change.
 > 
 > 2) If LUN 0 gets a DID_NO_CONNECT (and we do return the
 > SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via
 > a sequential scan.

	we need to have something on LUN 0 to be able to send REPORT LUN, in
this case we don't.

 > 3) We have a problem for REPORT LUN with sparse lun devices, in short,
 > you won't get a REPORT LUN scan if no LUN 0 is configured. I thought Kurt
 > was working on a fix for that, this post:
 > 
 > http://marc.theaimsgroup.com/?t=109545967900002&r=1&w=2

	here it was the controller which disabled the LUNs if there were no
devices connected to the LUN. Now (after the patch) we can scan the storage
with REPORT LUNS (if supported) or sequential.

 > 4) You posted as part of your logs:
 > 
 > Oct 28 02:31:38 zerg-b kernel:   Vendor: APPLE     Model: Xserve RAID Rev: 1.21
 > Oct 28 02:31:38 zerg-b kernel:   Type:   Direct-Access ANSI SCSI revision: 02
 > 
 > For SCSI 2 devices by default the REPORT LUN scan won't be used. Some
 > storage arrays have an option as to what SCSI level to report, try that or
 > use the BLIST_REPORTLUN2 devinfo flag. 
 > 
 > If you had configured in only LUNs 0 1 and 5, the sequential scan (with no
 > sparse lun) will stop scanning after LUN 2 is not seen.

	that is where the BLIST_SPARSELUN patch comes to help for non-SCSI-3
devices.

 > In short you need to first figure out why LUN 0 is getting a
 > DID_NO_CONNECT, and after that the sequential scan should work OK for your

	it's because the qla2300 does not enable in it's list LUNs that have
no devices attached. I still think that it should configure all available
LUNs if there is a target present, but this is ok.

 > *current* configuration (only LUN 5 configured), though the REPORT LUN
 > scan or sparse lun scan will be needed to avoid sparse LUNs as noted
 > above.

	Do you need me to test some other stuff or we can "close" this
issue?	

 > -- Patrick Mansfield

-- 
 Catalin Muresan
 ASTRAL Internet & Data
 Technology Manager

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Patrick Mansfield]

[ resending ... not sure if this went out at all ]

On Mon, Nov 01, 2004 at 12:56:11PM +0200, Catalin Muresan wrote:
> 
> 	I have the attached patch from Andrew which fixes the qla2300 part
> of the problem (attached) and a patch is still needed for the case where
> there are LUN0,3 to blacklist the device as BLIST_SPARSELUN.

No attachment ...

>  > 2) If LUN 0 gets a DID_NO_CONNECT (and we do return the
>  > SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via
>  > a sequential scan.
> 
> 	we need to have something on LUN 0 to be able to send REPORT LUN, in
> this case we don't.

You don't need storage configured on LUN 0 to send it a REPORT LUN, if
there is a target there, you can (well should) be able to send it a REPORT
LUN.

You do need the driver to allow commands to be issued to LUN 0 (that is
what the patch should have fixed).

SCSI spec says:

	A SCSI device shall support a REPORT LUNS command that is
	addressed to logical unit zero. Support of the REPORT LUNS command
	by logical units other than logical unit zero is optional.

> 	Do you need me to test some other stuff or we can "close" this
> issue?	

In order of best to worst solutions:

If device supports REPORT LUNs:

1) configure the array to report as SCSI-3

2) black list as BLIST_REPORTLUN2

If no REPORT LUNs support:

3) black list as BLIST_SPARSELUN

-- Patrick Mansfield

Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Douglas Gilbert]

Patrick Mansfield wrote:
> [ resending ... not sure if this went out at all ]
> 
> On Mon, Nov 01, 2004 at 12:56:11PM +0200, Catalin Muresan wrote:
> 
>>	I have the attached patch from Andrew which fixes the qla2300 part
>>of the problem (attached) and a patch is still needed for the case where
>>there are LUN0,3 to blacklist the device as BLIST_SPARSELUN.
> 
> 
> No attachment ...
> 
> 
>> > 2) If LUN 0 gets a DID_NO_CONNECT (and we do return the
>> > SCSI_SCAN_NO_RESPONSE), we won't scan anything, via REPORT LUN or via
>> > a sequential scan.
>>
>>	we need to have something on LUN 0 to be able to send REPORT LUN, in
>>this case we don't.
> 
> 
> You don't need storage configured on LUN 0 to send it a REPORT LUN, if
> there is a target there, you can (well should) be able to send it a REPORT
> LUN.
> 
> You do need the driver to allow commands to be issued to LUN 0 (that is
> what the patch should have fixed).
> 
> SCSI spec says:
> 
> 	A SCSI device shall support a REPORT LUNS command that is
> 	addressed to logical unit zero. Support of the REPORT LUNS command
> 	by logical units other than logical unit zero is optional.

Patrick,
That would be SAM or SAM-2. In SAM-3 (and SPC-3) that requirement
has been widened to:
"All SCSI devices shall support LUN 0 (i.e. 00000000 00000000h) or
the REPORT LUNS well-known logical unit". [SAM-3 rev 14 section 4.9.3]

Having recently implemented sg_luns (in sg3_utils) I was surprised
to find a "select" field (byte 2) in REPORT LUN's cdb. When it is 0
(as per scsi_scan.c) the returned lun inventory does _not_ include
any well known lus. When select is 1 the returned lun inventory
only includes well known lus (and when select==2 all lus are
returned). Still this is not going to help in SPC-3 since the
device need not respond to lun 0 at all.

So if that fails scsi_scan.c might issue a REPORT LUNS command to
this lun: 0xc101 which is the value of the REPORT LUNS well-known lu.
Note that this is a 16 bit quantity (not 64 bit as I said in a
previous post about luns). For simplicity let us assume that
that it is at the top level. [If a device returns HiSup=1 in its
standard INQUIRY response then it claims to support a 4 level
hierarchy in its luns (with 16 bits in each level).]

SPC-3 defines 3 well known lus:
   - REPORT LUNS
   - ACCESS CONTROLS [0xc102]
   - TARGET LOG PAGES [0xc103]
and each of these supports a small number of (mandatory) SCSI
commands.

So if a modern SCSI enclosure or bridge uses well known lus
rather than the lun 0 hack, how will applications in linux
access them? They are certainly not block devices.

Doug Gilbert

Re: Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Luben Tuikov]

Douglas Gilbert wrote:
> So if that fails scsi_scan.c might issue a REPORT LUNS command to
> this lun: 0xc101 which is the value of the REPORT LUNS well-known lu.
 > Note that this is a 16 bit quantity (not 64 bit as I said in a
> previous post about luns). For simplicity let us assume that
> that it is at the top level. [If a device returns HiSup=1 in its
> standard INQUIRY response then it claims to support a 4 level
> hierarchy in its luns (with 16 bits in each level).]

And also supports REPORT LUNS.

So in either case, we send REPORT LUNS to LUN 0 and if
this fails to REPORT LUNS W-LUN (0xC101...).

Since the standard says that depending on the transport
protocol LUNs are either 16 or 64 bit values, how do
we decide this from SCSI Core/app. client?  Wouldn't it
be better to just send 64 bits with bytes 2 to 7 all 0-ed?

So in general SCSI Core always uses 64 bits and the LLDD
decides between 16 and 64 bits, since the transport could
be messier.

> So if a modern SCSI enclosure or bridge uses well known lus
> rather than the lun 0 hack, how will applications in linux
> access them? They are certainly not block devices.

Ah, good question.  No, they cannot be block devices since
they are not mountable.  But they are _addressable_ devices.

If it is decided to not create a new OS type, then those
could be char devices on the bus, and accessed via sg.

If it is decided to create a new OS type, then 'a' would
be appropriate to denote addressable devices, again
being accessed via sg.  Do you think expanders could
be represented by those device types too?

	Luben

Re: Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Patrick Mansfield]

> Douglas Gilbert wrote:
> >So if that fails scsi_scan.c might issue a REPORT LUNS command to
> >this lun: 0xc101 which is the value of the REPORT LUNS well-known lu.
> > Note that this is a 16 bit quantity (not 64 bit as I said in a
> >previous post about luns). For simplicity let us assume that
> >that it is at the top level. [If a device returns HiSup=1 in its
> >standard INQUIRY response then it claims to support a 4 level
> >hierarchy in its luns (with 16 bits in each level).]
> 
> And also supports REPORT LUNS.
> 
> So in either case, we send REPORT LUNS to LUN 0 and if
> this fails to REPORT LUNS W-LUN (0xC101...).

Do we also have to change to send the INQUIRY to the well known LUN?

And then someday add black and white list flags and code for them ...

Why is stuff like this even added to the standard??? Why not use LUN 0
(all zeroes) or at worst have one well known LUN like the all f's?

The hierarchical LUN stuff and the various address modes are dumb, just
treat the LUN as a tag/identifier (similiar to a TCP/IP network port) and
leave the interpretation of it up to the target device (or at worst the
transport).

-- Patrick Mansfield

Re: Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Luben Tuikov]

Patrick Mansfield wrote:
> Do we also have to change to send the INQUIRY to the well known LUN?
> 
> And then someday add black and white list flags and code for them ...
> 
> Why is stuff like this even added to the standard??? Why not use LUN 0
> (all zeroes) or at worst have one well known LUN like the all f's?
> 
> The hierarchical LUN stuff and the various address modes are dumb, just
> treat the LUN as a tag/identifier (similiar to a TCP/IP network port) and
> leave the interpretation of it up to the target device (or at worst the
> transport).

And I believe this is what a lot of vendors will do: support
REPORT LUN on LUN 0 regardless if whether there's a device
there or not.  BTW the standard says "LUN 0 or REPORT LUNS W-LUN",
so the vendors would take advantage of this for compatibility.

	Luben

Re: Report luns

[Douglas Gilbert]

Patrick Mansfield wrote:
>>Douglas Gilbert wrote:
>>
>>>So if that fails scsi_scan.c might issue a REPORT LUNS command to
>>>this lun: 0xc101 which is the value of the REPORT LUNS well-known lu.
>>>Note that this is a 16 bit quantity (not 64 bit as I said in a
>>>previous post about luns). For simplicity let us assume that
>>>that it is at the top level. [If a device returns HiSup=1 in its
>>>standard INQUIRY response then it claims to support a 4 level
>>>hierarchy in its luns (with 16 bits in each level).]
>>
>>And also supports REPORT LUNS.
>>
>>So in either case, we send REPORT LUNS to LUN 0 and if
>>this fails to REPORT LUNS W-LUN (0xC101...).
> 
> 
> Do we also have to change to send the INQUIRY to the well known LUN?

Yes, all well known lus have mandatory support for INQUIRY.

> And then someday add black and white list flags and code for them ...

Hope not. If an INQUIRY to lun 0 fails (and perhaps the transport level
doesn't object) then send an INQUIRY to lun 0xc101 . If that succeeds
then send a REPORT LUNS to lun 0xc101 ...

> Why is stuff like this even added to the standard??? Why not use LUN 0
> (all zeroes) or at worst have one well known LUN like the all f's?

All f's is already defined in SAM-3 as "lu not specified", whatever
that means :-)

> The hierarchical LUN stuff and the various address modes are dumb, just
> treat the LUN as a tag/identifier (similiar to a TCP/IP network port) and
> leave the interpretation of it up to the target device (or at worst the
> transport).

As you have worked in the scsi_scan.c area you are well placed
to comment. I have only recently been looking at luns. I also
wonder about the device peripheral type returned by an INQUIRY
sent to a W-LUN (i.e. well known lun). If it was closely associated
with disks (e.g. a RAID) and returned a "direct access" device then
the sd driver wouldn't make much sense of a W-LUN device. Note this
is not a problem while select=0 is set in scsi_scan's REPORT LUN
commands.


An example, inspired by the BCC draft, involving well
known logical units might help. Say we have
a bridge from SAS (or FC, it doesn't matter) to SPI-4 (U320).
One way to implement the bridge is to map the 15 possible
devices on the U320 bus to luns behind a single target in
a SAS domain (or SAS bus, it's the same thing). Lets assume
there are 5 U320 disks connected to the SPI bus and assume
SPI target ids 0 to 4 map to SAS luns 0 to 4.

So now if one sends a REPORT LUNS to lun 0 that U320 disk is
expected to know about its sibling disks!? That is why I used
the term "hack" with respect to lun 0. What really happens
is the device server in the bridge intercepts the REPORT LUNS
command and produces a response reflecting its knowledge of
those 5 U320 disks present. Why play this game? Why not talk
directly to the device server (addressed via a well known lu).
This way an INQUIRY to a W-LUN yields information about
the bridge, while an INQUIRY to lun 0 yields information about
that disk.
If the TARGET LOG PAGES W-LUN is supported then the
bridge can supply logs from its point of view (a SAS
bridge device) while a LOG SENSE command sent to lun 4
reports that U320 disk's logs.


Luben is probably correct in saying manufacturers will
support both lun 0 and the REPORT LUNS well known lu
in advanced products such as bridges. In Linux currently
we can detect the presence of well known lus (e.g.
sg_luns --select=1 /dev/sg3). If anything is found we
have no mechanism to probe them.

Doug Gilbert

Re: Report luns

[Luben Tuikov]

Douglas Gilbert wrote:
> 
> So now if one sends a REPORT LUNS to lun 0 that U320 disk is
> expected to know about its sibling disks!? That is why I used
> the term "hack" with respect to lun 0. What really happens
> is the device server in the bridge intercepts the REPORT LUNS
> command and produces a response reflecting its knowledge of
> those 5 U320 disks present. Why play this game? Why not talk
> directly to the device server (addressed via a well known lu).
> This way an INQUIRY to a W-LUN yields information about
> the bridge, while an INQUIRY to lun 0 yields information about
> that disk.

Very good point!

So we can reverse the logic into sending REPORT LUNS to
the REPORT LUNS W-LUN *first* and if this fails (most probably
in early years), then issue REPORT LUNS to LUN 0.

	Luben

Re: Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Douglas Gilbert]

Luben Tuikov wrote:
> Douglas Gilbert wrote:
> 
>> So if that fails scsi_scan.c might issue a REPORT LUNS command to
>> this lun: 0xc101 which is the value of the REPORT LUNS well-known lu.
> 
>  > Note that this is a 16 bit quantity (not 64 bit as I said in a
> 
>> previous post about luns). For simplicity let us assume that
>> that it is at the top level. [If a device returns HiSup=1 in its
>> standard INQUIRY response then it claims to support a 4 level
>> hierarchy in its luns (with 16 bits in each level).]
> 
> 
> And also supports REPORT LUNS.
> 
> So in either case, we send REPORT LUNS to LUN 0 and if
> this fails to REPORT LUNS W-LUN (0xC101...).
> 
> Since the standard says that depending on the transport
> protocol LUNs are either 16 or 64 bit values, how do
> we decide this from SCSI Core/app. client?  Wouldn't it
> be better to just send 64 bits with bytes 2 to 7 all 0-ed?
> 
> So in general SCSI Core always uses 64 bits and the LLDD
> decides between 16 and 64 bits, since the transport could
> be messier.
> 
>> So if a modern SCSI enclosure or bridge uses well known lus
>> rather than the lun 0 hack, how will applications in linux
>> access them? They are certainly not block devices.
> 
> 
> Ah, good question.  No, they cannot be block devices since
> they are not mountable.  But they are _addressable_ devices.
> 
> If it is decided to not create a new OS type, then those
> could be char devices on the bus, and accessed via sg.
> 
> If it is decided to create a new OS type, then 'a' would
> be appropriate to denote addressable devices, again
> being accessed via sg.  Do you think expanders could
> be represented by those device types too?

Luben,
I liked the way bsg had a single char device and used
a bind mechanism. However it is only possible to bind
to an existing block device.

SAS expanders (or at least their SMP target ports) are
a similar problem to W-LUNs. Namely we have no way to
"talk" to them from the user space currently in Linux.
SMP is not a SCSI command set (but it is close). Even if
sg could issue SMP commands, I'm not sure the scsi mid
level would be helpful when error conditions occurred
(e.g. SMP command timeout).

As for the addressability problem, sockets and connect
calls come to mind.

Doug Gilbert

Re: Report luns [was: Apple Xserve RAID and qlogic ISP2312 (qla2300)]

[Luben Tuikov]

Douglas Gilbert wrote:
> 
> SAS expanders (or at least their SMP target ports) are
> a similar problem to W-LUNs. Namely we have no way to
> "talk" to them from the user space currently in Linux.
> SMP is not a SCSI command set (but it is close). Even if
> sg could issue SMP commands, I'm not sure the scsi mid
> level would be helpful when error conditions occurred
> (e.g. SMP command timeout).
> 
> As for the addressability problem, sockets and connect
> calls come to mind.

Yeah, that's a pickle.

	Luben

Re: Apple Xserve RAID and qlogic ISP2312 (qla2300)

[Catalin Muresan]

On Thu, Oct 28, 2004 at 09:42:10AM -0700, Patrick Mansfield wrote:
 > What tricks and ugly patch? I haven't tried boot via qlogic on an HS20.

	trick: echo "scsi add-single-device 0 0 0 5" > /proc/scsi/scsi
	ugly patch, only to help me get past this issue:

--- linux-2.6.9-orig/drivers/scsi/scsi_scan.c	2004-10-19 00:54:55.000000000 +0300
+++ linux-2.6.9/drivers/scsi/scsi_scan.c	2004-10-28 14:53:28.355982680 +0300
@@ -1199,7 +1199,7 @@
 			 */
 			scsi_sequential_lun_scan(shost, channel, id, bflags,
 				       	res, sdev->scsi_level, rescan);
-	} else if (res == SCSI_SCAN_TARGET_PRESENT) {
+	} else if (res == SCSI_SCAN_TARGET_PRESENT || res == SCSI_SCAN_NO_RESPONSE) {
 		/*
 		 * There's a target here, but lun 0 is offline so we
 		 * can't use the report_lun scan.  Fall back to a



 > 
 > -- Patrick Mansfield

-- 
 Catalin Muresan
 ASTRAL Internet & Data
 Technology Manager