Re: Multi-Initiator SAS

Re: Multi-Initiator SAS

[Douglas Gilbert]

So it seems that the English word beginning with "affil"
caused my posts to be categorized as spam by vger.kernel.org
Still doesn't make a lot of sense because Darrick replied
to one of my posts with that word repeated from my post.

So this is a test that truncates those words.


-------- Original Message --------
Subject: Re: Multi-Initiator SAS
Date: Tue, 19 Sep 2006 19:54:30 -0400
From: Douglas Gilbert <dougg@torque.net>
Reply-To: dougg@torque.net
To: linux-scsi@vger.kernel.org
CC: Darrick J. Wong <djwong@us.ibm.com>,  Eric.Moore@lsil.com

Hmm, my initial reply to Darrick didn't seem to
make it to the list, but Darrick and Eric saw it.
The original post is forwarded at the bottom of
this post.


I forgot to mention that the SAS address of the expander is
needed. That is because I set environment variables thus:
# export SMP_UTILS_SAS_ADDR=0x500605b000000af0
# export SMP_UTILS_DEVICE=/dev/mptctl,0
on my test system.

Darrick J. Wong wrote:
<snip>
> # smp_discover --sa=0x50001c1716004000 -m /dev/mptctl
> Device <50001c1716004000>, expander:
>   phy   0:T:attached:[0000000000000000:00]
>   phy   1:T:attached:[0000000000000000:00]
>   phy   2:T:attached:[0000000000000000:00]
>   phy   3:T:attached:[0000000000000000:00]
>   phy   4:T:attached:[50001c1716004005:00  t(STP+SATA)]  1.5 Gbps
>   phy   5:T:attached:[0000000000000000:00]
>   phy   6:T:attached:[0000000000000000:00]
>   phy   7:T:attached:[0000000000000000:00]
>   phy   8:S:attached:[0000000000000000:00]
>   phy   9:S:attached:[0000000000000000:00]
>   phy  10:S:attached:[500062b000002fdc:03  i(SSP+STP+SMP)]  3 Gbps
>   phy  11:S:attached:[500062b000002fdc:02  i(SSP+STP+SMP)]  3 Gbps
>   phy  12:D:attached:[50001c171600400d:00  V i(SSP) t(SSP)]  3 Gbps

Judging from the OUI in the SAS address of the
expander, it is made by Vitesse with 12 external
phys and one "virtual" phy (phy_id 12). As Eric
pointed out 'smp_rep_manufacturer' utility should
tell us the manufacturer and the model of the expander.

Notice that the first 8 expander phys are table routed,
the next 4 are subtractive and the virtual (internal) one
has direct routing. When I was having some problems
someone advised me to keep away from the subtractive
routed phys (not sure why).

3 devices are attached to the expander:
  - a SATA disk (phy id 4)
  - a LSI HBA is attached via a two link wide port
    (expander phy ids 10+11)
  - an internal attachment to a SCSI device that can
    be both an initiator and target (interesting)

There is no sign of an adaptec HBA connected to the expander.
[Have a look at the "smp_discover --multiple --brief" output
 on http://www.torque.net/sg/smp_utils.html for comparison.]

> # smp_rep_phy_sata -p4 --sa=0x50001c1716004000 /dev/mptctl
> Report phy SATA response:
>   phy identifier: 4
>   STP I_T nexus loss occurred: 0
>   affil* supported: 1
>   affil* valid: 1
>   STP SAS address: 0x50001c1716004005
>   register device to host FIS:
>     34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
>   affil* STP initiator SAS address: 0x500062b000002fdc

This is SATA specific information on phy id 4. So it looks
like the LSI HBA (0x500062b000002fdc) is holding a "long
term" affil* as I have seen with the expander I test
with. Perhaps Eric could remind me whether the mptsas
LLD or firmware calls the CLOSE(CLEAR AFFIL*) link layer
primitive to end a STP tunnelled ATA command. Anyway the fact
the 'affil* valid' is set will lock thet SATA disk
to that HBA. While there is only one STP initiator (on the
LSI HBA) there is no contention to worry about.

>
> 0x500062b000002fdc is the SAS address of the LSI HBA, so it appears that
> we're in agreement.  Unfortunately, all attempts to access the disk
> result in I/O errors:
>
> end_request: I/O error, dev sda, sector 1024

Well I have been able to talk to a SATA (I+II) disks
attached to a LSI SASx12 expander from either a LSI
(1068e based) HBA or an adaptec 48300 HBA. For both HBAs
I used narrow links. So I would try narrowing the port
between the HBA and the expander
(e.g. 'smp_phy_control -p 11 -o dis ...') to see if that
makes any difference.

After that try another expander.

Doug Gilbert


-------- Original Message --------
Subject: Re: Multi-Initiator SAS
Date: Tue, 19 Sep 2006 15:45:08 -0400
From: Douglas Gilbert <dougg@torque.net>
Reply-To: dougg@torque.net
To: Darrick J. Wong <djwong@us.ibm.com>
CC: linux-scsi@vger.kernel.org,  Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
References: <451041AA.4060702@us.ibm.com>

Darrick J. Wong wrote:
> Hi everybody!
> 
> Alexis and I connected both a Adaptec 9410 and a LSI 1064E SAS initiator
> to an expander.  Both machines saw the disks attached to the expander,
> and both could send I/Os to the disks.  So far, so good.
> 
> We then connected a SATA disk to the expander.  libsas BUGd up in
> sas_ex_discover_end_dev (sas_expander.c:636):
> 
> BUG_ON(sas_port_add(phy->port) != 0);
> 
> mptsas didn't seem to do anything with the SATA device at all, though
> when we disconnected the SATA disk it recognized that a SATA device went
> away.  We'll have a look at the libsas problem in a jiffy.

Darrick,
The latter effect may be a feature of your expander rather
than a problem with libsas.

With the mptsas driver you can use smp_utils to look
at that expander via /dev/mptctl ('modprobe mptctl' first).
To get an overview of what the expander sees, try:
 # smp_discover -mb /dev/mptctl

Now have a closer look at the expander phy connected
to the SATA disk, assuming that is phy id 3 something like:
 # smp_rep_phy_sata -p 3 /dev/mptctl

What I have been seeing is affil* supported and valid,
and the "affil* STP initiator SAS address" as being
the SAS address of the HBA that can see the SATA disk.

My reading of affil* is that they are short term,
just covering the FIS register write then read associated
with one ATA command. I have been told by an authority on
the matter that they can be much longer term than that
which is exactly what my expander seems to be doing.
It seems like first come first serve, IOW whichever HBA
gets there first.

To make it a fair fight between the two HBAs, try to
disconnect that phy, then send a link reset, and see
who wins (and if that HBA continues to win over several
experiments).

  # smp_phy_control -p 3 -o dis /dev/mptctl
  # smp_phy_control -p 3 -o lr /dev/mptctl


When you think about it, do you really want two ATA hosts
on different machines trying to do NCQ at the same time
to the same SATA disk??

There is something that libsas and/or SAS LLDs may offer to
help this "only one initiator gets the SATA disk" feature:
the ability to exclude a SAS address/phy_id (on an expander)
from its attempts to make a device available.

Doug Gilbert

RE: Multi-Initiator SAS

[Moore, Eric]

On Tuesday, September 19, 2006 5:55 PM, Douglas Gilbert wrote: 


> This is SATA specific information on phy id 4. So it looks
> like the LSI HBA (0x500062b000002fdc) is holding a "long
> term" affiliation as I have seen with the expander I test
> with. Perhaps Eric could remind me whether the mptsas
> LLD or firmware calls the CLOSE(CLEAR AFFILIATION) link layer
> primitive to end a STP tunnelled ATA command. Anyway the fact
> the 'affiliation valid' is set will lock thet SATA disk
> to that HBA. While there is only one STP initiator (on the
> LSI HBA) there is no contention to worry about.
> 

Primatives are well below driver, as well as transport frames.
All the fusion driver does is pass message frames with SCSI_IO/cdbs.
We have access to config pages that provide info on the topology,
and lets use configure somethings. We have sas_iounit control, that
allows use to reset phys and links.  We have sata and smp passthrus,
other than that, everything is down in firmware.

If you want to know more about the affiliations, we will need
to start an internal discussion with firmware folks. Let me know.

Eric

RE: Multi-Initiator SAS

[Moore, Eric]

Darrick J. Wong wrote:

> Douglas Gilbert wrote:
> 
> > With the mptsas driver you can use smp_utils to look
> > at that expander via /dev/mptctl ('modprobe mptctl' first).
> > To get an overview of what the expander sees, try:
> >  # smp_discover -mb /dev/mptctl
> 
> Unfortunately, I see this:
> 


You need to pass the sas address of the expander.

./smp_discover --sa=sas_address /dev/mptctl

where sas_address can be found in
/sys/class/sas_device/expander-X:0/sas_address

RE: Multi-Initiator SAS

[Moore, Eric]

On Tuesday, September 19, 2006 1:15 PM,  Darrick J. Wong wrote:

> Alexis and I connected both a Adaptec 9410 and a LSI 1064E 
> SAS initiator
> to an expander.  Both machines saw the disks attached to the expander,
> and both could send I/Os to the disks.  So far, so good.
> 
> We then connected a SATA disk to the expander.  libsas BUGd up in
> sas_ex_discover_end_dev (sas_expander.c:636):
> 
> BUG_ON(sas_port_add(phy->port) != 0);
> 

Which expander are you using?  I believe James has some sata work
arounds
for the sasx12, which has byte ordering issues.

Eric

Re: Multi-Initiator SAS

[Darrick J. Wong]

Moore, Eric wrote:

> Which expander are you using?  I believe James has some sata work
> arounds
> for the sasx12, which has byte ordering issues.

A Vitesse VSC7160.

Also, I think I might have complained about I/O errors accessing the
SATA disk via mptsas--the SATA disk that we were using (Maxtor 6Y080M0)
yields the same errors when attached to the HBA.  I put in a Hitachi
SATA disk and it was fine with both controllers.

--D

Multi-Initiator SAS

[Darrick J. Wong]

Hi everybody!

Alexis and I connected both a Adaptec 9410 and a LSI 1064E SAS initiator
to an expander.  Both machines saw the disks attached to the expander,
and both could send I/Os to the disks.  So far, so good.

We then connected a SATA disk to the expander.  libsas BUGd up in
sas_ex_discover_end_dev (sas_expander.c:636):

BUG_ON(sas_port_add(phy->port) != 0);

mptsas didn't seem to do anything with the SATA device at all, though
when we disconnected the SATA disk it recognized that a SATA device went
away.  We'll have a look at the libsas problem in a jiffy.

--D

Re: Multi-Initiator SAS

[Darrick J. Wong]

Douglas Gilbert wrote:

> With the mptsas driver you can use smp_utils to look
> at that expander via /dev/mptctl ('modprobe mptctl' first).
> To get an overview of what the expander sees, try:
>  # smp_discover -mb /dev/mptctl

Unfortunately, I see this:

root@elm3a253:~# smp_discover -mb /dev/mptctl
MPTCOMMAND ioctl failed: No such device
ioctl failed
smp_send_req failed, res=-1

...a strace of the app reveals this:

stat("/dev/mptctl", {st_mode=S_IFCHR|0660, st_rdev=makedev(10, 220),
...}) = 0
open("/dev/mptctl", O_RDWR)             = 3
brk(0)                                  = 0x508000
brk(0x529000)                           = 0x529000
ioctl(3, 0xc0486d14, 0x508010)          = -1 ENODEV (No such device)

--D