mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD]

Hi,

I am seing a bug with the mpt2sas driver from the "Release 2.6.35.11" 
kernel (commit d6c90f5b218c1ddf1496045e3939b1c960c7cb9f, tag v2.6.35.11, 
long term support kernel).

I have a system that has both a LSI 1068e B3 (3G) based HBA (a SAS3801E) 
as well as a LSI SAS 2008 03 (6G) HBA (a SAS9200-8e). They both work as 
expected, but I am seeing a major difference in their respective /sysfs 
structure, especially regarding their phy's "sas_address" field, which 
seems to be a bug.

The 3G HBA's SAS addresses are associated to a specific port in sysfs, 
while the 6G one are associated to the actual HBA.

The 3G HBA is configured to have two wide ports, made up of 4 phys each, 
port 0 and port 1, and same configuration applies for the 6G HBA.
The ports are not in "auto" nor in narrow.
There are only two enabled ports on each HBA.


I get the following sysfs entries regarding the 3G one:

# cat 
/sys/class/sas_host/host5/device/phy-5\:0/sas_phy\:phy-5\:0/sas_address
0x50015b2a2000060f
# cat 
/sys/class/sas_host/host5/device/phy-5\:1/sas_phy\:phy-5\:1/sas_address
0x50015b2a2000060f
# cat 
/sys/class/sas_host/host5/device/phy-5\:2/sas_phy\:phy-5\:2/sas_address
0x50015b2a2000060f
# cat 
/sys/class/sas_host/host5/device/phy-5\:3/sas_phy\:phy-5\:3/sas_address
0x50015b2a2000060f
# cat 
/sys/class/sas_host/host5/device/phy-5\:4/sas_phy\:phy-5\:4/sas_address
0x50015b2a20000613
# cat 
/sys/class/sas_host/host5/device/phy-5\:5/sas_phy\:phy-5\:5/sas_address
0x50015b2a20000613
# cat 
/sys/class/sas_host/host5/device/phy-5\:6/sas_phy\:phy-5\:6/sas_address
0x50015b2a20000613
# cat 
/sys/class/sas_host/host5/device/phy-5\:7/sas_phy\:phy-5\:7/sas_address
0x50015b2a20000613

And these ones on the 6G:

# cat 
/sys/class/sas_host/host0/device/phy-0\:0/sas_phy\:phy-0\:0/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:1/sas_phy\:phy-0\:1/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:2/sas_phy\:phy-0\:2/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:3/sas_phy\:phy-0\:3/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:4/sas_phy\:phy-0\:4/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:5/sas_phy\:phy-0\:5/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:6/sas_phy\:phy-0\:6/sas_address
0x500605b002c99150
# cat 
/sys/class/sas_host/host0/device/phy-0\:7/sas_phy\:phy-0\:7/sas_address
0x500605b002c99150

As we can see above, the 3G HBA's phy "sas_address" sysfs entry displays 
correct information: A single SAS address for phy 0-3 and a single one 
for phy 4-7.

As we can also see using LSIUtil, each phy is numbered from a base 
address and incremented.
Also, the port sas address corresponds to the address of the starting 
phy of the port.
Port0's sas address is "0x50015b2a2000060f", which corresponds to the 
HBA's phy0 sas address.
Port1's sas address is "0x50015b2a20000613" which corresponds to the 
HBA's phy3 sas address, the starting phys for each of these ports.

Output of menu "16" from LSIUtil on the 3G HBA (1068e):

  B___T     SASAddress     PhyNum  Handle  Parent  Type
         50015b2a2000060f           0001           SAS Initiator
         50015b2a20000610           0002           SAS Initiator
         50015b2a20000611           0003           SAS Initiator
         50015b2a20000612           0004           SAS Initiator
         50015b2a20000613           0005           SAS Initiator
         50015b2a20000614           0006           SAS Initiator
         50015b2a20000615           0007           SAS Initiator
         50015b2a20000616           0008           SAS Initiator

However, when looking at the 6G HBA's sysfs information above, we can 
see that all phy's SAS address are identical, like if there was a single 
port made up of the entire 8 phys from the HBA.

When looking at LSIUtil below, we get more strangeness, where all phys 
have exactly the same SAS address:

Output of menu "16" from LSIUtil on the 6G HBA (2008):

  B___T     SASAddress     PhyNum  Handle  Parent  Type
         500605b002c99150           0001           SAS Initiator
         500605b002c99150           0002           SAS Initiator
         500605b002c99150           0003           SAS Initiator
         500605b002c99150           0004           SAS Initiator
         500605b002c99150           0005           SAS Initiator
         500605b002c99150           0006           SAS Initiator
         500605b002c99150           0007           SAS Initiator
         500605b002c99150           0008           SAS Initiator

But looking at LSIUtil's menu "13" on the 6G HBA proves that we indeed 
have 2 ports on that HBA:

PhyNum  Link      MinRate  MaxRate  Initiator  Target    Port
    0    Enabled     1.5      6.0    Enabled    Disabled    0
    1    Enabled     1.5      6.0    Enabled    Disabled    0
    2    Enabled     1.5      6.0    Enabled    Disabled    0
    3    Enabled     1.5      6.0    Enabled    Disabled    0
    4    Enabled     1.5      6.0    Enabled    Disabled    1
    5    Enabled     1.5      6.0    Enabled    Disabled    1
    6    Enabled     1.5      6.0    Enabled    Disabled    1
    7    Enabled     1.5      6.0    Enabled    Disabled    1

Somehow, on the 6G HBA using mpt2sas, all phys from a HBA seem to have 
the same SAS address, and all ports on that HBA, whether narrow (1Phy) 
or wide (4Phys), will seemingly have the same SAS address.

This is causing a good few issues with our system scripting, as we 
relied on SAS addresses to identify ports.

I searched on Google extensively before posting and couldn't find any 
mention of this issue.

Is this a known issue? If so, will this be resolved in a later version 
of the driver?

Thanks a lot in advance for your reply.

Regards,
Ben.

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Douglas Gilbert]

On 11-08-17 06:42 AM, Benjamin ESTRABAUD wrote:
> Hi,
>
> I am seing a bug with the mpt2sas driver from the "Release 2.6.35.11" kernel
> (commit d6c90f5b218c1ddf1496045e3939b1c960c7cb9f, tag v2.6.35.11, long term
> support kernel).
>
> I have a system that has both a LSI 1068e B3 (3G) based HBA (a SAS3801E) as well
> as a LSI SAS 2008 03 (6G) HBA (a SAS9200-8e). They both work as expected, but I
> am seeing a major difference in their respective /sysfs structure, especially
> regarding their phy's "sas_address" field, which seems to be a bug.
>
> The 3G HBA's SAS addresses are associated to a specific port in sysfs, while the
> 6G one are associated to the actual HBA.
>
> The 3G HBA is configured to have two wide ports, made up of 4 phys each, port 0
> and port 1, and same configuration applies for the 6G HBA.
> The ports are not in "auto" nor in narrow.
> There are only two enabled ports on each HBA.
>
>
> I get the following sysfs entries regarding the 3G one:
>
> # cat /sys/class/sas_host/host5/device/phy-5\:0/sas_phy\:phy-5\:0/sas_address
> 0x50015b2a2000060f
> # cat /sys/class/sas_host/host5/device/phy-5\:1/sas_phy\:phy-5\:1/sas_address
> 0x50015b2a2000060f
> # cat /sys/class/sas_host/host5/device/phy-5\:2/sas_phy\:phy-5\:2/sas_address
> 0x50015b2a2000060f
> # cat /sys/class/sas_host/host5/device/phy-5\:3/sas_phy\:phy-5\:3/sas_address
> 0x50015b2a2000060f
> # cat /sys/class/sas_host/host5/device/phy-5\:4/sas_phy\:phy-5\:4/sas_address
> 0x50015b2a20000613
> # cat /sys/class/sas_host/host5/device/phy-5\:5/sas_phy\:phy-5\:5/sas_address
> 0x50015b2a20000613
> # cat /sys/class/sas_host/host5/device/phy-5\:6/sas_phy\:phy-5\:6/sas_address
> 0x50015b2a20000613
> # cat /sys/class/sas_host/host5/device/phy-5\:7/sas_phy\:phy-5\:7/sas_address
> 0x50015b2a20000613
>
> And these ones on the 6G:
>
> # cat /sys/class/sas_host/host0/device/phy-0\:0/sas_phy\:phy-0\:0/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:1/sas_phy\:phy-0\:1/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:2/sas_phy\:phy-0\:2/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:3/sas_phy\:phy-0\:3/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:4/sas_phy\:phy-0\:4/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:5/sas_phy\:phy-0\:5/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:6/sas_phy\:phy-0\:6/sas_address
> 0x500605b002c99150
> # cat /sys/class/sas_host/host0/device/phy-0\:7/sas_phy\:phy-0\:7/sas_address
> 0x500605b002c99150
>
> As we can see above, the 3G HBA's phy "sas_address" sysfs entry displays correct
> information: A single SAS address for phy 0-3 and a single one for phy 4-7.
>
> As we can also see using LSIUtil, each phy is numbered from a base address and
> incremented.
> Also, the port sas address corresponds to the address of the starting phy of the
> port.
> Port0's sas address is "0x50015b2a2000060f", which corresponds to the HBA's phy0
> sas address.
> Port1's sas address is "0x50015b2a20000613" which corresponds to the HBA's phy3
> sas address, the starting phys for each of these ports.
>
> Output of menu "16" from LSIUtil on the 3G HBA (1068e):
>
> B___T SASAddress PhyNum Handle Parent Type
> 50015b2a2000060f 0001 SAS Initiator
> 50015b2a20000610 0002 SAS Initiator
> 50015b2a20000611 0003 SAS Initiator
> 50015b2a20000612 0004 SAS Initiator
> 50015b2a20000613 0005 SAS Initiator
> 50015b2a20000614 0006 SAS Initiator
> 50015b2a20000615 0007 SAS Initiator
> 50015b2a20000616 0008 SAS Initiator
>
> However, when looking at the 6G HBA's sysfs information above, we can see that
> all phy's SAS address are identical, like if there was a single port made up of
> the entire 8 phys from the HBA.
>
> When looking at LSIUtil below, we get more strangeness, where all phys have
> exactly the same SAS address:
>
> Output of menu "16" from LSIUtil on the 6G HBA (2008):
>
> B___T SASAddress PhyNum Handle Parent Type
> 500605b002c99150 0001 SAS Initiator
> 500605b002c99150 0002 SAS Initiator
> 500605b002c99150 0003 SAS Initiator
> 500605b002c99150 0004 SAS Initiator
> 500605b002c99150 0005 SAS Initiator
> 500605b002c99150 0006 SAS Initiator
> 500605b002c99150 0007 SAS Initiator
> 500605b002c99150 0008 SAS Initiator
>
> But looking at LSIUtil's menu "13" on the 6G HBA proves that we indeed have 2
> ports on that HBA:
>
> PhyNum Link MinRate MaxRate Initiator Target Port
> 0 Enabled 1.5 6.0 Enabled Disabled 0
> 1 Enabled 1.5 6.0 Enabled Disabled 0
> 2 Enabled 1.5 6.0 Enabled Disabled 0
> 3 Enabled 1.5 6.0 Enabled Disabled 0
> 4 Enabled 1.5 6.0 Enabled Disabled 1
> 5 Enabled 1.5 6.0 Enabled Disabled 1
> 6 Enabled 1.5 6.0 Enabled Disabled 1
> 7 Enabled 1.5 6.0 Enabled Disabled 1
>
> Somehow, on the 6G HBA using mpt2sas, all phys from a HBA seem to have the same
> SAS address, and all ports on that HBA, whether narrow (1Phy) or wide (4Phys),
> will seemingly have the same SAS address.
>
> This is causing a good few issues with our system scripting, as we relied on SAS
> addresses to identify ports.
>
> I searched on Google extensively before posting and couldn't find any mention of
> this issue.
>
> Is this a known issue? If so, will this be resolved in a later version of the
> driver?

Ben,
I noticed something similar when comparing LSI 3 Gbps and
6 Gbps SAS HBAs. I was looking at the same thing but from
an expander's perspective. I'm not so sure that the newer
6 Gbps HBAs are incorrect. IOW the older 3 Gbps HBAs might
be playing some tricks with the HBA's SAS addresses so that
it is not possible to set up a wide link spanning the
lower and upper 4-phy banks (e.g. a 5 phy wide link).

Do you know a reason why it is not preferably for every
phy on a SAS HBA to respond with the same SAS address?


As a practical matter a SAS HBA needs a single SAS address,
preferably printed on the board or its box. Then if you
manage to wipe its SAS address (e.g. by erasing its flash
to move from IR to IT firmware) then you know which SAS
address to re-instate :-)

Doug Gilbert

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD]

On 17/08/11 16:35, Douglas Gilbert wrote:
> On 11-08-17 06:42 AM, Benjamin ESTRABAUD wrote:
>> Hi,
>>
>> I am seing a bug with the mpt2sas driver from the "Release 2.6.35.11" 
>> kernel
>> (commit d6c90f5b218c1ddf1496045e3939b1c960c7cb9f, tag v2.6.35.11, 
>> long term
>> support kernel).
>>
>> I have a system that has both a LSI 1068e B3 (3G) based HBA (a 
>> SAS3801E) as well
>> as a LSI SAS 2008 03 (6G) HBA (a SAS9200-8e). They both work as 
>> expected, but I
>> am seeing a major difference in their respective /sysfs structure, 
>> especially
>> regarding their phy's "sas_address" field, which seems to be a bug.
>>
>> The 3G HBA's SAS addresses are associated to a specific port in 
>> sysfs, while the
>> 6G one are associated to the actual HBA.
>>
>> The 3G HBA is configured to have two wide ports, made up of 4 phys 
>> each, port 0
>> and port 1, and same configuration applies for the 6G HBA.
>> The ports are not in "auto" nor in narrow.
>> There are only two enabled ports on each HBA.
>>
>>
>> I get the following sysfs entries regarding the 3G one:
>>
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:0/sas_phy\:phy-5\:0/sas_address
>> 0x50015b2a2000060f
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:1/sas_phy\:phy-5\:1/sas_address
>> 0x50015b2a2000060f
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:2/sas_phy\:phy-5\:2/sas_address
>> 0x50015b2a2000060f
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:3/sas_phy\:phy-5\:3/sas_address
>> 0x50015b2a2000060f
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:4/sas_phy\:phy-5\:4/sas_address
>> 0x50015b2a20000613
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:5/sas_phy\:phy-5\:5/sas_address
>> 0x50015b2a20000613
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:6/sas_phy\:phy-5\:6/sas_address
>> 0x50015b2a20000613
>> # cat 
>> /sys/class/sas_host/host5/device/phy-5\:7/sas_phy\:phy-5\:7/sas_address
>> 0x50015b2a20000613
>>
>> And these ones on the 6G:
>>
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:0/sas_phy\:phy-0\:0/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:1/sas_phy\:phy-0\:1/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:2/sas_phy\:phy-0\:2/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:3/sas_phy\:phy-0\:3/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:4/sas_phy\:phy-0\:4/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:5/sas_phy\:phy-0\:5/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:6/sas_phy\:phy-0\:6/sas_address
>> 0x500605b002c99150
>> # cat 
>> /sys/class/sas_host/host0/device/phy-0\:7/sas_phy\:phy-0\:7/sas_address
>> 0x500605b002c99150
>>
>> As we can see above, the 3G HBA's phy "sas_address" sysfs entry 
>> displays correct
>> information: A single SAS address for phy 0-3 and a single one for 
>> phy 4-7.
>>
>> As we can also see using LSIUtil, each phy is numbered from a base 
>> address and
>> incremented.
>> Also, the port sas address corresponds to the address of the starting 
>> phy of the
>> port.
>> Port0's sas address is "0x50015b2a2000060f", which corresponds to the 
>> HBA's phy0
>> sas address.
>> Port1's sas address is "0x50015b2a20000613" which corresponds to the 
>> HBA's phy3
>> sas address, the starting phys for each of these ports.
>>
>> Output of menu "16" from LSIUtil on the 3G HBA (1068e):
>>
>> B___T SASAddress PhyNum Handle Parent Type
>> 50015b2a2000060f 0001 SAS Initiator
>> 50015b2a20000610 0002 SAS Initiator
>> 50015b2a20000611 0003 SAS Initiator
>> 50015b2a20000612 0004 SAS Initiator
>> 50015b2a20000613 0005 SAS Initiator
>> 50015b2a20000614 0006 SAS Initiator
>> 50015b2a20000615 0007 SAS Initiator
>> 50015b2a20000616 0008 SAS Initiator
>>
>> However, when looking at the 6G HBA's sysfs information above, we can 
>> see that
>> all phy's SAS address are identical, like if there was a single port 
>> made up of
>> the entire 8 phys from the HBA.
>>
>> When looking at LSIUtil below, we get more strangeness, where all 
>> phys have
>> exactly the same SAS address:
>>
>> Output of menu "16" from LSIUtil on the 6G HBA (2008):
>>
>> B___T SASAddress PhyNum Handle Parent Type
>> 500605b002c99150 0001 SAS Initiator
>> 500605b002c99150 0002 SAS Initiator
>> 500605b002c99150 0003 SAS Initiator
>> 500605b002c99150 0004 SAS Initiator
>> 500605b002c99150 0005 SAS Initiator
>> 500605b002c99150 0006 SAS Initiator
>> 500605b002c99150 0007 SAS Initiator
>> 500605b002c99150 0008 SAS Initiator
>>
>> But looking at LSIUtil's menu "13" on the 6G HBA proves that we 
>> indeed have 2
>> ports on that HBA:
>>
>> PhyNum Link MinRate MaxRate Initiator Target Port
>> 0 Enabled 1.5 6.0 Enabled Disabled 0
>> 1 Enabled 1.5 6.0 Enabled Disabled 0
>> 2 Enabled 1.5 6.0 Enabled Disabled 0
>> 3 Enabled 1.5 6.0 Enabled Disabled 0
>> 4 Enabled 1.5 6.0 Enabled Disabled 1
>> 5 Enabled 1.5 6.0 Enabled Disabled 1
>> 6 Enabled 1.5 6.0 Enabled Disabled 1
>> 7 Enabled 1.5 6.0 Enabled Disabled 1
>>
>> Somehow, on the 6G HBA using mpt2sas, all phys from a HBA seem to 
>> have the same
>> SAS address, and all ports on that HBA, whether narrow (1Phy) or wide 
>> (4Phys),
>> will seemingly have the same SAS address.
>>
>> This is causing a good few issues with our system scripting, as we 
>> relied on SAS
>> addresses to identify ports.
>>
>> I searched on Google extensively before posting and couldn't find any 
>> mention of
>> this issue.
>>
>> Is this a known issue? If so, will this be resolved in a later 
>> version of the
>> driver?
>
> Ben,
> I noticed something similar when comparing LSI 3 Gbps and
> 6 Gbps SAS HBAs. I was looking at the same thing but from
> an expander's perspective. I'm not so sure that the newer
> 6 Gbps HBAs are incorrect. IOW the older 3 Gbps HBAs might
> be playing some tricks with the HBA's SAS addresses so that
> it is not possible to set up a wide link spanning the
> lower and upper 4-phy banks (e.g. a 5 phy wide link).
>
Hi Doug,

Thanks a lot for your reply!

It is very possible indeed that this was done so that nobody would ever 
try to have an 8phy SAS port (possible because it couldn't handle it?), 
or as you said, a port spanning on 5Phys or spanning phys from two 
different physical ports.

Because we worked mainly with these HBAs (the 3G ones), we assumed that 
this was the norm in SAS: one port is issued a unique SAS address. We 
therefore assume that a port always has a unique SAS address and work 
that way, which, I find today, is probably a wrong assertion.
> Do you know a reason why it is not preferably for every
> phy on a SAS HBA to respond with the same SAS address?
>
No real reason here, we just were working with a wrong assumption (that 
port sas addresses had to be unique) before. We need to update our 
scripts to work with ports having the same SAS address.
>
> As a practical matter a SAS HBA needs a single SAS address,
> preferably printed on the board or its box. Then if you
> manage to wipe its SAS address (e.g. by erasing its flash
> to move from IR to IT firmware) then you know which SAS
> address to re-instate :-)
>
LSIUtil's "Set new WWN/SAS Address" menu (and even on 6G HBA!) will 
request the input of a single sas address, and will set each sas phy's 
address differently by incrementing the user-inputed sas address. This 
could also be happening because LSIUtil might not have been updated 
fully yet.

In fact, LSIUtil and the MPT Fusion driver used to find a HBA's sas 
address by doing something in the like of ioc->port->phy[0]->sas_address 
(low and high), rather than reading a single ioc->sas_address value.

Thanks again for your reply, you helped me clarify some bits here.
> Doug Gilbert
>
Regards,
Ben.

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Ravi Shankar]

>
>
> Do you know a reason why it is not preferably for every
> phy on a SAS HBA to respond with the same SAS address?
>
>
> As a practical matter a SAS HBA needs a single SAS address,
> preferably printed on the board or its box. Then if you
> manage to wipe its SAS address (e.g. by erasing its flash
> to move from IR to IT firmware) then you know which SAS
> address to re-instate :-)
>
HBA SAS phy could have same SAS address when they are directly connected.
however when connected to expanders, each logical port/phy need unique SAS
address.

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Douglas Gilbert]

On 11-08-18 01:57 PM, Ravi Shankar wrote:
>
>>
>>
>> Do you know a reason why it is not preferably for every
>> phy on a SAS HBA to respond with the same SAS address?
>>
>>
>> As a practical matter a SAS HBA needs a single SAS address,
>> preferably printed on the board or its box. Then if you
>> manage to wipe its SAS address (e.g. by erasing its flash
>> to move from IR to IT firmware) then you know which SAS
>> address to re-instate :-)
>>
> HBA SAS phy could have same SAS address when they are directly connected.
> however when connected to expanders, each logical port/phy need unique SAS
> address.

No.

SAS HBAs and expanders should always be trying to maximize
the width of a link. By definition all physical ** phys on an
expander should (must) have the same SAS address. So if
you connect 5 phys from a SAS HBA to the same expander (and
the HBA supports links wider than 4 phys) then those
5 HBA phys should have the same SAS address. Those 5 HBA
phys then form a SCSI port.

Just tested a triangular arrangement: a LSI SAS-2 HBA
(9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
and one SAS-1 expander (narrow link). And the expanders
where connected to each other. Two disks were connected
to the SAS-2 expander.
Both expanders reported the same SAS address for the
5 HBA attached phys. You might argue that is two
separate SCSI initiator ports with the same port
identifier (SAS address) in the same SAS domain.

Anyway there was an interesting difference between the HBA's
BIOS and Linux (lk 3.0.3): the BIOS reported those two
disks twice while Linux only reported them once. That seems
to suggest that the BIOS set up the routing table in the
SAS-1 expander while Linux did not. smp_discover in Linux
confirms that the SAS-1 expander's route table was not set up.

The trouble with testing is that is raises more questions
than it supplies answers.


SAS disks have two phys which are typically given two
different SAS addresses. This stops them forming a wide link
if, for example, they were both connected to the same
expander. Typically the two SAS disk phys would be connected
to different expanders for redundancy. However if the
interest was speed (e.g. with a SAS SSD) then both the
disk phys might be given the same SAS address.


** SAS-2 expanders often have an integrated SES target
    on a virtual phy in the expander chip, and that
    virtual phy has a different SAS address.

Doug Gilbert


P.S. Why is my post cc-ing to "unlisted-recipients:;"

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD]

On 18/08/11 20:52, Douglas Gilbert wrote:
> On 11-08-18 01:57 PM, Ravi Shankar wrote:
>>
>>>
>>>
>>> Do you know a reason why it is not preferably for every
>>> phy on a SAS HBA to respond with the same SAS address?
>>>
>>>
>>> As a practical matter a SAS HBA needs a single SAS address,
>>> preferably printed on the board or its box. Then if you
>>> manage to wipe its SAS address (e.g. by erasing its flash
>>> to move from IR to IT firmware) then you know which SAS
>>> address to re-instate :-)
>>>
>> HBA SAS phy could have same SAS address when they are directly 
>> connected.
>> however when connected to expanders, each logical port/phy need 
>> unique SAS
>> address.
>
> No.
>
> SAS HBAs and expanders should always be trying to maximize
> the width of a link. By definition all physical ** phys on an
> expander should (must) have the same SAS address. So if
> you connect 5 phys from a SAS HBA to the same expander (and
> the HBA supports links wider than 4 phys) then those
> 5 HBA phys should have the same SAS address. Those 5 HBA
> phys then form a SCSI port.
>
> Just tested a triangular arrangement: a LSI SAS-2 HBA
> (9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
> and one SAS-1 expander (narrow link). And the expanders
> where connected to each other. Two disks were connected
> to the SAS-2 expander.
> Both expanders reported the same SAS address for the
> 5 HBA attached phys. You might argue that is two
> separate SCSI initiator ports with the same port
> identifier (SAS address) in the same SAS domain.
>
> Anyway there was an interesting difference between the HBA's
> BIOS and Linux (lk 3.0.3): the BIOS reported those two
> disks twice while Linux only reported them once. That seems
> to suggest that the BIOS set up the routing table in the
> SAS-1 expander while Linux did not. smp_discover in Linux
> confirms that the SAS-1 expander's route table was not set up.
>
> The trouble with testing is that is raises more questions
> than it supplies answers.
>
>
> SAS disks have two phys which are typically given two
> different SAS addresses. This stops them forming a wide link
> if, for example, they were both connected to the same
> expander. Typically the two SAS disk phys would be connected
> to different expanders for redundancy. However if the
> interest was speed (e.g. with a SAS SSD) then both the
> disk phys might be given the same SAS address.
>
>
> ** SAS-2 expanders often have an integrated SES target
>    on a virtual phy in the expander chip, and that
>    virtual phy has a different SAS address.
>
> Doug Gilbert
>
>
> P.S. Why is my post cc-ing to "unlisted-recipients:;"
>

Hi Douglas, Ravi,

According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on 
page 45, they state that that a port is formed by a unique tuple of the 
SAS Phy address and the attached SAS Phy address.

For instance, if you take 2 * 2 phy wide ports, where all 4 phys from 
these two ports have the same sas address, let's call it "A" and connect 
them each to another port that each has a different address, "B" and 
"C", they state that two ports will be formed, one connecting "A" to "B" 
and one connecting "A" to "C".

This is what Douglas is saying with the SAS disks for instance, that are 
typically given two separate SAS addresses to avoid forming a wide port 
with the expander (since the expander will have the same sas address on 
all phys), and to allow for dual expander multiplexing for redundancy.

But what I don't understand is that, in the context of two HBAs 
connected together, things seem to be different:

I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
addresses from being the same to being incremental, therefore the last 
byte of each SAS phy address changed from:

0   1   2   3   4   5   6   7
b0 b0 b0 b0 b0 b0 b0 b0
to:
b0 b1 b2 b3 b4 b5 b6 b7

I also changed the "ports" setup from "Auto" to "Wide", making two 
4*phys ports:

     Port 0       |      Port 1
b0 b1 b2 b3 | b4 b5 b6 b7

I also set all these ports to Target.

I then connected this HBA to another 9200-8e HBA, which was left setup 
as default:

               Auto
               Initiator
0   1   2   3   4   5   6   7
10 10 10 10 10 10 10 10

However, when I looked up the SAS topology on either side in LSIUtil, I 
saw that there was two ports connected on each HBAs, one connected on 
phy 0 and one on phy 4.

On the second (Initiator) HBA, the two ports appeared as b0 and b4, with 
two separate handles.

On the first (Target) HBA, both ports appeared as 10, with two separate 
handles.

What I don't understand above, is since all phys on the Target HBAs have 
a different SAS address, and all the ones on the Initiator one have the 
same, 8 narrow ports should have been created there.

However, there is a separate notion of "port" in LSIUtil, does that mean 
that agglomerating 4 phys with different SAS addresses in a logical 
LSIUtil "port" forces the HBA FW to transmit the same sas address on 
these 4 Phys, to make them look like a single port? Or is there an extra 
separate notion of "port", that does not rely on the phy SAS address and 
its attached SAS address?

I guess my question is: Is there an extra information ontop of phy sas 
address and phy id that is transmitted in SAS, like a "port" id or a handle?

Also, in the above case, if we assume that the HBA FW was transmitting 
the same phys for phy 0-3 and phy 4-8 on the Target HBA, it would make 
sense that we have two ports, since there is two pairs of SAS addresses 
/ attached SAS addresses here.

Am I holding on to the right belief?

Thanks in advance for your reply.

Regards,
Ben.

PS: The "Undisclosed recipient" was because Ravi seemed to have replied 
with a hidden "To:" and CC'ed to the linux-scsi ML.
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Douglas Gilbert]

On 11-08-19 08:30 AM, Benjamin ESTRABAUD wrote:
> On 18/08/11 20:52, Douglas Gilbert wrote:
>> On 11-08-18 01:57 PM, Ravi Shankar wrote:
>>>
>>>>
>>>>
>>>> Do you know a reason why it is not preferably for every
>>>> phy on a SAS HBA to respond with the same SAS address?
>>>>
>>>>
>>>> As a practical matter a SAS HBA needs a single SAS address,
>>>> preferably printed on the board or its box. Then if you
>>>> manage to wipe its SAS address (e.g. by erasing its flash
>>>> to move from IR to IT firmware) then you know which SAS
>>>> address to re-instate :-)
>>>>
>>> HBA SAS phy could have same SAS address when they are directly connected.
>>> however when connected to expanders, each logical port/phy need unique SAS
>>> address.
>>
>> No.
>>
>> SAS HBAs and expanders should always be trying to maximize
>> the width of a link. By definition all physical ** phys on an
>> expander should (must) have the same SAS address. So if
>> you connect 5 phys from a SAS HBA to the same expander (and
>> the HBA supports links wider than 4 phys) then those
>> 5 HBA phys should have the same SAS address. Those 5 HBA
>> phys then form a SCSI port.
>>
>> Just tested a triangular arrangement: a LSI SAS-2 HBA
>> (9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
>> and one SAS-1 expander (narrow link). And the expanders
>> where connected to each other. Two disks were connected
>> to the SAS-2 expander.
>> Both expanders reported the same SAS address for the
>> 5 HBA attached phys. You might argue that is two
>> separate SCSI initiator ports with the same port
>> identifier (SAS address) in the same SAS domain.
>>
>> Anyway there was an interesting difference between the HBA's
>> BIOS and Linux (lk 3.0.3): the BIOS reported those two
>> disks twice while Linux only reported them once. That seems
>> to suggest that the BIOS set up the routing table in the
>> SAS-1 expander while Linux did not. smp_discover in Linux
>> confirms that the SAS-1 expander's route table was not set up.
>>
>> The trouble with testing is that is raises more questions
>> than it supplies answers.
>>
>>
>> SAS disks have two phys which are typically given two
>> different SAS addresses. This stops them forming a wide link
>> if, for example, they were both connected to the same
>> expander. Typically the two SAS disk phys would be connected
>> to different expanders for redundancy. However if the
>> interest was speed (e.g. with a SAS SSD) then both the
>> disk phys might be given the same SAS address.
>>
>>
>> ** SAS-2 expanders often have an integrated SES target
>> on a virtual phy in the expander chip, and that
>> virtual phy has a different SAS address.
>>
>> Doug Gilbert
>>
>>
>> P.S. Why is my post cc-ing to "unlisted-recipients:;"
>>
>
> Hi Douglas, Ravi,
>
> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on page 45,
> they state that that a port is formed by a unique tuple of the SAS Phy address
> and the attached SAS Phy address.
>
> For instance, if you take 2 * 2 phy wide ports, where all 4 phys from these two
> ports have the same sas address, let's call it "A" and connect them each to
> another port that each has a different address, "B" and "C", they state that two
> ports will be formed, one connecting "A" to "B" and one connecting "A" to "C".
>
> This is what Douglas is saying with the SAS disks for instance, that are
> typically given two separate SAS addresses to avoid forming a wide port with the
> expander (since the expander will have the same sas address on all phys), and to
> allow for dual expander multiplexing for redundancy.
>
> But what I don't understand is that, in the context of two HBAs connected
> together, things seem to be different:
>
> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys addresses from
> being the same to being incremental, therefore the last byte of each SAS phy
> address changed from:
>
> 0 1 2 3 4 5 6 7
> b0 b0 b0 b0 b0 b0 b0 b0
> to:
> b0 b1 b2 b3 b4 b5 b6 b7
>
> I also changed the "ports" setup from "Auto" to "Wide", making two 4*phys ports:
>
> Port 0 | Port 1
> b0 b1 b2 b3 | b4 b5 b6 b7
>
> I also set all these ports to Target.
>
> I then connected this HBA to another 9200-8e HBA, which was left setup as default:
>
> Auto
> Initiator
> 0 1 2 3 4 5 6 7
> 10 10 10 10 10 10 10 10
>
> However, when I looked up the SAS topology on either side in LSIUtil, I saw that
> there was two ports connected on each HBAs, one connected on phy 0 and one on
> phy 4.
>
> On the second (Initiator) HBA, the two ports appeared as b0 and b4, with two
> separate handles.
>
> On the first (Target) HBA, both ports appeared as 10, with two separate handles.
>
> What I don't understand above, is since all phys on the Target HBAs have a
> different SAS address, and all the ones on the Initiator one have the same, 8
> narrow ports should have been created there.
>
> However, there is a separate notion of "port" in LSIUtil, does that mean that
> agglomerating 4 phys with different SAS addresses in a logical LSIUtil "port"
> forces the HBA FW to transmit the same sas address on these 4 Phys, to make them
> look like a single port? Or is there an extra separate notion of "port", that
> does not rely on the phy SAS address and its attached SAS address?
>
> I guess my question is: Is there an extra information ontop of phy sas address
> and phy id that is transmitted in SAS, like a "port" id or a handle?

Ben,
In my previous post I assumed that the SAS HBA was a SCSI
initiator.

And yes, I also felt that LSI has its own meaning for "port"
and that is not the same as a "SCSI port" or a "SAS port"
defined in the t10 documents.

There is another piece of the identification puzzle but
as far as I can see its has not been properly implemented.
Namely: device name. spl2r02.pdf section 4.2.6:

   "Each expander device and SAS device shall include a
    SAS address (see 4.2.4) as its device name.
    A SAS address used as a device name shall not be used
    as any other name or identifier (e.g., a device name,
    port name, port identifier, or logical unit name (see
    SAM-5)), except:
      a) the SAS address of an expander device is the same
         as the SAS address of the SMP port in that expander
         device."

SAS disks comply with this but not SAS initiators (at least
not the ones that I have checked). So if a SAS HBA in
initiator mode is to be regarded as a SCSI device then
it should have a device name which is a SAS address that
is distinct from the SAS addresses of any of its ports.


BTW Why not put an expander between the HBAs set up as
an initiator and a target? That way you can check (via
SMP) that LSIUtil is reporting the same thing as the
phy does "down the wire". The SMP DISCOVER response
reports the attached device name.

Doug Gilbert

> Also, in the above case, if we assume that the HBA FW was transmitting the same
> phys for phy 0-3 and phy 4-8 on the Target HBA, it would make sense that we have
> two ports, since there is two pairs of SAS addresses / attached SAS addresses here.
>
> Am I holding on to the right belief?
>
> Thanks in advance for your reply.
>
> Regards,
> Ben.
>
> PS: The "Undisclosed recipient" was because Ravi seemed to have replied with a
> hidden "To:" and CC'ed to the linux-scsi ML.
>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD]

On 19/08/11 15:58, Douglas Gilbert wrote:
> On 11-08-19 08:30 AM, Benjamin ESTRABAUD wrote:
>> On 18/08/11 20:52, Douglas Gilbert wrote:
>>> On 11-08-18 01:57 PM, Ravi Shankar wrote:
>>>>
>>>>>
>>>>>
>>>>> Do you know a reason why it is not preferably for every
>>>>> phy on a SAS HBA to respond with the same SAS address?
>>>>>
>>>>>
>>>>> As a practical matter a SAS HBA needs a single SAS address,
>>>>> preferably printed on the board or its box. Then if you
>>>>> manage to wipe its SAS address (e.g. by erasing its flash
>>>>> to move from IR to IT firmware) then you know which SAS
>>>>> address to re-instate :-)
>>>>>
>>>> HBA SAS phy could have same SAS address when they are directly 
>>>> connected.
>>>> however when connected to expanders, each logical port/phy need 
>>>> unique SAS
>>>> address.
>>>
>>> No.
>>>
>>> SAS HBAs and expanders should always be trying to maximize
>>> the width of a link. By definition all physical ** phys on an
>>> expander should (must) have the same SAS address. So if
>>> you connect 5 phys from a SAS HBA to the same expander (and
>>> the HBA supports links wider than 4 phys) then those
>>> 5 HBA phys should have the same SAS address. Those 5 HBA
>>> phys then form a SCSI port.
>>>
>>> Just tested a triangular arrangement: a LSI SAS-2 HBA
>>> (9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
>>> and one SAS-1 expander (narrow link). And the expanders
>>> where connected to each other. Two disks were connected
>>> to the SAS-2 expander.
>>> Both expanders reported the same SAS address for the
>>> 5 HBA attached phys. You might argue that is two
>>> separate SCSI initiator ports with the same port
>>> identifier (SAS address) in the same SAS domain.
>>>
>>> Anyway there was an interesting difference between the HBA's
>>> BIOS and Linux (lk 3.0.3): the BIOS reported those two
>>> disks twice while Linux only reported them once. That seems
>>> to suggest that the BIOS set up the routing table in the
>>> SAS-1 expander while Linux did not. smp_discover in Linux
>>> confirms that the SAS-1 expander's route table was not set up.
>>>
>>> The trouble with testing is that is raises more questions
>>> than it supplies answers.
>>>
>>>
>>> SAS disks have two phys which are typically given two
>>> different SAS addresses. This stops them forming a wide link
>>> if, for example, they were both connected to the same
>>> expander. Typically the two SAS disk phys would be connected
>>> to different expanders for redundancy. However if the
>>> interest was speed (e.g. with a SAS SSD) then both the
>>> disk phys might be given the same SAS address.
>>>
>>>
>>> ** SAS-2 expanders often have an integrated SES target
>>> on a virtual phy in the expander chip, and that
>>> virtual phy has a different SAS address.
>>>
>>> Doug Gilbert
>>>
>>>
>>> P.S. Why is my post cc-ing to "unlisted-recipients:;"
>>>
>>
>> Hi Douglas, Ravi,
>>
>> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on 
>> page 45,
>> they state that that a port is formed by a unique tuple of the SAS 
>> Phy address
>> and the attached SAS Phy address.
>>
>> For instance, if you take 2 * 2 phy wide ports, where all 4 phys from 
>> these two
>> ports have the same sas address, let's call it "A" and connect them 
>> each to
>> another port that each has a different address, "B" and "C", they 
>> state that two
>> ports will be formed, one connecting "A" to "B" and one connecting 
>> "A" to "C".
>>
>> This is what Douglas is saying with the SAS disks for instance, that are
>> typically given two separate SAS addresses to avoid forming a wide 
>> port with the
>> expander (since the expander will have the same sas address on all 
>> phys), and to
>> allow for dual expander multiplexing for redundancy.
>>
>> But what I don't understand is that, in the context of two HBAs 
>> connected
>> together, things seem to be different:
>>
>> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
>> addresses from
>> being the same to being incremental, therefore the last byte of each 
>> SAS phy
>> address changed from:
>>
>> 0 1 2 3 4 5 6 7
>> b0 b0 b0 b0 b0 b0 b0 b0
>> to:
>> b0 b1 b2 b3 b4 b5 b6 b7
>>
>> I also changed the "ports" setup from "Auto" to "Wide", making two 
>> 4*phys ports:
>>
>> Port 0 | Port 1
>> b0 b1 b2 b3 | b4 b5 b6 b7
>>
>> I also set all these ports to Target.
>>
>> I then connected this HBA to another 9200-8e HBA, which was left 
>> setup as default:
>>
>> Auto
>> Initiator
>> 0 1 2 3 4 5 6 7
>> 10 10 10 10 10 10 10 10
>>
>> However, when I looked up the SAS topology on either side in LSIUtil, 
>> I saw that
>> there was two ports connected on each HBAs, one connected on phy 0 
>> and one on
>> phy 4.
>>
>> On the second (Initiator) HBA, the two ports appeared as b0 and b4, 
>> with two
>> separate handles.
>>
>> On the first (Target) HBA, both ports appeared as 10, with two 
>> separate handles.
>>
>> What I don't understand above, is since all phys on the Target HBAs 
>> have a
>> different SAS address, and all the ones on the Initiator one have the 
>> same, 8
>> narrow ports should have been created there.
>>
>> However, there is a separate notion of "port" in LSIUtil, does that 
>> mean that
>> agglomerating 4 phys with different SAS addresses in a logical 
>> LSIUtil "port"
>> forces the HBA FW to transmit the same sas address on these 4 Phys, 
>> to make them
>> look like a single port? Or is there an extra separate notion of 
>> "port", that
>> does not rely on the phy SAS address and its attached SAS address?
>>
>> I guess my question is: Is there an extra information ontop of phy 
>> sas address
>> and phy id that is transmitted in SAS, like a "port" id or a handle?
>
> Ben,
> In my previous post I assumed that the SAS HBA was a SCSI
> initiator.
>
> And yes, I also felt that LSI has its own meaning for "port"
> and that is not the same as a "SCSI port" or a "SAS port"
> defined in the t10 documents.
>
> There is another piece of the identification puzzle but
> as far as I can see its has not been properly implemented.
> Namely: device name. spl2r02.pdf section 4.2.6:
>
>   "Each expander device and SAS device shall include a
>    SAS address (see 4.2.4) as its device name.
>    A SAS address used as a device name shall not be used
>    as any other name or identifier (e.g., a device name,
>    port name, port identifier, or logical unit name (see
>    SAM-5)), except:
>      a) the SAS address of an expander device is the same
>         as the SAS address of the SMP port in that expander
>         device."
>
> SAS disks comply with this but not SAS initiators (at least
> not the ones that I have checked). So if a SAS HBA in
> initiator mode is to be regarded as a SCSI device then
> it should have a device name which is a SAS address that
> is distinct from the SAS addresses of any of its ports.
Yes, I observed the same thing with all the initiators I have.
It looks like the phys on LSI HBAs are set by reading the SAS ioc or 
that the SAS ioc sas address is read by reading the first phy's sas address.
>
>
> BTW Why not put an expander between the HBAs set up as
> an initiator and a target? That way you can check (via
> SMP) that LSIUtil is reporting the same thing as the
> phy does "down the wire". The SMP DISCOVER response
> reports the attached device name.
>
That's a good idea,

I'll try if possible.


Thanks for your reply.

Regards,
Ben.

> Doug Gilbert
>
>> Also, in the above case, if we assume that the HBA FW was 
>> transmitting the same
>> phys for phy 0-3 and phy 4-8 on the Target HBA, it would make sense 
>> that we have
>> two ports, since there is two pairs of SAS addresses / attached SAS 
>> addresses here.
>>
>> Am I holding on to the right belief?
>>
>> Thanks in advance for your reply.
>>
>> Regards,
>> Ben.
>>
>> PS: The "Undisclosed recipient" was because Ravi seemed to have 
>> replied with a
>> hidden "To:" and CC'ed to the linux-scsi ML.
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>
>

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Ravi Shankar]

>
> Hi Douglas, Ravi,
>
> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on 
> page 45, they state that that a port is formed by a unique tuple of 
> the SAS Phy address and the attached SAS Phy address.
>
> For instance, if you take 2 * 2 phy wide ports, where all 4 phys from 
> these two ports have the same sas address, let's call it "A" and 
> connect them each to another port that each has a different address, 
> "B" and "C", they state that two ports will be formed, one connecting 
> "A" to "B" and one connecting "A" to "C".
>
> This is what Douglas is saying with the SAS disks for instance, that 
> are typically given two separate SAS addresses to avoid forming a wide 
> port with the expander (since the expander will have the same sas 
> address on all phys), and to allow for dual expander multiplexing for 
> redundancy.
>
> But what I don't understand is that, in the context of two HBAs 
> connected together, things seem to be different:
>
> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
> addresses from being the same to being incremental, therefore the last 
> byte of each SAS phy address changed from:
>
> 0   1   2   3   4   5   6   7
> b0 b0 b0 b0 b0 b0 b0 b0
> to:
> b0 b1 b2 b3 b4 b5 b6 b7
>
> I also changed the "ports" setup from "Auto" to "Wide", making two 
> 4*phys ports:
>
>     Port 0       |      Port 1
> b0 b1 b2 b3 | b4 b5 b6 b7
>
> I also set all these ports to Target.
>
> I then connected this HBA to another 9200-8e HBA, which was left setup 
> as default:
>
>               Auto
>               Initiator
> 0   1   2   3   4   5   6   7
> 10 10 10 10 10 10 10 10
>
> However, when I looked up the SAS topology on either side in LSIUtil, 
> I saw that there was two ports connected on each HBAs, one connected 
> on phy 0 and one on phy 4.
>
> On the second (Initiator) HBA, the two ports appeared as b0 and b4, 
> with two separate handles.
>
> On the first (Target) HBA, both ports appeared as 10, with two 
> separate handles.
>
> What I don't understand above, is since all phys on the Target HBAs 
> have a different SAS address, and all the ones on the Initiator one 
> have the same, 8 narrow ports should have been created there.
>
> However, there is a separate notion of "port" in LSIUtil, does that 
> mean that agglomerating 4 phys with different SAS addresses in a 
> logical LSIUtil "port" forces the HBA FW to transmit the same sas 
> address on these 4 Phys, to make them look like a single port? Or is 
> there an extra separate notion of "port", that does not rely on the 
> phy SAS address and its attached SAS address?
>
> I guess my question is: Is there an extra information ontop of phy sas 
> address and phy id that is transmitted in SAS, like a "port" id or a 
> handle?
>
> Also, in the above case, if we assume that the HBA FW was transmitting 
> the same phys for phy 0-3 and phy 4-8 on the Target HBA, it would make 
> sense that we have two ports, since there is two pairs of SAS 
> addresses / attached SAS addresses here.
>
Ben,

Port 0       |      Port 1
b0 b1 b2 b3 | b4 b5 b6 b7

In above configuration you are assigning different SAS Address for each 
PHY but over riding with WIDE port clause. After
individual PHY are reset, it transmits IDENTIFICATION address frames as 
part of identification sequence so down stream devices
know the attributes of the attached devices.

PHY 0-3: Transmit Identification frame with SAS address xxxxxxxxb0

PHY 4-7: Transmit identification frame with SAS address xxxxxxxxb4

The second HBA (with SAS address xxxxxxxx10 with Auto mode) receives 
above identification frame on PHY 0-3 and 4-7 respectively. So this
essentially forms x4 wide port instead of narrow as you expected.

As far I know there are no port id or handle transmitted on the fabric.

Couple of interesting question regarding wide port. From bandwidth 
perspective, x4 port are termed as 24 Gb/sec ( 6 Gb/s * 4). But do we
really get 24 Gb/sec bandwidth ?. I see questions being raised that SSD 
disks need wide ports for bandwidth aggregation. SAS protocol
and Expander has following limitation which could be problematic 
depending on topology.

1) Unlike FC, SAS is a connection oriented protocol (full duplex Class 1 
vs Class 3 FC)
2) Flow control primitives (K words) are transmitted inside connection 
(without being packetized).
3) When connecting through Expanders, typically only x4 or x8 physical 
links are used. If there are hundreds of Initiator/Target exist in such
     fabric, the number of active I/O transfers across devices are 
limited to number of links between Expanders (due to Class 1 protocol).

My understanding for SSD disks with wide ports the HBA and Disks can 
queue several commands using Tagged Queuing. This way we can
maximize number of  commands and data frames across devices.

Wondering has anyone measured performance under such scenario ?. It 
would be great to see Expanders terminating SSP frames to over come
some of above limitation. Links between HBA and Expander and Expander to 
Disk can be still Class 1.

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Douglas Gilbert]

On 11-08-19 03:06 PM, Ravi Shankar wrote:
>
>>
>> Hi Douglas, Ravi,
>>
>> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on page 45,
>> they state that that a port is formed by a unique tuple of the SAS Phy address
>> and the attached SAS Phy address.
>>
>> For instance, if you take 2 * 2 phy wide ports, where all 4 phys from these
>> two ports have the same sas address, let's call it "A" and connect them each
>> to another port that each has a different address, "B" and "C", they state
>> that two ports will be formed, one connecting "A" to "B" and one connecting
>> "A" to "C".
>>
>> This is what Douglas is saying with the SAS disks for instance, that are
>> typically given two separate SAS addresses to avoid forming a wide port with
>> the expander (since the expander will have the same sas address on all phys),
>> and to allow for dual expander multiplexing for redundancy.
>>
>> But what I don't understand is that, in the context of two HBAs connected
>> together, things seem to be different:
>>
>> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys addresses from
>> being the same to being incremental, therefore the last byte of each SAS phy
>> address changed from:
>>
>> 0 1 2 3 4 5 6 7
>> b0 b0 b0 b0 b0 b0 b0 b0
>> to:
>> b0 b1 b2 b3 b4 b5 b6 b7
>>
>> I also changed the "ports" setup from "Auto" to "Wide", making two 4*phys ports:
>>
>> Port 0 | Port 1
>> b0 b1 b2 b3 | b4 b5 b6 b7
>>
>> I also set all these ports to Target.
>>
>> I then connected this HBA to another 9200-8e HBA, which was left setup as
>> default:
>>
>> Auto
>> Initiator
>> 0 1 2 3 4 5 6 7
>> 10 10 10 10 10 10 10 10
>>
>> However, when I looked up the SAS topology on either side in LSIUtil, I saw
>> that there was two ports connected on each HBAs, one connected on phy 0 and
>> one on phy 4.
>>
>> On the second (Initiator) HBA, the two ports appeared as b0 and b4, with two
>> separate handles.
>>
>> On the first (Target) HBA, both ports appeared as 10, with two separate handles.
>>
>> What I don't understand above, is since all phys on the Target HBAs have a
>> different SAS address, and all the ones on the Initiator one have the same, 8
>> narrow ports should have been created there.
>>
>> However, there is a separate notion of "port" in LSIUtil, does that mean that
>> agglomerating 4 phys with different SAS addresses in a logical LSIUtil "port"
>> forces the HBA FW to transmit the same sas address on these 4 Phys, to make
>> them look like a single port? Or is there an extra separate notion of "port",
>> that does not rely on the phy SAS address and its attached SAS address?
>>
>> I guess my question is: Is there an extra information ontop of phy sas address
>> and phy id that is transmitted in SAS, like a "port" id or a handle?
>>
>> Also, in the above case, if we assume that the HBA FW was transmitting the
>> same phys for phy 0-3 and phy 4-8 on the Target HBA, it would make sense that
>> we have two ports, since there is two pairs of SAS addresses / attached SAS
>> addresses here.
>>
> Ben,
>
> Port 0 | Port 1
> b0 b1 b2 b3 | b4 b5 b6 b7
>
> In above configuration you are assigning different SAS Address for each PHY but
> over riding with WIDE port clause. After
> individual PHY are reset, it transmits IDENTIFICATION address frames as part of
> identification sequence so down stream devices
> know the attributes of the attached devices.
>
> PHY 0-3: Transmit Identification frame with SAS address xxxxxxxxb0
>
> PHY 4-7: Transmit identification frame with SAS address xxxxxxxxb4
>
> The second HBA (with SAS address xxxxxxxx10 with Auto mode) receives above
> identification frame on PHY 0-3 and 4-7 respectively. So this
> essentially forms x4 wide port instead of narrow as you expected.
>
> As far I know there are no port id or handle transmitted on the fabric.
>
> Couple of interesting question regarding wide port. From bandwidth perspective,
> x4 port are termed as 24 Gb/sec ( 6 Gb/s * 4). But do we
> really get 24 Gb/sec bandwidth ?. I see questions being raised that SSD disks
> need wide ports for bandwidth aggregation. SAS protocol
> and Expander has following limitation which could be problematic depending on
> topology.
>
> 1) Unlike FC, SAS is a connection oriented protocol (full duplex Class 1 vs
> Class 3 FC)
> 2) Flow control primitives (K words) are transmitted inside connection (without
> being packetized).
> 3) When connecting through Expanders, typically only x4 or x8 physical links are
> used. If there are hundreds of Initiator/Target exist in such
> fabric, the number of active I/O transfers across devices are limited to number
> of links between Expanders (due to Class 1 protocol).
>
> My understanding for SSD disks with wide ports the HBA and Disks can queue
> several commands using Tagged Queuing. This way we can
> maximize number of commands and data frames across devices.

spl2r02.pdf section 6.18.2 [link layer, SSP, Full duplex]:
   "SSP is a full duplex protocol. An SSP phy may receive
    an SSP frame or primitive in a connection while it is
    transmitting an SSP frame or primitive in the same
    connection. A wide SSP port may send and/or receive
    SSP frames or primitives concurrently on different
    connections (i.e., on different phys)."

For a SCSI command like READ(10) a connection consumes
one initiator phy and one target phy plus the pathway
between them until it is closed. Typically a READ
would have two connections: one to send the CDB and a
second connection later to return the data and response
(SCSI status and possibly sense data). For a spinning
disk there could be milliseconds between those two
connections; with an SSD less (do they use only one
connection?).

Due to the full duplex nature of a connection, DATA
frames associated with a WRITE could overlap with DATA
frames associated with an READ CDB sent earlier.

In SAS-2, a single READ's maximum data rate is 6 Gbps.
If a 2-phy wide link is available (along the whole pathway
(see Figure 129 in spl2r02.pdf)) then two READs, sent one
after the other or concurrently, could have their DATA
frames returned concurrently. So the combined maximum
data rate of the two READs would be 12 Gbps.

Expanders don't change what is stated above. Pathways
become an interconnection of links. A small latency is
added to the opening of connections. And there is the
possibility that no links are available to establish a
connection (e.g. target to expander has available link(s)
but all expander to initiator links are occupied).

> Wondering has anyone measured performance under such scenario ?. It would be
> great to see Expanders terminating SSP frames to over come
> some of above limitation. Links between HBA and Expander and Expander to Disk
> can be still Class 1.

Not sure I follow. Expanders come into play when
connections are being established.

Doug Gilbert

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Ravi Shankar]

>
> spl2r02.pdf section 6.18.2 [link layer, SSP, Full duplex]:
>   "SSP is a full duplex protocol. An SSP phy may receive
>    an SSP frame or primitive in a connection while it is
>    transmitting an SSP frame or primitive in the same
>    connection. A wide SSP port may send and/or receive
>    SSP frames or primitives concurrently on different
>    connections (i.e., on different phys)."
>
> For a SCSI command like READ(10) a connection consumes
> one initiator phy and one target phy plus the pathway
> between them until it is closed. Typically a READ
> would have two connections: one to send the CDB and a
> second connection later to return the data and response
> (SCSI status and possibly sense data). For a spinning
> disk there could be milliseconds between those two
> connections; with an SSD less (do they use only one
> connection?).
>
> Due to the full duplex nature of a connection, DATA
> frames associated with a WRITE could overlap with DATA
> frames associated with an READ CDB sent earlier.
>
> In SAS-2, a single READ's maximum data rate is 6 Gbps.
> If a 2-phy wide link is available (along the whole pathway
> (see Figure 129 in spl2r02.pdf)) then two READs, sent one
> after the other or concurrently, could have their DATA
> frames returned concurrently. So the combined maximum
> data rate of the two READs would be 12 Gbps.
>
> Expanders don't change what is stated above. Pathways
> become an interconnection of links. A small latency is
> added to the opening of connections. And there is the
> possibility that no links are available to establish a
> connection (e.g. target to expander has available link(s)
> but all expander to initiator links are occupied).
>
>> Wondering has anyone measured performance under such scenario ?. It 
>> would be
>> great to see Expanders terminating SSP frames to over come
>> some of above limitation. Links between HBA and Expander and Expander 
>> to Disk
>> can be still Class 1.
>
> Not sure I follow. Expanders come into play when
> connections are being established.
>
If you've single Expander with multiple initiators and disks then there 
are no issue unless multiple initiators/targets
trying to access to same resources. There is no QoS (other than Path way 
count and Disconnect/Reconnect mode page)
enforced by the expander. Once you start daisy chaining expanders 
pathway between expanders will be bottle neck due
to connection oriented nature of SAS protocol. 

Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD]

On 21/08/11 02:05, Douglas Gilbert wrote:
> On 11-08-19 03:06 PM, Ravi Shankar wrote:
>>
>>>
>>> Hi Douglas, Ravi,
>>>
>>> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), 
>>> on page 45,
>>> they state that that a port is formed by a unique tuple of the SAS 
>>> Phy address
>>> and the attached SAS Phy address.
>>>
>>> For instance, if you take 2 * 2 phy wide ports, where all 4 phys 
>>> from these
>>> two ports have the same sas address, let's call it "A" and connect 
>>> them each
>>> to another port that each has a different address, "B" and "C", they 
>>> state
>>> that two ports will be formed, one connecting "A" to "B" and one 
>>> connecting
>>> "A" to "C".
>>>
>>> This is what Douglas is saying with the SAS disks for instance, that 
>>> are
>>> typically given two separate SAS addresses to avoid forming a wide 
>>> port with
>>> the expander (since the expander will have the same sas address on 
>>> all phys),
>>> and to allow for dual expander multiplexing for redundancy.
>>>
>>> But what I don't understand is that, in the context of two HBAs 
>>> connected
>>> together, things seem to be different:
>>>
>>> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
>>> addresses from
>>> being the same to being incremental, therefore the last byte of each 
>>> SAS phy
>>> address changed from:
>>>
>>> 0 1 2 3 4 5 6 7
>>> b0 b0 b0 b0 b0 b0 b0 b0
>>> to:
>>> b0 b1 b2 b3 b4 b5 b6 b7
>>>
>>> I also changed the "ports" setup from "Auto" to "Wide", making two 
>>> 4*phys ports:
>>>
>>> Port 0 | Port 1
>>> b0 b1 b2 b3 | b4 b5 b6 b7
>>>
>>> I also set all these ports to Target.
>>>
>>> I then connected this HBA to another 9200-8e HBA, which was left 
>>> setup as
>>> default:
>>>
>>> Auto
>>> Initiator
>>> 0 1 2 3 4 5 6 7
>>> 10 10 10 10 10 10 10 10
>>>
>>> However, when I looked up the SAS topology on either side in 
>>> LSIUtil, I saw
>>> that there was two ports connected on each HBAs, one connected on 
>>> phy 0 and
>>> one on phy 4.
>>>
>>> On the second (Initiator) HBA, the two ports appeared as b0 and b4, 
>>> with two
>>> separate handles.
>>>
>>> On the first (Target) HBA, both ports appeared as 10, with two 
>>> separate handles.
>>>
>>> What I don't understand above, is since all phys on the Target HBAs 
>>> have a
>>> different SAS address, and all the ones on the Initiator one have 
>>> the same, 8
>>> narrow ports should have been created there.
>>>
>>> However, there is a separate notion of "port" in LSIUtil, does that 
>>> mean that
>>> agglomerating 4 phys with different SAS addresses in a logical 
>>> LSIUtil "port"
>>> forces the HBA FW to transmit the same sas address on these 4 Phys, 
>>> to make
>>> them look like a single port? Or is there an extra separate notion 
>>> of "port",
>>> that does not rely on the phy SAS address and its attached SAS address?
>>>
>>> I guess my question is: Is there an extra information ontop of phy 
>>> sas address
>>> and phy id that is transmitted in SAS, like a "port" id or a handle?
>>>
>>> Also, in the above case, if we assume that the HBA FW was 
>>> transmitting the
>>> same phys for phy 0-3 and phy 4-8 on the Target HBA, it would make 
>>> sense that
>>> we have two ports, since there is two pairs of SAS addresses / 
>>> attached SAS
>>> addresses here.
>>>
>> Ben,
>>
>> Port 0 | Port 1
>> b0 b1 b2 b3 | b4 b5 b6 b7
>>
>> In above configuration you are assigning different SAS Address for 
>> each PHY but
>> over riding with WIDE port clause. After
>> individual PHY are reset, it transmits IDENTIFICATION address frames 
>> as part of
>> identification sequence so down stream devices
>> know the attributes of the attached devices.
>>
>> PHY 0-3: Transmit Identification frame with SAS address xxxxxxxxb0
>>
>> PHY 4-7: Transmit identification frame with SAS address xxxxxxxxb4
>>
>> The second HBA (with SAS address xxxxxxxx10 with Auto mode) receives 
>> above
>> identification frame on PHY 0-3 and 4-7 respectively. So this
>> essentially forms x4 wide port instead of narrow as you expected.
>>
>> As far I know there are no port id or handle transmitted on the fabric.
>>
>> Couple of interesting question regarding wide port. From bandwidth 
>> perspective,
>> x4 port are termed as 24 Gb/sec ( 6 Gb/s * 4). But do we
>> really get 24 Gb/sec bandwidth ?. I see questions being raised that 
>> SSD disks
>> need wide ports for bandwidth aggregation. SAS protocol
>> and Expander has following limitation which could be problematic 
>> depending on
>> topology.
>>
>> 1) Unlike FC, SAS is a connection oriented protocol (full duplex 
>> Class 1 vs
>> Class 3 FC)
>> 2) Flow control primitives (K words) are transmitted inside 
>> connection (without
>> being packetized).
>> 3) When connecting through Expanders, typically only x4 or x8 
>> physical links are
>> used. If there are hundreds of Initiator/Target exist in such
>> fabric, the number of active I/O transfers across devices are limited 
>> to number
>> of links between Expanders (due to Class 1 protocol).
>>
>> My understanding for SSD disks with wide ports the HBA and Disks can 
>> queue
>> several commands using Tagged Queuing. This way we can
>> maximize number of commands and data frames across devices.
>
> spl2r02.pdf section 6.18.2 [link layer, SSP, Full duplex]:
>   "SSP is a full duplex protocol. An SSP phy may receive
>    an SSP frame or primitive in a connection while it is
>    transmitting an SSP frame or primitive in the same
>    connection. A wide SSP port may send and/or receive
>    SSP frames or primitives concurrently on different
>    connections (i.e., on different phys)."
>
> For a SCSI command like READ(10) a connection consumes
> one initiator phy and one target phy plus the pathway
> between them until it is closed. Typically a READ
> would have two connections: one to send the CDB and a
> second connection later to return the data and response
> (SCSI status and possibly sense data). For a spinning
> disk there could be milliseconds between those two
> connections; with an SSD less (do they use only one
> connection?).
>
> Due to the full duplex nature of a connection, DATA
> frames associated with a WRITE could overlap with DATA
> frames associated with an READ CDB sent earlier.
>
> In SAS-2, a single READ's maximum data rate is 6 Gbps.
> If a 2-phy wide link is available (along the whole pathway
> (see Figure 129 in spl2r02.pdf)) then two READs, sent one
> after the other or concurrently, could have their DATA
> frames returned concurrently. So the combined maximum
> data rate of the two READs would be 12 Gbps.
>
> Expanders don't change what is stated above. Pathways
> become an interconnection of links. A small latency is
> added to the opening of connections. And there is the
> possibility that no links are available to establish a
> connection (e.g. target to expander has available link(s)
> but all expander to initiator links are occupied).
>
Hi,

Thank you both for your replies.

I understand now a bit more how this works and how LSI is making it work.

Regarding the performances, hooking up a 6G HBA to one 6G expander 
hosting a lot of SSDs (maybe 20) using 4phy wide links got us 2000MB/sec 
IO performance, pretty much line speed.

So the performance is achievable.

Regard,
Ben.
>> Wondering has anyone measured performance under such scenario ?. It 
>> would be
>> great to see Expanders terminating SSP frames to over come
>> some of above limitation. Links between HBA and Expander and Expander 
>> to Disk
>> can be still Class 1.
>
> Not sure I follow. Expanders come into play when
> connections are being established.
>
> Doug Gilbert
>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>