Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

[Benjamin ESTRABAUD <be@mpstor.com>]

On 19/08/11 15:58, Douglas Gilbert wrote:
> On 11-08-19 08:30 AM, Benjamin ESTRABAUD wrote:
>> On 18/08/11 20:52, Douglas Gilbert wrote:
>>> On 11-08-18 01:57 PM, Ravi Shankar wrote:
>>>>
>>>>>
>>>>>
>>>>> Do you know a reason why it is not preferably for every
>>>>> phy on a SAS HBA to respond with the same SAS address?
>>>>>
>>>>>
>>>>> As a practical matter a SAS HBA needs a single SAS address,
>>>>> preferably printed on the board or its box. Then if you
>>>>> manage to wipe its SAS address (e.g. by erasing its flash
>>>>> to move from IR to IT firmware) then you know which SAS
>>>>> address to re-instate :-)
>>>>>
>>>> HBA SAS phy could have same SAS address when they are directly 
>>>> connected.
>>>> however when connected to expanders, each logical port/phy need 
>>>> unique SAS
>>>> address.
>>>
>>> No.
>>>
>>> SAS HBAs and expanders should always be trying to maximize
>>> the width of a link. By definition all physical ** phys on an
>>> expander should (must) have the same SAS address. So if
>>> you connect 5 phys from a SAS HBA to the same expander (and
>>> the HBA supports links wider than 4 phys) then those
>>> 5 HBA phys should have the same SAS address. Those 5 HBA
>>> phys then form a SCSI port.
>>>
>>> Just tested a triangular arrangement: a LSI SAS-2 HBA
>>> (9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
>>> and one SAS-1 expander (narrow link). And the expanders
>>> where connected to each other. Two disks were connected
>>> to the SAS-2 expander.
>>> Both expanders reported the same SAS address for the
>>> 5 HBA attached phys. You might argue that is two
>>> separate SCSI initiator ports with the same port
>>> identifier (SAS address) in the same SAS domain.
>>>
>>> Anyway there was an interesting difference between the HBA's
>>> BIOS and Linux (lk 3.0.3): the BIOS reported those two
>>> disks twice while Linux only reported them once. That seems
>>> to suggest that the BIOS set up the routing table in the
>>> SAS-1 expander while Linux did not. smp_discover in Linux
>>> confirms that the SAS-1 expander's route table was not set up.
>>>
>>> The trouble with testing is that is raises more questions
>>> than it supplies answers.
>>>
>>>
>>> SAS disks have two phys which are typically given two
>>> different SAS addresses. This stops them forming a wide link
>>> if, for example, they were both connected to the same
>>> expander. Typically the two SAS disk phys would be connected
>>> to different expanders for redundancy. However if the
>>> interest was speed (e.g. with a SAS SSD) then both the
>>> disk phys might be given the same SAS address.
>>>
>>>
>>> ** SAS-2 expanders often have an integrated SES target
>>> on a virtual phy in the expander chip, and that
>>> virtual phy has a different SAS address.
>>>
>>> Doug Gilbert
>>>
>>>
>>> P.S. Why is my post cc-ing to "unlisted-recipients:;"
>>>
>>
>> Hi Douglas, Ravi,
>>
>> According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on 
>> page 45,
>> they state that that a port is formed by a unique tuple of the SAS 
>> Phy address
>> and the attached SAS Phy address.
>>
>> For instance, if you take 2 * 2 phy wide ports, where all 4 phys from 
>> these two
>> ports have the same sas address, let's call it "A" and connect them 
>> each to
>> another port that each has a different address, "B" and "C", they 
>> state that two
>> ports will be formed, one connecting "A" to "B" and one connecting 
>> "A" to "C".
>>
>> This is what Douglas is saying with the SAS disks for instance, that are
>> typically given two separate SAS addresses to avoid forming a wide 
>> port with the
>> expander (since the expander will have the same sas address on all 
>> phys), and to
>> allow for dual expander multiplexing for redundancy.
>>
>> But what I don't understand is that, in the context of two HBAs 
>> connected
>> together, things seem to be different:
>>
>> I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
>> addresses from
>> being the same to being incremental, therefore the last byte of each 
>> SAS phy
>> address changed from:
>>
>> 0 1 2 3 4 5 6 7
>> b0 b0 b0 b0 b0 b0 b0 b0
>> to:
>> b0 b1 b2 b3 b4 b5 b6 b7
>>
>> I also changed the "ports" setup from "Auto" to "Wide", making two 
>> 4*phys ports:
>>
>> Port 0 | Port 1
>> b0 b1 b2 b3 | b4 b5 b6 b7
>>
>> I also set all these ports to Target.
>>
>> I then connected this HBA to another 9200-8e HBA, which was left 
>> setup as default:
>>
>> Auto
>> Initiator
>> 0 1 2 3 4 5 6 7
>> 10 10 10 10 10 10 10 10
>>
>> However, when I looked up the SAS topology on either side in LSIUtil, 
>> I saw that
>> there was two ports connected on each HBAs, one connected on phy 0 
>> and one on
>> phy 4.
>>
>> On the second (Initiator) HBA, the two ports appeared as b0 and b4, 
>> with two
>> separate handles.
>>
>> On the first (Target) HBA, both ports appeared as 10, with two 
>> separate handles.
>>
>> What I don't understand above, is since all phys on the Target HBAs 
>> have a
>> different SAS address, and all the ones on the Initiator one have the 
>> same, 8
>> narrow ports should have been created there.
>>
>> However, there is a separate notion of "port" in LSIUtil, does that 
>> mean that
>> agglomerating 4 phys with different SAS addresses in a logical 
>> LSIUtil "port"
>> forces the HBA FW to transmit the same sas address on these 4 Phys, 
>> to make them
>> look like a single port? Or is there an extra separate notion of 
>> "port", that
>> does not rely on the phy SAS address and its attached SAS address?
>>
>> I guess my question is: Is there an extra information ontop of phy 
>> sas address
>> and phy id that is transmitted in SAS, like a "port" id or a handle?
>
> Ben,
> In my previous post I assumed that the SAS HBA was a SCSI
> initiator.
>
> And yes, I also felt that LSI has its own meaning for "port"
> and that is not the same as a "SCSI port" or a "SAS port"
> defined in the t10 documents.
>
> There is another piece of the identification puzzle but
> as far as I can see its has not been properly implemented.
> Namely: device name. spl2r02.pdf section 4.2.6:
>
>   "Each expander device and SAS device shall include a
>    SAS address (see 4.2.4) as its device name.
>    A SAS address used as a device name shall not be used
>    as any other name or identifier (e.g., a device name,
>    port name, port identifier, or logical unit name (see
>    SAM-5)), except:
>      a) the SAS address of an expander device is the same
>         as the SAS address of the SMP port in that expander
>         device."
>
> SAS disks comply with this but not SAS initiators (at least
> not the ones that I have checked). So if a SAS HBA in
> initiator mode is to be regarded as a SCSI device then
> it should have a device name which is a SAS address that
> is distinct from the SAS addresses of any of its ports.
Yes, I observed the same thing with all the initiators I have.
It looks like the phys on LSI HBAs are set by reading the SAS ioc or 
that the SAS ioc sas address is read by reading the first phy's sas address.
>
>
> BTW Why not put an expander between the HBAs set up as
> an initiator and a target? That way you can check (via
> SMP) that LSIUtil is reporting the same thing as the
> phy does "down the wire". The SMP DISCOVER response
> reports the attached device name.
>
That's a good idea,

I'll try if possible.


Thanks for your reply.

Regards,
Ben.

> Doug Gilbert
>
>> Also, in the above case, if we assume that the HBA FW was 
>> transmitting the same
>> phys for phy 0-3 and phy 4-8 on the Target HBA, it would make sense 
>> that we have
>> two ports, since there is two pairs of SAS addresses / attached SAS 
>> addresses here.
>>
>> Am I holding on to the right belief?
>>
>> Thanks in advance for your reply.
>>
>> Regards,
>> Ben.
>>
>> PS: The "Undisclosed recipient" was because Ravi seemed to have 
>> replied with a
>> hidden "To:" and CC'ed to the linux-scsi ML.
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>
>