Re: Handling multiple paths to enclosure devices?

[Douglas Gilbert <dgilbert@interlog.com>]

On 11-07-28 04:33 PM, James Bottomley wrote:
> On Thu, 2011-07-28 at 10:05 -0700, Roland Dreier wrote:
>> Hi,
>>
>> I'm seeing an issue with the current design of our enclosure
>> handling.  In a system with a bunch of drives in an enclosure, it's
>> definitely helpful to have a way to go from sdXX to which slot in the
>> enclosure that drive is in, and that's what the symlink
>> /sys/block/sdXX/device/enclosure_device:NN provides.
>>
>> However, in a system with multiple paths to the enclosure, eg an HBA
>> with two external SAS ports, both connected to a SAS expander in a
>> JBOD, ie in lame ASCII graphics, something like:
>>
>>       +-----+                 /-- drv1
>>       |     |       +-----+  /---  .
>>       |     |==SAS==|     |-/----  .
>>       | HBA |       | exp |------  .
>>       |     |==SAS==|     |-\----  .
>>       |     |       +-----+  \---  .
>>       +-----+                 \-- drvN
>
> So this configuration should form a single wide port and thus not
> actually be multiple paths.  However, if you have two HBAs instead of
> one (or a non-SAS HBA), I grant it becomes multipath.

Here are some conflicting results for the single HBA case.
I tested two LSI HBAs separately, each with 8 phys and wired 5
of those phys to a LSI SAS-2 expander.

Here are the results for a SAS-1.1 (3 Gbps) 3444E HBA seen
from a SMP DISCOVER on the expander:
   phy   5:T:attached:[5001517e85c3efe5:00  t(SATA)]  3 Gbps
   phy   7:T:attached:[5000c500215725bd:00  t(SSP)]  6 Gbps
   phy  12:T:attached:[500605b00006f263:03  i(SSP+STP+SMP)]  3 Gbps
   phy  20:T:attached:[500605b00006f264:04  i(SSP+STP+SMP)]  3 Gbps
   phy  21:T:attached:[500605b00006f264:05  i(SSP+STP+SMP)]  3 Gbps
   phy  22:T:attached:[500605b00006f264:06  i(SSP+STP+SMP)]  3 Gbps
   phy  23:T:attached:[500605b00006f264:07  i(SSP+STP+SMP)]  3 Gbps
   phy  24:D:attached:[5001517e85c3effd:00  V i(SSP+SMP) t(SSP)]  6 Gbps

That is two ports:
   - a narrow port [HBA phy 3 to expander phy 12]
     note the different SAS address of HBA phy 3: 500605b00006f263
   - a wide port [HBA phys 4-7 to expander phys 20-23]
Should the HBA report as two hosts?? [It only reports as one host
in my test.]

So if a SAS HBA can't handle a wide port with more than
4 phys, it can just change the SAS addresses on one or more
phys at the HBA end.


Here are the results for a SAS-2 (6 Gbps) 9212 HBA seen
from a SMP DISCOVER on the same expander:
   phy   5:T:attached:[5001517e85c3efe5:00  t(SATA)]  3 Gbps
   phy   7:T:attached:[5000c500215725bd:00  t(SSP)]  6 Gbps
   phy  12:T:attached:[500605b001d0d3e0:04  i(SSP+STP+SMP)]  6 Gbps
   phy  20:T:attached:[500605b001d0d3e0:03  i(SSP+STP+SMP)]  6 Gbps
   phy  21:T:attached:[500605b001d0d3e0:02  i(SSP+STP+SMP)]  6 Gbps
   phy  22:T:attached:[500605b001d0d3e0:01  i(SSP+STP+SMP)]  6 Gbps
   phy  23:T:attached:[500605b001d0d3e0:00  i(SSP+STP+SMP)]  6 Gbps
   phy  24:D:attached:[5001517e85c3effd:00  V i(SSP+SMP) t(SSP)]  6 Gbps

Now that is a single wide port (5 phys wide) since all
5 expander phys have the same SAS address (not shown)
and all 5 HBA phys have the same address.

>> we have two paths to each drive, so each gets two names, sdXX and
>> sdYY.  However, in drivers/misc/enclosure.c, the code only allows one
>> device in each component and so what happens is that sdXX gets
>> discovered, then gets an enclosure_device:NN link, then sdYY is
>> discovered, so sdXX's enclosure_device:NN link is removed and one is
>> added for sdYY.  And so if I want to figure out which enclosure slot
>> sdXX is in, I'm in for a hard time.
>>
>> It would be a simple matter of writing code to allow all the block
>> devices in a slot to link back to that slot -- we would have to be a
>> bit more careful of keeping track of what links exist, but it should
>> be doable.
>>
>> The wrinkle is that there are also /sys/class/enclosure/ZZZ/NN/device
>> symlinks that allow going the other way.  And it's harder to see how
>> to express multiple block devices in one enclosure slot.
>>
>> Thoughts on how to improve our enclosure handling?

IN SAS-2 the SMP DISCOVER response should contain slot information.
Looking at sysfs for my test system:

# lsscsi
[1:0:0:0]    disk    ATA      ST3320620AS      3.AA  /dev/sda
[6:0:0:0]    disk    ATA      ST31000528AS     CC38  /dev/sdb
[6:0:1:0]    disk    SEAGATE  ST32000444SS     0006  /dev/sdc
[6:0:2:0]    enclosu Intel    RES2SV240        0600  -

then fetching the corresponding SAS port addresses:

# lsscsi -t
[1:0:0:0]    disk    sata:                           /dev/sda
[6:0:0:0]    disk    sas:0x5001517e85c3efe5          /dev/sdb
[6:0:1:0]    disk    sas:0x5000c500215725bd          /dev/sdc
[6:0:2:0]    enclosu sas:0x5001517e85c3effd          -

Referring to the SMP DISCOVER response above, it can be seen
that /dev/sdc is connected to expander phy 7. So getting the
long form output for a SMP DISCOVER on phy 7 :

# smp_discover -p 7 /dev/bsg/expander-6\:0
Discover response:
   ...
   attached SAS address: 0x5000c500215725bd
   attached phy identifier: 0
   ...
   device slot number: 255
   device slot group number: 255
   device slot group output connector:

Sadly the value of 255 means not available. YMMV

> My initial thought is that in a multi-path situation, as above, we get
> two enclosures appearing as well (one down each path).  If we
> incorporated the idea of topological subtrees into the identity matching
> code, we'd end up filling each of the enclosures with the path connected
> devices.  That seems to be an easy situation for multi-path drivers to
> sort out and one requiring no alteration of the existing enclosure code
> (except to do the topological subtree search).
>
> How does that sound?

Does this also solve the problem reported a few weeks back
in which a SES logical unit reported duplicate element
descriptor names?

Doug Gilbert