RE: [PATCH] minimal SAS transport class

RE: [PATCH] minimal SAS transport class

[James.Smart]

Ok, so I'm going to have to revert my statement. After perusing the
sas transport, I went to the previous patch, which added pre-init
data to scsi_scan_target calls.  I didn't understand why this was
needed.

In tracking how you were using the patch, it highlighted that you 
were skipping a step.  The api is such that it does not expect remote
SAS ports to be instantiated.  They should have be (just like FC).
Your api is written only to instantiate local initiator SAS ports.
It needs to instantiate remote SAS ports as well. If it does so, the
remote port is the parent, and the pre-init data doesn't need to be
passed around. The remote SAS ports need to also implement consistent
target id bindings, just like FC.

Also, it appears that your api is allowing multiple SAS initiator
ports to SCSI host. This doesn't make sense, unless you are going to
map SAS ports to the scsi "channel" designator in the address.
In FC, there is a 1:1 correlation of the scsi_host to the local FC
port, so there's no need for a local port transport object as it's
simply the transport for the host.  Also, I prefer a distinction
between the local port and remote port as the functionality of each
will be different (e.g. for local port, you'll have statistics, 
more attributes, and more functions that can be performed. The remote
ports are rarely more than a reporting element, so they are less
featured).

-- James



> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org
> [mailto:linux-scsi-owner@vger.kernel.org]On Behalf Of Smart, James
> Sent: Thursday, August 18, 2005 7:58 AM
> To: hch@lst.de; jejb@steeleye.com
> Cc: ltuikov@yahoo.com; Eric.Moore@lsil.com; andrew.patterson@hp.com;
> linux-scsi@vger.kernel.org
> Subject: RE: [PATCH] minimal SAS transport class
> 
> 
> Christoph,
> 
> I like it, and have no real complaints.  
> 
> As familiar as this looks, there were a couple of conventions 
> in the FC
> transport that I thought should have carried over here. Namely, I saw
> not all attributes being the same, thus I created 3 categories of
> attributes:
>   Private:
>     These are attributes fully owned by the transport. The LLDD does
>     not directly access them, or participate in the sysfs calls.
>     LLDD interaction is strictly indirect via transport functions.
>   Fixed:
>     These attributes, once set, are not expected to change. The LLDD
>     does set these values directly, but should only do so at
>     initialization. The sysfs functions will be handled solely by
>     the transport w/ no interaction with the LLDD.
>   Dynamic (Normal):
>     Values can change. Sysfs functions utilize LLDD to get/set values.
> 
> I expect that the SAS transport has much the same categories, and it
> would be beneficial to indicate which category the different 
> attributes
> fall into. This can be as simple as comments, grouping, or 
> name prefixes.
> 
> Also, I see your enums set explicit values. Just a caution. When I did
> the FC transport, there were cases where I specifically did not use
> specification-specific values, as the transport was a subset 
> or extension
> of the spec. This made anyone interacting with the transport 
> realize that
> this area was not just a reincarnation of the spec (plus, spec writers
> sometimes do stupid things).
> 
> -- james s
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] minimal SAS transport class

[Christoph Hellwig]

On Thu, Aug 18, 2005 at 10:02:21AM -0400, James.Smart@Emulex.Com wrote:
> In tracking how you were using the patch, it highlighted that you 
> were skipping a step.  The api is such that it does not expect remote
> SAS ports to be instantiated.  They should have be (just like FC).
> Your api is written only to instantiate local initiator SAS ports.
> It needs to instantiate remote SAS ports as well. If it does so, the
> remote port is the parent, and the pre-init data doesn't need to be
> passed around. The remote SAS ports need to also implement consistent
> target id bindings, just like FC.

I was trying to avoid the need for an rport object, but I'm not yet sure
it's feasible.  We certainly won't put in target-persistency support
similar to FC, that was just a hack to get the existing drivers migrated
without lots of complaints, it's not going in for new transports like
SAS or iSCSI.  What we might need an rport for is to support SMP.  I'm
not yet sure how to do SMP passthrough, but we will need some object
to represent SMP ports.

> Also, it appears that your api is allowing multiple SAS initiator
> ports to SCSI host.

Yes.  I think we will have to do that because all the resource managment
is per chip, and thus we need to treat the different ports as a single
scsi_host so all the queueing logic gets the resource handling right.

> In FC, there is a 1:1 correlation of the scsi_host to the local FC
> port, so there's no need for a local port transport object as it's
> simply the transport for the host.  Also, I prefer a distinction
> between the local port and remote port as the functionality of each
> will be different (e.g. for local port, you'll have statistics, 
> more attributes, and more functions that can be performed. The remote
> ports are rarely more than a reporting element, so they are less
> featured).

The port object in the patch is purely a local port, the remove port
attributes are in the target currently.

Re: [PATCH] minimal SAS transport class

[Luben Tuikov]

On 08/18/05 13:56, Christoph Hellwig wrote:
> I was trying to avoid the need for an rport object, but I'm not yet sure
> it's feasible.  We certainly won't put in target-persistency support
> similar to FC, that was just a hack to get the existing drivers migrated
> without lots of complaints, it's not going in for new transports like
> SAS or iSCSI.

Hmm, I remember working on iSCSI exactly 5 years ago.
Is this considered new?  (er, I mean in light of SCSI Core)

I also remember it was that time when I asked about 8 byte LUNs
to be supported transparently by SCSI Core.

It was also that time when I asked for "target" to be supported
without "channel" and "id" as this had to be invented by
the iSCSI LLDD as it needs to be invented by FC and USB.

It was also that time when I asked for "target" to be
represented transparently so that SCSI Core can do LU
discovery and register the devices found and register
them with the Command Set Drivers (sd, st, etc).

On all three points I saw great opposition from the maintainers.

Sadly, I had implemented all those things, but the products
(target and initiator) was closed source.

This is where scsi_done_q and scsi commands from slab cache came from...

>  What we might need an rport for is to support SMP.  I'm
> not yet sure how to do SMP passthrough, but we will need some object
> to represent SMP ports.

Why?  How is this going to be used?  What is the architecture model?

	Luben

Re: [PATCH] minimal SAS transport class

[Christoph Hellwig]

On Thu, Aug 18, 2005 at 04:05:29PM -0400, Luben Tuikov wrote:
> On 08/18/05 13:56, Christoph Hellwig wrote:
> > I was trying to avoid the need for an rport object, but I'm not yet sure
> > it's feasible.  We certainly won't put in target-persistency support
> > similar to FC, that was just a hack to get the existing drivers migrated
> > without lots of complaints, it's not going in for new transports like
> > SAS or iSCSI.
> 
> Hmm, I remember working on iSCSI exactly 5 years ago.
> Is this considered new?  (er, I mean in light of SCSI Core)

We've just added the first iSCSI LLDD (the open-iscsi software
initiator) to the scsi git trees, so yes, this is considered new.

> I also remember it was that time when I asked about 8 byte LUNs
> to be supported transparently by SCSI Core.
> 
> It was also that time when I asked for "target" to be supported
> without "channel" and "id" as this had to be invented by
> the iSCSI LLDD as it needs to be invented by FC and USB.

Or just ignored.

> >  What we might need an rport for is to support SMP.  I'm
> > not yet sure how to do SMP passthrough, but we will need some object
> > to represent SMP ports.
> 
> Why?  How is this going to be used?  What is the architecture model?

We need some way to implement SMP passthrough.  One of the options is
to have a passthrough node that frames can be written to - for that we
obviously need an object for non-scsi target sas remote ports.

SAS Management Protocol (SMP)

[Douglas Gilbert]

Christoph Hellwig wrote:
> On Thu, Aug 18, 2005 at 04:05:29PM -0400, Luben Tuikov wrote:
> 
>>On 08/18/05 13:56, Christoph Hellwig wrote:

<snip/>

>>> What we might need an rport for is to support SMP.  I'm
>>>not yet sure how to do SMP passthrough, but we will need some object
>>>to represent SMP ports.
>>
>>Why?  How is this going to be used?  What is the architecture model?
> 
> 
> We need some way to implement SMP passthrough.  One of the options is
> to have a passthrough node that frames can be written to - for that we
> obviously need an object for non-scsi target sas remote ports.

Well lets talk about SMP for a bit. We need to
be able to use this protocol for more complex SAS
topologies. Monitoring simple SAS topologies with
only one expander would be aided by using SMP.

Like many management protocols, SMP violates
the layer abstraction. Since it sits besides SSP and STP
above the transport and port layers in the SAS stack then
it should be addressing via a (source) port. However its
functions talk about phys (not ports).

Further, if you look at SDI (revision 0), specifically
the SDI_SMP_PASSTHROUGH in section 5.12, then it assumes
the SAS initiator (rather than an initiator port) can
have an ioctl sent to it. On the source side it routes
via either:
  - a phy  ***
  - a port
  - or both (a phy within a port)

It also specifies the connection rate (SAS defines both
1.5 and 3 Gbps), and the destination SAS address (most likely
of an expander). That ioctl reports two levels of errors:
  - associated with the connection (e.g. ...REJECT_BAD_DESTINATION)
  - from the SMP target (e.g. UNKNOWN_SMP_FUNCTION)

and if all is well the response may include information
from the target (i.e. an expander). The ioctl also handles
security issues which a pretty simple in the case of
SMP [codes 0x0->0x7f fetch information while codes
0x80->0xff potentially change the state of an expander].

BTW There is nothing to stop a SAS HBA implementing a
SMP target, which would allow multiple initiators to
at least see one another. Also SMP has its own pass-through
to read and write a GPIO register.

Now the SDI_SMP_PASSTHROUGH ioctl seems pretty ugly,
passing over 2048 bytes of data for each function
(1024 bytes for the request and 1024 bytes for the
response). That 1024 matches the maximum payload
for SMP requests and responses (SAS 1.1 transport
layer). Given the definition of SAS-1.1 SMP functions
that is massive overkill. However zoning in SAS-2
will increase the number and size of some SMP functions
(and responses). [One company has already announced
silicon for SAS-2 zoning.]
So the SDI_SMP_PASSTHROUGH is closely tuned to the
capabilities of SMP rather than a generic pass through.

A SAS expander must have an SMP target whose SAS address
is by definition the expander's address. Expanders are
not SCSI devices (but may include or be associated with
SCSI devices (e.g. a SES device)). Expanders directly
connected to a SAS HBA are visible as "attached" (SAS)
devices; other expanders are discovered via SMP.

My theme here is if we are not going to use the
SDI_SMP_PASSTHROUGH at least we should understand what
it does. It is different from the SG_IO ioctl (for
example) which opens a connection to an end device
that has a device node and a sysfs identity (in
/sys/class/scsi_device). The SG_IO ioctl has no
addressing information in its metadata. Rather
SDI_SMP_PASSTHROUGH talks via a SAS HBA which has a
sysfs identity (and could have a device node) and
passes addressing information (source port/phy and
SMP target address) through as part of its metadata.
That way expanders (especially those not directly
attached to the HBA) do not have to appear in sysfs.

I'll stop here to see who objects to the above.


*** SMP seems to consume one connection along a path for
    the duration of its request-response cycle. So when
    an SMP initiator sends a request through a source phy then
    nothing else will happen on that phy until the response
    (or some error) arrives back on that phy.

Doug Gilbert

Re: SAS Management Protocol (SMP)

[Luben Tuikov]

On 08/24/05 08:46, Douglas Gilbert wrote:
> 
> Well lets talk about SMP for a bit. We need to
> be able to use this protocol for more complex SAS
> topologies. Monitoring simple SAS topologies with
> only one expander would be aided by using SMP.
> 
> Like many management protocols, SMP violates
> the layer abstraction. Since it sits besides SSP and STP
> above the transport and port layers in the SAS stack then
> it should be addressing via a (source) port. However its
> functions talk about phys (not ports).

Ok.

> Further, if you look at SDI (revision 0), specifically
> the SDI_SMP_PASSTHROUGH in section 5.12, then it assumes
> the SAS initiator (rather than an initiator port) can
> have an ioctl sent to it. On the source side it routes
> via either:
>   - a phy  ***
>   - a port
>   - or both (a phy within a port)
> 
> It also specifies the connection rate (SAS defines both
> 1.5 and 3 Gbps), and the destination SAS address (most likely
> of an expander). That ioctl reports two levels of errors:
>   - associated with the connection (e.g. ...REJECT_BAD_DESTINATION)
>   - from the SMP target (e.g. UNKNOWN_SMP_FUNCTION)
> 
> and if all is well the response may include information
> from the target (i.e. an expander). The ioctl also handles
> security issues which a pretty simple in the case of
> SMP [codes 0x0->0x7f fetch information while codes
> 0x80->0xff potentially change the state of an expander].
> 
> BTW There is nothing to stop a SAS HBA implementing a
> SMP target, which would allow multiple initiators to
> at least see one another. Also SMP has its own pass-through
> to read and write a GPIO register.
> 
> Now the SDI_SMP_PASSTHROUGH ioctl seems pretty ugly,

Yeah. ;-)

> passing over 2048 bytes of data for each function
> (1024 bytes for the request and 1024 bytes for the
> response). That 1024 matches the maximum payload
> for SMP requests and responses (SAS 1.1 transport
> layer). Given the definition of SAS-1.1 SMP functions
> that is massive overkill. However zoning in SAS-2

Exactly my sentiments.

> will increase the number and size of some SMP functions
> (and responses). [One company has already announced
> silicon for SAS-2 zoning.]

I love zoning.

> So the SDI_SMP_PASSTHROUGH is closely tuned to the
> capabilities of SMP rather than a generic pass through.
> 
> A SAS expander must have an SMP target whose SAS address
> is by definition the expander's address. Expanders are
> not SCSI devices (but may include or be associated with
> SCSI devices (e.g. a SES device)). Expanders directly
> connected to a SAS HBA are visible as "attached" (SAS)
> devices; other expanders are discovered via SMP.
> 
> My theme here is if we are not going to use the
> SDI_SMP_PASSTHROUGH at least we should understand what
> it does.

Forget about it.  I think we should go by the spec -- it's
our best friend.

> It is different from the SG_IO ioctl (for
> example) which opens a connection to an end device

True.

> that has a device node and a sysfs identity (in
> /sys/class/scsi_device). The SG_IO ioctl has no
> addressing information in its metadata. Rather
> SDI_SMP_PASSTHROUGH talks via a SAS HBA which has a
> sysfs identity (and could have a device node) and
> passes addressing information (source port/phy and
> SMP target address) through as part of its metadata.
> That way expanders (especially those not directly
> attached to the HBA) do not have to appear in sysfs.
> 
> I'll stop here to see who objects to the above.

Nothing to object to.

Question:  What if the transport layer (please see my
email to Christoph I just sent), "shows" you a picture
of how the physical world "looks" and then you just
"point" at an object with SMP port and voila you can
send an SMP request and receive and SMP response back.

No frame limitations, nothing.

Note, SCSI Core, abiding by SPC/SAM, will be completely
unaware of SMP devices.  That is, SMP is a _protocol_
"thing", we should keep it at the protocol layer.

	Luben


 
> *** SMP seems to consume one connection along a path for
>     the duration of its request-response cycle. So when
>     an SMP initiator sends a request through a source phy then
>     nothing else will happen on that phy until the response
>     (or some error) arrives back on that phy.
> 
> Doug Gilbert
> 
>