bidi support: FC transport layer...

bidi support: FC transport layer...

[Seokmann Ju]

Hi,

With starting to implement FC-CT/ELS support on the qla2xxx module, I  
would like to get some idea about the bidi-bidirectional.
As I understand that the bidi is packet transporting infra-structure,  
I think it could be good candidate for the FC specific FC-CT/ELS  
packet delivery in between the application and the individual devices  
given topology.

As I have limited understanding about the bid, here are my questions,
0. would the bidi be a right choice for the FC-CT/ELS packet delivery?
1. where should I get any kind of documents/briefs about the bidi?
2. which part of code that I have to walk-through for further  
understanding about the bidi mechanism?
3. with the bidi, what's the application interface should look like?

I expect to have some changes/additions in the FC transport layer in  
regards to the support and in general.

Any comment/guidance would be greatly helpful.

Thank you,
Seokmann

Re: bidi support: FC transport layer...

[Boaz Harrosh]

Seokmann Ju wrote:
> Hi,
> 
> With starting to implement FC-CT/ELS support on the qla2xxx module, I  
> would like to get some idea about the bidi-bidirectional.
Hi Dear Seokmann

Searching for "FC-CT ELS" did not produce any comprehensive results
for me. If you could send me exact pointer to the t11 paper I could
investigate it deeper.

> As I understand that the bidi is packet transporting infra-structure,  
> I think it could be good candidate for the FC specific FC-CT/ELS  
> packet delivery in between the application and the individual devices  
> given topology.
> 

bidi, as in scsi-bidirectional commands, is any scsi command (in any 
command set) where a single CDB utilizes both an in-buffer and an
out-buffer, for sending-while-receiving data to-from a target device.
The write-from-host buffers and read-to-host buffers happens concurrently
with out predetermined serialization and can in fact happen all at once.

So to answer your question: It is a must candidate for bidi, if the t11
standard states that a single CDB specifies both an out-buffer and an 
in-buffer, to complete the command.

An example of a bidi command is the XOR family of commands as defined
by SCSI for the block command set.

> As I have limited understanding about the bid, here are my questions,
> 0. would the bidi be a right choice for the FC-CT/ELS packet delivery?
Please point me to the normative documentation so I can check.

> 1. where should I get any kind of documents/briefs about the bidi?

In code and git I'm afraid. Look in git for the history of both
drivers/scsi/libiscsi.c and drivers/scsi/scsi_debug.c and see how they
are changed to receive bidirectional commands.
And also include/scsi/scsi_cmnd.h and drivers/scsi/scsi_lib.c for
these patches that introduce the bidi support, they have some explanations
and you can inspect their implementation.

> 2. which part of code that I have to walk-through for further  
> understanding about the bidi mechanism?

Best is to look in git for the relevant patches both at scsi-midlevel
and iscsi.

> 3. with the bidi, what's the application interface should look like?
> 

if you have for example:
int xxx_write(const void* out_buff, int len);
and
int xxx_read(void * in_buff, int_len);
then you might have something like
int xxx_xor(const void* out_buff, void * in_buff, int len);
which is a bidi command that writes and reads results all at once.

> I expect to have some changes/additions in the FC transport layer in  
> regards to the support and in general.
> 
> Any comment/guidance would be greatly helpful.
> 

scsi-bidirectional commands was crafted for scsi command sets that specifically
called for both in/out buffers participating in a single command. If the commands
in question are like that then this is the right tool to use.

> Thank you,
> Seokmann
> --

When you advance farther I can send you code examples of how an initiator pushes
block-requests that carry bidi payload that's the easy part. Getting your driver
to support and expect bidi commands is a bit harder. (Just a bit)

Feel free to ask any questions you have.

Cheers 
Boaz

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 10, 2008, at 8:30 AM, Boaz Harrosh wrote:

> Seokmann Ju wrote:
>> Hi,
>>
>> With starting to implement FC-CT/ELS support on the qla2xxx module, I
>> would like to get some idea about the bidi-bidirectional.
> Hi Dear Seokmann
>
> Searching for "FC-CT ELS" did not produce any comprehensive results
> for me. If you could send me exact pointer to the t11 paper I could
> investigate it deeper.
Hello Boaz,

First of all, I should express thanks for your comments on the  
questions.

Here is the link for ElS,
http://www.t11.org/index.html
If you search 'Extended Link Service', you may find details about it.

Basically, the FC-CT and ELS are FC specific frames and their usage for
management and control of Fibre Channel systems.
>
>
>> As I understand that the bidi is packet transporting infra-structure,
>> I think it could be good candidate for the FC specific FC-CT/ELS
>> packet delivery in between the application and the individual devices
>> given topology.
>>
>
> bidi, as in scsi-bidirectional commands, is any scsi command (in any
> command set) where a single CDB utilizes both an in-buffer and an
> out-buffer, for sending-while-receiving data to-from a target device.
> The write-from-host buffers and read-to-host buffers happens  
> concurrently
> with out predetermined serialization and can in fact happen all at  
> once.
Great, thanks for the key of the concept.
>
>
> So to answer your question: It is a must candidate for bidi, if the  
> t11
> standard states that a single CDB specifies both an out-buffer and an
> in-buffer, to complete the command.
FC-CT and ELS are FC specifics.
So, as far as I understood, the idea is to control and manage the  
given FC layout via those frames without having any subsystem  
interventions.
One of the goal of the implementation is to provide pipe line in  
between the application and the target devices under the given layout.
Is the bidi tightly coupled with the SCSI CDB, or it can transport any  
packet regardless its contents?

>
>
> An example of a bidi command is the XOR family of commands as defined
> by SCSI for the block command set.
>
>> As I have limited understanding about the bid, here are my questions,
>> 0. would the bidi be a right choice for the FC-CT/ELS packet  
>> delivery?
> Please point me to the normative documentation so I can check.
>
>> 1. where should I get any kind of documents/briefs about the bidi?
>
> In code and git I'm afraid. Look in git for the history of both
> drivers/scsi/libiscsi.c and drivers/scsi/scsi_debug.c and see how they
> are changed to receive bidirectional commands.
> And also include/scsi/scsi_cmnd.h and drivers/scsi/scsi_lib.c for
> these patches that introduce the bidi support, they have some  
> explanations
> and you can inspect their implementation.
Great, I will walk-through the files.
I've seen some of your emails which describing the idea of the bidi  
patches.
If you have any other, would be helpful, too.

>
>
>> 2. which part of code that I have to walk-through for further
>> understanding about the bidi mechanism?
>
> Best is to look in git for the relevant patches both at scsi-midlevel
> and iscsi.
>
>> 3. with the bidi, what's the application interface should look like?
>>
>
> if you have for example:
> int xxx_write(const void* out_buff, int len);
> and
> int xxx_read(void * in_buff, int_len);
> then you might have something like
> int xxx_xor(const void* out_buff, void * in_buff, int len);
> which is a bidi command that writes and reads results all at once.
Thanks.
One of the sample file that I've looked at was: expander_conf.c which  
shows how to handle SAS SMP packet.

>
>
>> I expect to have some changes/additions in the FC transport layer in
>> regards to the support and in general.
>>
>> Any comment/guidance would be greatly helpful.
>>
>
> scsi-bidirectional commands was crafted for scsi command sets that  
> specifically
> called for both in/out buffers participating in a single command. If  
> the commands
> in question are like that then this is the right tool to use.
>
>> Thank you,
>> Seokmann
>> --
>
> When you advance farther I can send you code examples of how an  
> initiator pushes
> block-requests that carry bidi payload that's the easy part. Getting  
> your driver
> to support and expect bidi commands is a bit harder. (Just a bit)
>
> Feel free to ask any questions you have.
Thanks again, I will get to you as I move forward.

Seokmann
>
>
> Cheers
> Boaz

Re: bidi support: FC transport layer...

[Boaz Harrosh]

Seokmann Ju wrote:
<snip>
> FC-CT and ELS are FC specifics.
> So, as far as I understood, the idea is to control and manage the  
> given FC layout via those frames without having any subsystem  
> interventions.
> One of the goal of the implementation is to provide pipe line in  
> between the application and the target devices under the given layout.
> Is the bidi tightly coupled with the SCSI CDB, or it can transport any  
> packet regardless its contents?
> 

There is *no* restrictions or filtering what so ever. You can send any 
kind of CDB with any kind of buffers as BIOs. Even better, the scsi
midlayer is very careful to immediately fail a bidi command and not 
try any kind of reties or error handling. So yes if you are emulating 
a command execution in SW you should be safe. What I said is true at 
scsi level, at block level of course there is no restrictions for 
any command.

<snip>

Boaz

RE: bidi support: FC transport layer...

[James.Smart]

Seokmann,

Yes - the idea was that when SGIO v4 and BIDI support was finally in the
kernel
(and stable), we would utilize that interface for ELS and CT pass thru.
We will
be abusing the "scsi-ness" of this infrastructure, but should be ok.  We
can use
the "CDB" area to pass request control information, and the "Sense
Data/Response"
area to pass response control information, and the BIDI payloads for the
actual
write and read buffers.

We have a few hurdles to cross to get there though:
- We're going to create an sgio object - is the object 1:1 with
sysfs/device objects ?
  or do we allow the adapter/port to be the base object and we then pass
addressing
  information in the request control area.  I lean toward the latter, as
there may
  be things on the fabric the adapter/port does not want or does not
know about, to
  present to the OS. However, I know James is in the other camp - always
have a node
  so that the addressing is implicit in the node being talked to. But,
in order to
  do this for things that may not already be enumerated, we have to
build in a 
  funky "temporarily create object X" interface so you can then talk to
it, then
  tear it down.

- We have to figure out a way to address asynchronous reception. It's
easy to do
  the request side via BIDI, but how about an ELS/CT request to the port
? where
  do we get the buffers ? is it DMA-based ? is there multiple buffer
pools and
  a traffic classification policy ? how do we associate traffic to the
right 
  user entity ? and so on.

- Will the ideas of "abusing" the SGIO SCSI request work for generic FC
traffic.
  I believe we've got the primitives right when we could actually
support FCP
  traffic through this raw interface.

- This will all need to be clean integrated into the transport, and we
have
  to consider the open-FCOE stack as well. 

-- james s


> -----Original Message-----
> From: linux-scsi-owner@vger.kernel.org 
> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Seokmann Ju
> Sent: Thursday, July 10, 2008 8:56 AM
> To: linux-scsi@vger.kernel.org
> Cc: Andrew Vasquez
> Subject: bidi support: FC transport layer...
> 
> Hi,
> 
> With starting to implement FC-CT/ELS support on the qla2xxx 
> module, I  
> would like to get some idea about the bidi-bidirectional.
> As I understand that the bidi is packet transporting 
> infra-structure,  
> I think it could be good candidate for the FC specific FC-CT/ELS  
> packet delivery in between the application and the individual 
> devices  
> given topology.
> 
> As I have limited understanding about the bid, here are my questions,
> 0. would the bidi be a right choice for the FC-CT/ELS packet delivery?
> 1. where should I get any kind of documents/briefs about the bidi?
> 2. which part of code that I have to walk-through for further  
> understanding about the bidi mechanism?
> 3. with the bidi, what's the application interface should look like?
> 
> I expect to have some changes/additions in the FC transport layer in  
> regards to the support and in general.
> 
> Any comment/guidance would be greatly helpful.
> 
> Thank you,
> Seokmann
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 10, 2008, at 9:10 AM, James.Smart@Emulex.Com wrote:

> Seokmann,
>
> Yes - the idea was that when SGIO v4 and BIDI support was finally in  
> the
> kernel
> (and stable), we would utilize that interface for ELS and CT pass  
> thru.
> We will
> be abusing the "scsi-ness" of this infrastructure, but should be  
> ok.  We
> can use
> the "CDB" area to pass request control information, and the "Sense
> Data/Response"
> area to pass response control information, and the BIDI payloads for  
> the
> actual
> write and read buffers.
OK. So the bidi has coupled with the scsi somehow.
Could you please elaborate a bit further about how to abuse the "scsi- 
ness" and make it workable for the ELS/CT purpose?

>
>
> We have a few hurdles to cross to get there though:
> - We're going to create an sgio object - is the object 1:1 with
> sysfs/device objects ?
>  or do we allow the adapter/port to be the base object and we then  
> pass
> addressing
>  information in the request control area.  I lean toward the latter,  
> as
> there may
>  be things on the fabric the adapter/port does not want or does not
> know about, to
>  present to the OS. However, I know James is in the other camp -  
> always
> have a node
>  so that the addressing is implicit in the node being talked to. But,
> in order to
>  do this for things that may not already be enumerated, we have to
> build in a
>  funky "temporarily create object X" interface so you can then talk to
> it, then
>  tear it down.
I see...
I'm also more toward latter from my gut feeling.
And, it should also be great to know further details about the  
alternative approach, too.

>
>
> - We have to figure out a way to address asynchronous reception. It's
> easy to do
>  the request side via BIDI, but how about an ELS/CT request to the  
> port
> ? where
>  do we get the buffers ? is it DMA-based ? is there multiple buffer
> pools and
>  a traffic classification policy ? how do we associate traffic to the
> right
>  user entity ? and so on.
I see...
Regarding the traffic association, a similar approach that I only know  
of is: AEN (Asynchronous Event Notification) on the Solaris.
It is kind of registration being done by each of the application upon  
loading so that the AEN queue, later on, could identify the one among  
the registered when actual AEN happens.

>
>
> - Will the ideas of "abusing" the SGIO SCSI request work for generic  
> FC
> traffic.
>  I believe we've got the primitives right when we could actually
> support FCP
>  traffic through this raw interface.
>
> - This will all need to be clean integrated into the transport, and we
> have
>  to consider the open-FCOE stack as well.
OK. Thanks.

In general, I haven't got clear picture of what to do and trying to  
grab as much as possible.
I will get back to you with some more questions.

Seokmann
>
>
> -- james s
>
>
>> -----Original Message-----
>> From: linux-scsi-owner@vger.kernel.org
>> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Seokmann Ju
>> Sent: Thursday, July 10, 2008 8:56 AM
>> To: linux-scsi@vger.kernel.org
>> Cc: Andrew Vasquez
>> Subject: bidi support: FC transport layer...
>>
>> Hi,
>>
>> With starting to implement FC-CT/ELS support on the qla2xxx
>> module, I
>> would like to get some idea about the bidi-bidirectional.
>> As I understand that the bidi is packet transporting
>> infra-structure,
>> I think it could be good candidate for the FC specific FC-CT/ELS
>> packet delivery in between the application and the individual
>> devices
>> given topology.
>>
>> As I have limited understanding about the bid, here are my questions,
>> 0. would the bidi be a right choice for the FC-CT/ELS packet  
>> delivery?
>> 1. where should I get any kind of documents/briefs about the bidi?
>> 2. which part of code that I have to walk-through for further
>> understanding about the bidi mechanism?
>> 3. with the bidi, what's the application interface should look like?
>>
>> I expect to have some changes/additions in the FC transport layer in
>> regards to the support and in general.
>>
>> Any comment/guidance would be greatly helpful.
>>
>> Thank you,
>> Seokmann
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-scsi" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 10, 2008, at 1:28 PM, Seokmann Ju wrote:

>
> On Jul 10, 2008, at 9:10 AM, James.Smart@Emulex.Com wrote:
>> ...
>> We have a few hurdles to cross to get there though:
>> - We're going to create an sgio object - is the object 1:1 with
>> sysfs/device objects ?
>> or do we allow the adapter/port to be the base object and we then  
>> pass
>> addressing
>> information in the request control area.  I lean toward the latter,  
>> as
>> there may
>> be things on the fabric the adapter/port does not want or does not
>> know about, to
>> present to the OS. However, I know James is in the other camp -  
>> always
>> have a node
>> so that the addressing is implicit in the node being talked to. But,
>> in order to
>> do this for things that may not already be enumerated, we have to
>> build in a
>> funky "temporarily create object X" interface so you can then talk to
>> it, then
>> tear it down.
> I see...
> I'm also more toward latter from my gut feeling.
> And, it should also be great to know further details about the  
> alternative
> approach, too.
 From the code walk-through, for the SAS SMP case, the transport layer  
create
a file under sysfs per each device detection, which is former case.
Could you elaborate why we have to "temporarily create object X and  
destroy"?

In general, I've been walking through the code to figure out overall  
design
and implementation details about SAS SMP handling.
1. when each of target device detected, a request_queue for the device
  allocated/associated to it.
2. each of device gets device file under sysfs and the device file is
  used to send SMP via bsg and the packets get serviced in the nature of
  bidi.

I also think it should be safe to implement FC-CT/ELS support in the
manner similar to the SAS SMP.
Please let me know for any other missing pieces.


>>
>> - We have to figure out a way to address asynchronous reception. It's
>> easy to do
>> the request side via BIDI, but how about an ELS/CT request to the  
>> port
>> ? where
>> do we get the buffers ? is it DMA-based ? is there multiple buffer
>> pools and
>> a traffic classification policy ? how do we associate traffic to the
>> right
>> user entity ? and so on.
> I see...
> Regarding the traffic association, a similar approach that I only  
> know of is: AEN (Asynchronous Event Notification) on the Solaris.
> It is kind of registration being done by each of the application  
> upon loading so that the AEN queue, later on, could identify the one  
> among the registered when actual AEN happens.
>
>>
>>
>> - Will the ideas of "abusing" the SGIO SCSI request work for  
>> generic FC
>> traffic.
>> I believe we've got the primitives right when we could actually
>> support FCP
>> traffic through this raw interface.
>>
>> - This will all need to be clean integrated into the transport, and  
>> we
>> have
>> to consider the open-FCOE stack as well.
> OK. Thanks.
>
>
> In general, I haven't got clear picture of what to do and trying to  
> grab as much as possible.
> I will get back to you with some more questions.
>
> Seokmann
>>
>>
>> -- james s
>>
>>
>>> -----Original Message-----
>>> From: linux-scsi-owner@vger.kernel.org
>>> [mailto:linux-scsi-owner@vger.kernel.org] On Behalf Of Seokmann Ju
>>> Sent: Thursday, July 10, 2008 8:56 AM
>>> To: linux-scsi@vger.kernel.org
>>> Cc: Andrew Vasquez
>>> Subject: bidi support: FC transport layer...
>>>
>>> Hi,
>>>
>>> With starting to implement FC-CT/ELS support on the qla2xxx
>>> module, I
>>> would like to get some idea about the bidi-bidirectional.
>>> As I understand that the bidi is packet transporting
>>> infra-structure,
>>> I think it could be good candidate for the FC specific FC-CT/ELS
>>> packet delivery in between the application and the individual
>>> devices
>>> given topology.
>>>
>>> As I have limited understanding about the bid, here are my  
>>> questions,
>>> 0. would the bidi be a right choice for the FC-CT/ELS packet  
>>> delivery?
>>> 1. where should I get any kind of documents/briefs about the bidi?
>>> 2. which part of code that I have to walk-through for further
>>> understanding about the bidi mechanism?
>>> 3. with the bidi, what's the application interface should look like?
>>>
>>> I expect to have some changes/additions in the FC transport layer in
>>> regards to the support and in general.
>>>
>>> Any comment/guidance would be greatly helpful.
>>>
>>> Thank you,
>>> Seokmann
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>

RE: bidi support: FC transport layer...

[James.Smart]

 
> -----Original Message-----
> From: Seokmann Ju [mailto:seokmann.ju@qlogic.com] 
> Sent: Tuesday, July 15, 2008 2:55 PM
> To: Smart, James
> Cc: linux-scsi@vger.kernel.org; Andrew Vasquez
> Subject: Re: bidi support: FC transport layer...
>
>  From the code walk-through, for the SAS SMP case, the 
> transport layer  
> create
> a file under sysfs per each device detection, which is former case.
> Could you elaborate why we have to "temporarily create object X and  
> destroy"?

Well - as long as you are only going to talk to what the driver exports,
ok...

But, what if I want to talk to a Fabric Service or Switch entity the
driver
doesn't typically log into or stay logged into ?  Or what if I want to
talk
to another FCP initiator (not FCP traffic, but rather just ELS's/CT
traffic)
or want to talk to a non-FCP port, but the driver only enumerated the
rports
associated with FCP Targets ?  Or what if I just want to FC-Ping some
random
FC address ?

Somehow we have to make the driver/transport enumerate the device that
we
want to talk to.  Along with this, I'm assuming the driver has to
initiate/do
the basic PLOGI to the other address, and be told when to tear it down.

Another part I don't like about the "create a file for each enumeration"
is,
we rarely use these files. If the SAN is big, the system has lots of
adapters,
and so on, why make the system allocate/maintain/etc all of these things
that
are rarely used ? When you start enumerating per phy, and with logical
entities
layering on top of the physical thingies, it can really grow. At what
point does
SGIO run out of dev_t space ?  Consider that we are enumerating many
adapters in 
an enterprise system; and on the physical level down to phys, and
everything
on the transport such as expanders and phys on expanders and targets and
luns;
and add in virtualization such as NPIV or VSANs, each with potentially
large
san connectivity and their own large trees of these things... 
Does SGIO actually scale ?

> In general, I've been walking through the code to figure out overall  
> design
> and implementation details about SAS SMP handling.
> 1. when each of target device detected, a request_queue for the device
>   allocated/associated to it.
> 2. each of device gets device file under sysfs and the device file is
>   used to send SMP via bsg and the packets get serviced in 
> the nature of
>   bidi.
> 
> I also think it should be safe to implement FC-CT/ELS support in the
> manner similar to the SAS SMP.
> Please let me know for any other missing pieces.

Agreed - for the basic part.  However, you're still under the assumption
that SAS
enumerated everything.  Given that some SAS parts could perform
discovery on it's
own, with the LLDD interacting with the transport just enough to
enumerate what it
found useful, this may not always be the case. Additionally, there
aren't a lot
of other "protocols" or "port roles" that exist in SAS - but there are
in FC.

The more interesting part is to those things (switch services, non-FCP
targets)
that the driver did not enumerate.  We should allow the directed
enumeration of them
to perform some basic diagnostics or san queries.  And once you have
this, 
we can really open some interesting doors - why not an app that talks
FICON ? or
something that just message passes between 2 ports ? and so on...

-- james

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 15, 2008, at 12:35 PM, james.smart@emulex.com wrote:

>
>> -----Original Message-----
>> From: Seokmann Ju [mailto:seokmann.ju@qlogic.com]
>> Sent: Tuesday, July 15, 2008 2:55 PM
>> To: Smart, James
>> Cc: linux-scsi@vger.kernel.org; Andrew Vasquez
>> Subject: Re: bidi support: FC transport layer...
>>
>> From the code walk-through, for the SAS SMP case, the
>> transport layer
>> create
>> a file under sysfs per each device detection, which is former case.
>> Could you elaborate why we have to "temporarily create object X and
>> destroy"?
>
> Well - as long as you are only going to talk to what the driver  
> exports,
> ok...
>
> But, what if I want to talk to a Fabric Service or Switch entity the
> driver
> doesn't typically log into or stay logged into ?  Or what if I want to
> talk
> to another FCP initiator (not FCP traffic, but rather just ELS's/CT
> traffic)
> or want to talk to a non-FCP port, but the driver only enumerated the
> rports
> associated with FCP Targets ?  Or what if I just want to FC-Ping some
> random
> FC address ?
Just for my learning, should this (FC-ping the random FC address) case
also be covered by the implementation?

>
>
> Somehow we have to make the driver/transport enumerate the device that
> we
> want to talk to.  Along with this, I'm assuming the driver has to
> initiate/do
> the basic PLOGI to the other address, and be told when to tear it  
> down.
>
> Another part I don't like about the "create a file for each  
> enumeration"
> is,
> we rarely use these files. If the SAN is big, the system has lots of
> adapters,
> and so on, why make the system allocate/maintain/etc all of these  
> things
> that
> are rarely used ? When you start enumerating per phy, and with logical
> entities
> layering on top of the physical thingies, it can really grow. At what
> point does
> SGIO run out of dev_t space ?  Consider that we are enumerating many
> adapters in
> an enterprise system; and on the physical level down to phys, and
> everything
> on the transport such as expanders and phys on expanders and targets  
> and
> luns;
> and add in virtualization such as NPIV or VSANs, each with potentially
> large
> san connectivity and their own large trees of these things...
> Does SGIO actually scale ?
The kernel specifies max. as 32768.
~/block/gsg.c:BSG_MAX_DEVS	32768

It is not enough for 1.6 M max ports, clearly.
I'm just thinking that how the 32768 is small practically.

>
>
>> In general, I've been walking through the code to figure out overall
>> design
>> and implementation details about SAS SMP handling.
>> 1. when each of target device detected, a request_queue for the  
>> device
>>  allocated/associated to it.
>> 2. each of device gets device file under sysfs and the device file is
>>  used to send SMP via bsg and the packets get serviced in
>> the nature of
>>  bidi.
>>
>> I also think it should be safe to implement FC-CT/ELS support in the
>> manner similar to the SAS SMP.
>> Please let me know for any other missing pieces.
>
> Agreed - for the basic part.  However, you're still under the  
> assumption
> that SAS
> enumerated everything.  Given that some SAS parts could perform
> discovery on it's
> own, with the LLDD interacting with the transport just enough to
> enumerate what it
> found useful, this may not always be the case. Additionally, there
> aren't a lot
> of other "protocols" or "port roles" that exist in SAS - but there are
> in FC.
I see..
There are a couple of more roles have defined in the rports besides
FC_RPORT_ROLE_FCP_TARGET.

>
>
> The more interesting part is to those things (switch services, non-FCP
> targets)
> that the driver did not enumerate.  We should allow the directed
> enumeration of them
> to perform some basic diagnostics or san queries.  And once you have
> this,
> we can really open some interesting doors - why not an app that talks
> FICON ? or
> something that just message passes between 2 ports ? and so on...
OK.

Overall, I guess that having a device file under the sysfs for the
given port of the HBA should be the approach we need to pursue, then.
And single device file per each port on the instantiated HBA is also
be the case, I think.
For the approach, following are kind of questions that came to me,
1. tte driver has to maintain some sort of table of
  the all end ports, and identify one from the table with some index or
  opaque data provided by the application for a given FC-CT/ELS pkt.
2. Overall implementation framework should be similar to the one for
  SMP in the SAS transport.

Thank you,
Seokmann
>
>
> -- james
>
>
>

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 16, 2008, at 10:53 AM, Seokmann Ju wrote:

>
> On Jul 15, 2008, at 12:35 PM, james.smart@emulex.com wrote:
...
>>
>> Agreed - for the basic part.  However, you're still under the  
>> assumption
>> that SAS
>> enumerated everything.  Given that some SAS parts could perform
>> discovery on it's
>> own, with the LLDD interacting with the transport just enough to
>> enumerate what it
>> found useful, this may not always be the case. Additionally, there
>> aren't a lot
>> of other "protocols" or "port roles" that exist in SAS - but there  
>> are
>> in FC.
> I see..
> There are a couple of more roles have defined in the rports besides
> FC_RPORT_ROLE_FCP_TARGET.
>
>>
>>
>> The more interesting part is to those things (switch services, non- 
>> FCP
>> targets)
>> that the driver did not enumerate.  We should allow the directed
>> enumeration of them
>> to perform some basic diagnostics or san queries.  And once you have
>> this,
>> we can really open some interesting doors - why not an app that talks
>> FICON ? or
>> something that just message passes between 2 ports ? and so on...
> OK.
>
> Overall, I guess that having a device file under the sysfs for the
> given port of the HBA should be the approach we need to pursue, then.
> And single device file per each port on the instantiated HBA is also
> be the case, I think.
> For the approach, following are kind of questions that came to me,
> 1. tte driver has to maintain some sort of table of
> the all end ports, and identify one from the table with some index or
> opaque data provided by the application for a given FC-CT/ELS pkt.
> 2. Overall implementation framework should be similar to the one for
> SMP in the SAS transport.
I just wanted make sure that I've delivered clear communication.
Please let me know otherwise.

And now, I've got a couple of more questions on the approach.
1. Should the device file created per instantiate HBA, or per port on
  the HBA?
    I think it should be per port base, for example, when 2-ports HBA
  detected, we will create 2 device for the HBA.
2. Should we worry about the 4KB size limit?
    I've heard that sysfs attribute has limited size of data it can
   transport at one time and that is 4KB.

Thank you,
Seokmann



>
>
> Thank you,
> Seokmann
>>
>>
>> -- james
>>
>>
>>
>

RE: bidi support: FC transport layer...

[James.Smart]

Seokmann Ju wrote:
> Just for my learning, should this (FC-ping the random FC address) case
> also be covered by the implementation?

I would think this to be a great example. A small user-app that, given a
fc_host name, ping's a N_Port_ID or WWN via that port.

Assuming we create a sgio file per object :
Phase 1 : have the app could peruse the /sys/class/fc_rport/* elements
  for the fc_host, finds the one with the address or WWN, and invokes
the
  SGIO ioctl for the ELS.
Phase 2 : we deal with pinging something the hba didn't enumerate.

We would want to GPL-2 the app.

> It is not enough for 1.6 M max ports, clearly.

and I wouldn't expect it to be - and I'm not thinking about the 
FC 24 bit addr, but rather, from the system perspective, that the
sgio bits are shared by all things scsi on the system, which is SAS
and FC, etc.

> I'm just thinking that how the 32768 is small practically.

PS: I am concerned about the dev_t size, but it's not my first concern
for this api.

My concern is - this is shared by all things scsi. If you consider a
large system with say 64 adapters in it - some SAS some FC, and the
FC adapters with NPIV/VSANs, and throw in iSCSI and/or open-FCOE (it too
with vports) - and then consider that we use a bsg on every lun, on
every local SAS port, every remote SAS phy, every remote port. It would
not be hard to see a system with the equivalent of many hundred
adapters,
with say 50 or higher targets per adapter, with perhaps 16-2000 luns on
targets. At those multipliers, we easily could consume 50%, or all, of
a 32k space.  Yes, the predominant number of systems would never be
close,
but...

anyway - let's ignore this part for now.

> I see..
> There are a couple of more roles have defined in the rports besides
> FC_RPORT_ROLE_FCP_TARGET.

Um.. roles in the header file aren't what I'm talking about. FC has
a bunch of other FC4's defined.

> Overall, I guess that having a device file under the sysfs for the
> given port of the HBA should be the approach we need to pursue, then.
> And single device file per each port on the instantiated HBA is also
> be the case, I think.

Yes - we'll start with bsg dev_ts on the objects.

I'm assuming that in the fc transport, we do a bsg_register_queue()
for/under each fc_host and fc_rport created in sysfs.  Each of the 
objects will have a different request handler.

The request handler for fc_rport would contain primitives of :
  Perform ELS rqst; Perform CT rqst;

The request handler for fc_host would contain primitives of :
  <whatever we figure out for async traffic classification, buffer
   posting, etc - so we can handle recevied ELS and CT requests>
  Send ELS rsp; Send CT rsp
  
  and later:
  - Send ELS w/o login ?
  - Login to addr  (and assumes that an rport will be created for it)

We'll need to create an interface between the transport and the LLDD to
: 
- Perform an ELS; Perform a CT Request;
  Both of these referencing an rport  (note: if we didn't do it by
rport,
    it would be by fc_host and supplying an address/wwn).
  We can stop here and call it Phase 1a : e.g. we can send ELS or CT to
  a remote port and get an answer back - but, we aren't supporting
having
  the fc_host recieve an ELS/CT request and send back a response.  That
  latter part is Phase 1b, and we'll have to figure out the async
receive
  path for ELS/CT. (and no - I don't believe you can stop until you've
  completed both 1a and 1b).  We should look at the open-fcoe stack and
  make sure the interfaces are consistent - preferably the same.

- and the entry points corresponding to the fc_host primitives

- a generic bsg request handler when the request type wasn't picked off
  by the fc transport


> For the approach, following are kind of questions that came to me,
> 1. tte driver has to maintain some sort of table of
>   the all end ports, and identify one from the table with 
> some index or
>   opaque data provided by the application for a given FC-CT/ELS pkt.

I would think this just falls out of having rport structures. Should be
nothing special

> 2. Overall implementation framework should be similar to the one for
>   SMP in the SAS transport.

it will be, but the internals (e.g. send ELS/CT request) will be
different.


> I just wanted make sure that I've delivered clear communication.
> Please let me know otherwise.
> 
> And now, I've got a couple of more questions on the approach.
> 1. Should the device file created per instantiate HBA, or per port on
>   the HBA?
>     I think it should be per port base, for example, when 2-ports HBA
>   detected, we will create 2 device for the HBA.

per fc_host  - there should be a fc_host per each port.

> 2. Should we worry about the 4KB size limit?
>     I've heard that sysfs attribute has limited size of data it can
>    transport at one time and that is 4KB.

We're SGIO - so 4KB limits shouldn't matter/exist - it's not sysfs.

I have some assumptions on how the "request" is used though:
 - Whatever "control" information for the ELS or CT request should be
   passed in the "cmd" area of the request.
   I expect the FC transport to mandate something to be in the "cmd"
   area, with there minimally being a "request code".
 - Any FC-level status, such as driver generated "ELS timed out with no
   response" or BA_RJT/F_RJT/P_BSY status, etc - gets placed into the
   "sense" buffer area fo the request.
 - Transit payload is the request->bio
 - Receive payload is the request->next_rq->bio

I'm assuming we can initially live with a single bio vector from the
app.

-- james

Re: bidi support: FC transport layer...

[Mike Christie]

Seokmann Ju wrote:
> Hi,
> 
> With starting to implement FC-CT/ELS support on the qla2xxx module, I 
> would like to get some idea about the bidi-bidirectional.
> As I understand that the bidi is packet transporting infra-structure, I 
> think it could be good candidate for the FC specific FC-CT/ELS packet 
> delivery in between the application and the individual devices given 
> topology.

I think you might also want to look at bsg or really wanted to see bsg 
along with how to support bidi commands? You would want to look at 
block/bsg.c for the bsg driver. And see 
drivers/scsi/scsi_transport_sas.c's sas_bsg_initialize and its 
sas_non_host_smp_request functions for examples of how to send bsg to a 
transport class, and from a class to a class object like a rport 
(replace fc rport for a sas phy though).

Re: bidi support: FC transport layer...

[Seokmann Ju]

On Jul 10, 2008, at 9:14 AM, Mike Christie wrote:

> Seokmann Ju wrote:
>> Hi,
>> With starting to implement FC-CT/ELS support on the qla2xxx module,  
>> I would like to get some idea about the bidi-bidirectional.
>> As I understand that the bidi is packet transporting infra- 
>> structure, I think it could be good candidate for the FC specific  
>> FC-CT/ELS packet delivery in between the application and the  
>> individual devices given topology.
>
> I think you might also want to look at bsg or really wanted to see  
> bsg along with how to support bidi commands? You would want to look  
> at block/bsg.c for the bsg driver. And see drivers/scsi/ 
> scsi_transport_sas.c's sas_bsg_initialize and its  
> sas_non_host_smp_request functions for examples of how to send bsg  
> to a transport class, and from a class to a class object like a  
> rport (replace fc rport for a sas phy though).
Thank you for the point, Mike.
I'm looking at the files as you pointed out.

Seokmann