Re: Recommended HBA management interfaces

[James Smart <James.Smart@Emulex.Com>]

Mukker, Atul wrote:
> Thanks for restating my original question.
>
> 1. What interface should be used by the HBA management applications to obtain (non-generic) information from the HBA?
>   
My opinions:

sysfs :
  Pro: Good for singular data items and simple status  (link state, f/w 
rev, etc).
         Very good for things that really don't need a tool (simplistic 
admin commands):
             (show state, reset board, etc).
  Con: Doesn't work well for "transactions" that need multiple data elements
          Lack of insight to process life cycle, thus multi-step and 
concurrent
             transactions difficult.
          Doesn't work with binary data, buffers, etc.
          Difficult to use concurrently by multiple processes.
          Can't push async info to user.
          No support for complex things.
          The list of attributes can get big. Not a big deal, but...
          Security based on attribute permissions (not always the best 
model)

configfs:
  Pro: Basically sysfs but for transactions with multiple data elements
  Con: (same as sysfs, just minus multiple data element con).

netlink:
  Pro: Very good for "multi-cast" operations - pushing async events to 
multiple
              receivers.
         Handles requests and responses with multiple data elements easily.
         Can track per-process life cycles.
         Socket based so could even support mgmt from a different machine.
         Security checking easy to build in.
  Con: Doesn't work well for large payloads.
         Payloads can't be referenced via data pointer (they need to be 
inline to the pkt).
         Direct DMA not supported - has to be staged to driver buffer, 
copied in/out
            of socket.
         Multi-step transactions doable, but difficult. Maintaining 
relationships per
            pid difficult.
         Multiple machines means dealing with endian-ness and data typing.
         The netlink sockets do have memory-related issues that must be 
watched.

   Note: to not burn NETLINK id space, and perhaps collide in different 
distro
        kernels, please use the mid-layers netlink infrastructure, which 
does allow
        driver-specific messaging.

bsg:
  (Specifically the new midlayer sgio support that was recently added 
for ELS passthru)
  Pro: Support requests and responses with multiple data elements easily
         Supports separate request and response DMA-able payload buffers
         Supports big payloads easily
  Con: Lack of insight to process lifecycle, thus multi-step and concurrent
             transactions difficult.
          Async response generation (w/o associated request) very difficult.
          It's really a wrappered ioctl, with the midlayer protecting 
the kernel from
             bad ioctl practice via the way it converts the sgio ioctl 
into a midlayer
             request. Creates an odd programming interface, as you 
really want to
             wrapper the ioctl on the user side too.

Thus, when you look across the pros and cons, its easy to see why the 
transport
is using different things for different purposes.

> 2. How should driver notify such applications of asynchronous events happening on the HBA?
>   
This is already there with the midlayer netlink support.  Vendor-unique 
events
are already supported.

> Please keep in mind, all the data transfer between the applications and the HBA is a private protocol.
>   
Private or not, the code for the interface use will have to be in the 
driver.  Code will
be inspected for proper/safe usage of the interfaces.  Coding such that 
things in the
messaging are black-boxes will always be a point of contention.

-- james s


> Thanks
> Atul Mukker
>  
>
>   
>> -----Original Message-----
>> From: James Smart [mailto:James.Smart@Emulex.Com]
>> Sent: Monday, July 20, 2009 12:58 PM
>> To: Mukker, Atul
>> Cc: Brian King; linux-scsi@vger.kernel.org
>> Subject: Re: Recommended HBA management interfaces
>>
>> FYI - netlink (and sysfs, and I believe debugfs) do not exist with
>> vmware drivers...   Additionally, with netlink, many of the distros no
>> longer include libnl by default in their install images.  Even
>> interfaces that you think exist on vmware, may have very different
>> semantical behavior (almost all of the transport stuff either doesn't
>> exist or is only partially implemented).
>>
>> One big caveat I'd give you:  It's not so much the interface being used,
>> but rather, what are you doing over the interface.  One of the goals of
>> the community is to present a consistent management paradigm for like
>> things.  Thus, if what you are doing is generic, you should do it in a
>> generic manner so that all drivers for like hardware can utilize it.
>> This was the motivation for the protocol transports. Interestingly, even
>> the transports use different interfaces for different things. It all
>> depends on what it is.
>>
>> Lastly, some things are considered bad practice from a kernel safety
>> point of view. Example: driver-specific ioctls passing around user-space
>> buffer pointers.  In these cases, it doesn't matter what interface you
>> pick, they'll be rejected.
>>
>> -- james s
>>
>>
>> Mukker, Atul wrote:
>>     
>>> Thanks Brian. Netlink seems to be appropriate for our purpose as well,
>>>       
>> almost too good :-)
>>     
>>> That make me think, what's the catch? The SCSI drivers are not heavy
>>>       
>> usage of this interface for one.
>>     
>>> Are the other caveats associated with it?
>>>
>>> Best regards,
>>> Atul Mukker
>>>
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Brian King [mailto:brking@linux.vnet.ibm.com]
>>>> Sent: Friday, July 17, 2009 11:36 AM
>>>> To: Mukker, Atul
>>>> Cc: linux-scsi@vger.kernel.org
>>>> Subject: Re: Recommended HBA management interfaces
>>>>
>>>> Mukker, Atul wrote:
>>>>
>>>>         
>>>>> Hi All,
>>>>>
>>>>> We would like expert comments on the following questions regarding
>>>>> management of HBA from applications.
>>>>>
>>>>> Traditionally, our drivers create a character device node, whose
>>>>> file_operations are then used by the management applications to
>>>>> transfer HBA specific commands. In addition to being quirky, this
>>>>> interface has a few limitations which we would like to remove, most
>>>>> important being able to seamlessly handle asynchronous events with
>>>>> data transfer.
>>>>>
>>>>> 1. What is (are) the other standard/recommended interfaces which
>>>>> applications can use to transfer HBA specific commands and data.
>>>>>
>>>>>           
>>>> Depends on what the commands look like. With ipr, the commands that
>>>> the management application need to send to the HBA look sufficiently
>>>> like SCSI that I was able to report an sg device node for the adapter
>>>> and use SG_IO to send these commands.
>>>>
>>>> sysfs, debugfs, and configfs are options as well.
>>>>
>>>>
>>>>
>>>>         
>>>>> 2. How should an LLD implement interfaces to transmit asynchronous
>>>>> information to the management applications? The requirement is to be
>>>>> able to transmit data buffer as well as notifications for events.
>>>>>
>>>>>           
>>>> I've had good success with netlink. In my use I only send a
>>>>         
>> notification
>>     
>>>> to userspace and let the application send some commands to figure out
>>>> what happened, but netlink does allow to send data as well. It makes it
>>>> very
>>>> easy to have multiple concurrent readers of the data, which I've found
>>>> very
>>>> useful.
>>>>
>>>>
>>>>         
>>>>> 3. The interface should be able to work even if no SCSI devices are
>>>>> exported to the kernel.
>>>>>
>>>>>           
>>>> netlink allows this.
>>>>
>>>>
>>>>         
>>>>> 4. Should work seamlessly across vmware and xen kernels.
>>>>>
>>>>>           
>>>> netlink should work here too.
>>>>
>>>> -Brian
>>>>
>>>> --
>>>> Brian King
>>>> Linux on Power Virtualization
>>>> IBM Linux Technology Center
>>>>
>>>>
>>>>         
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>       
>
>