ANNOUCE: OpenIB InfiniBand software

ANNOUCE: OpenIB InfiniBand software

[Roland Dreier]

OPENIB.ORG INFINIBAND DRIVERS RELEASE -- 2004-03-05

  We are proud to announce the initial availability of the OpenIB.org
  InfiniBand stack.  This is a fully open source[1] InfiniBand stack
  that includes a low-level driver for Mellanox HCA hardware, a
  midlayer, and a number of useful upper-layer protocols including
  IP-over-InfiniBand, SCSI RDMA Protocol, Sockets Direct Protocol,
  uDAPL and MPI.  In addition userspace utilities are available,
  including the OpenSM subnet manager.

  [1]  See the section on SDP intellectual property for some important
       information, though.

  An initial drop of this source is available from
  <http://openib.org>.  Additional protocols and further enhancements
  will be added in upcoming releases.  In the next few days, we will
  also be bringing up mailing lists, source code control, and so on.

  We look forward to working with the entire Linux community to
  continue to develop these drivers.

TODO

  We are fully aware that our code is far from being acceptable for
  inclusion in the Linux kernel, and we are very committed to doing
  the hard work required to clean up the source.  We believe that in
  addition to the benefits of being in the standard kernel, we will
  find huge improvements in the quality, performance and robustness of
  our drivers.

  As a starting point, here is a list of some of the tasks we believe
  must be completed before these drivers can be seriously considered
  for inclusion in the kernel.  We would welcome any suggestions of
  tasks that we have forgotten.  Of course patches would be even
  better!

    * Use a more standard naming convention (tTS_IB_TYPE_NAME ->
      ib_type_name, tsIbFunctionName -> ib_function_name, etc)
    * Get rid of bizarro types such as tUINT32 -- replace with Linux
      standard u32 etc.
    * Run source through scripts/Lindent (and then fix by hand, since
      lots of things will no longer line up)
    * Get rid of XXX_exports.c files and move EXPORT_SYMBOL next to
      definition in source.
    * Change MODULE_PARM -> module_param
    * Use DMA mapping API so we work properly with non-coherent
      caches, IOMMUs etc. Add a function to get struct device * for an
      IB device so ULPs can call sync functions (embed struct device
      in struct ib_device?). (How do we handle systems with limited
      DMA mapping capacity?)
    * Fix up ioctls, userspace interface so that 32-bit userspace
      works with a 64-bit kernel.
    * Get rid of ib_poll module; move all timers to standard Linux
      timers (CM can use a work queue plus schedule_delayed_work).
    * Get rid of ib_services module (move TS_TRACE etc. to dev_printk)
    * Add missing pieces of API for currently unused IB features: deal
      with RD (EECs, etc -- including CM support), verbs extensions.
    * Get rid of IPoIB address hash and just use native 20-byte
      addresses (keep 6-byte hash as an option for compatibility
      with existing applications?)
    * Rewrite client query support (SA and DM) to be more extensible,
      simpler, deal with component mask.
    * Add sysfs support.
    * In Mellanox HCA driver, clean up mosal. Some obvious
      targets (there's lots more though):
          o Grunging through kernel code to find the mlock/munlock
            pointers especially has to go (mosal_mlock.c). We need to
            get VM experts from LKML to tell us how to handle memory
            translation and locking.
          o Turn MT_KERNEL back to __KERNEL__
          o Use native spinlocks.
          o Turn driver into a real PCI driver (rather than
            reimplementing a worse version of PCI bus scan).

SDP intellectual property

  Microsoft believes that they own certain intellectual property
  relating to the Sockets Direct Protocol (SDP)[2].  Therefore, we are
  including the following disclaimer required by Microsoft's license
  in SDP source that relates to the implementation of the protocol:

    "This source code may incorporate intellectual property owned by
    Microsoft Corporation. Our provision of this source code does not
    include any licenses or any other rights to you under any
    Microsoft intellectual property. If you would like a license from
    Microsoft (e.g., to rebrand, redistribute), you need to contact
    Microsoft directly."

  We realize that this is incompatible with open source licensing, and
  we are working to find a more satisfactory solution, but for the
  time being we are forced to comply with Microsoft's license.

  Please make sure you have fully understood the implications of
  Microsoft's claims before you redistribute any of the SDP source
  that contains the above disclaimer.

  [2]  <http://www.microsoft.com/mscorp/ip/standards/>

Re: ANNOUCE: OpenIB InfiniBand software

[Roland Dreier]

Several people have requested that we split the possibly-encumbered
SDP code into a separate package so that they don't have to download
it to get the free code.  To allow that, I have split the kernel code
into two packages:

    infiniband-kernel-2004-03-05.tar.bz2
    infiniband-kernel-sdp-2004-03-05.tar.bz2

If you wish to build SDP, you will need to download both packages and
move the SDP files into the appropriate place in the tree.  For all
other protocols, only the first package (containing only pure dual
GPL/BSD code) is required.

The new snapshot is available now from <http://openib.org/downloads>.
(It also contains a few fixes and code cleanups as compared to the
2004-02-26 drop)

Re: ANNOUCE: OpenIB InfiniBand software

[Troy Benjegerdes]

On Sat, Mar 06, 2004 at 01:00:59PM -0800, Roland Dreier wrote:
> Several people have requested that we split the possibly-encumbered
> SDP code into a separate package so that they don't have to download
> it to get the free code.  To allow that, I have split the kernel code
> into two packages:
> 
>     infiniband-kernel-2004-03-05.tar.bz2
>     infiniband-kernel-sdp-2004-03-05.tar.bz2
> 

Since the openib.org mailing lists don't seem to be alive yet, I'll
bring this up here..

Can we get this split out into the following components?

	* kernel level infiniband access layer
	* lowlevel hardware driver (aka mellanox driver)
	* all other 'upper layer protocols'

All the existing code has big problems with how it interfaces to the
kernel memory management.. it doesn't use the regular interfaces like
get_user_pages, pci_map_page, and pci_map_sq, and goes and looks at the
guts of the pagetables directly. 

This is either a problem with the kernel interfaces made available, or a
design issue with the existing codebases. Can someone please tell me
which it is?

-- 
--------------------------------------------------------------------------
Troy Benjegerdes                'da hozer'                hozer@hozed.org  

Somone asked my why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's why
I draw cartoons. It's my life." -- Charles Shultz

Re: ANNOUCE: OpenIB InfiniBand software

[Roland Dreier]

    Troy> Since the openib.org mailing lists don't seem to be alive
    Troy> yet, I'll bring this up here..

    Troy> Can we get this split out into the following components?
    Troy> * kernel level infiniband access layer
    Troy> * lowlevel hardware driver (aka mellanox driver)
    Troy> * all other 'upper layer protocols'

Right now, in the infiniband-kernel package, the Mellanox driver is
under drivers/infiniband/hw/mellanox-hca, the 'upper layer protocols'
are under drivers/infiniband/ulp, and the 'access layer' is everything
else in drivers/infiniband.  I'm not sure it's worth rolling three
packages for that split right now.

    Troy> All the existing code has big problems with how it
    Troy> interfaces to the kernel memory management.. it doesn't use
    Troy> the regular interfaces like get_user_pages, pci_map_page,
    Troy> and pci_map_sq, and goes and looks at the guts of the
    Troy> pagetables directly.

    Troy> This is either a problem with the kernel interfaces made
    Troy> available, or a design issue with the existing
    Troy> codebases. Can someone please tell me which it is?

I would say both.  Certainly the HCA driver code could have been
written much more cleanly.  However, it's not clear how the right way
to handle the memory management requirements of kernel bypass/RDMA
protocols, especially since the driver tries to support kernels all
the way back to the ancient 2.4.9 used by Red Hat AS 2.1 (which
doesn't even have get_user_pages()).

In any case, it's why I put the following items in the TODO list
of my announcement:

    * Use DMA mapping API so we work properly with non-coherent
      caches, IOMMUs etc. Add a function to get struct device * for an
      IB device so ULPs can call sync functions (embed struct device
      in struct ib_device?). (How do we handle systems with limited
      DMA mapping capacity?)

    * In Mellanox HCA driver, clean up mosal. Some obvious
      targets (there's lots more though):
          o Grunging through kernel code to find the mlock/munlock
            pointers especially has to go (mosal_mlock.c). We need to
            get VM experts from LKML to tell us how to handle memory
            translation and locking.

OpenIB mailing lists available (was Re: ANNOUNCE: OpenIB InfiniBand software)

[Roland Dreier]

We have moved the OpenIB web site to a new server and opened our mailing
list.  We invite everyone interested in InfiniBand support for Linux to
join us.  To sign up for the mailing list, send email with the subject
"subscribe" to openib-general@openib.org.  More information and list
archives can be found at <http://www.openib.org/openib-general>.

DNS has just been updated, so it may take some time for changes to
propagate to your site.

 - Roland