Updated InfiniBand/RDMA merge plans for 2.6.24

Updated InfiniBand/RDMA merge plans for 2.6.24

[Roland Dreier]

Since 2.6.23 still isn't out, and I've managed to reduce my patch
review backlog a bit, it's probably a good idea to give another update
about what I have queued for 2.6.24 already and what I hope to get to
before the merge window opens.

Core:

 - My user_mad P_Key index support patch.  Merged this, although I
   still owe Sasha a patch to update libraries to use this.

 - A fix to the user_mad 32-bit big-endian userspace 64/32 problem
   with the method_mask when registering agents.  Merged.

 - Sean's QoS changes.  Merged.

 - Sean's IB CM MRA interface changes.  I merged these -- what the
   heck, if it breaks we can back them out.

ULPs:

 - Pradeep's IPoIB CM support for devices that don't have SRQs.  Sean
   started reviewing but I didn't see any updated patches.

 - Moni's IPoIB bonding support.  Seems like we found a clean set of
   changes, and these will go in via another (Jeff Garzik's?) tree.

 - Rolf's IPoIB MGID scope changes.  No review progress here.

 - Eli and Michael's IPoIB stateless offload (checksum offload, LSO,
   LRO, etc).  Not much review progress here; I'll try to chip away at
   the series and see what we can get into 2.6.24.

 - Or's IPoIB/userspace multicast coexistence stuff.  I think we've
   converged on this; I'll merge this once a final version of the
   patch appears.

HW specific:

 - I already merged patches to enable MSI-X by default for mthca and
   mlx4.  I hope there aren't too many systems that get hosed if a
   MSI-X interrupt is generated.

 - Jack and Michael's mlx4 FMR support.  Merged.  I guess the fix for
   running in Xen domU may need to wait for 2.6.25, but I'll see what
   I can do.

 - ehca patch queue.  Merged everything I think.

 - Steve's mthca router mode support.  No one looked at it, seems like
   it's at risk of missing the window.

 - Arthur's mthca doorbell alignment fixes.  I still need to check
   various approaches;  I'll definitely merge something for 2.6.24.

 - Michael's mlx4 WQE shrinking patch.  May miss the window and go for
   2.6.25, I'll see if I can get to it.

Here are a few topics that I believe will not be ready in time for the
2.6.24 window and will need to wait for 2.6.25:

 - Multiple CQ event vector support.  I haven't seen any discussions
   about how ULPs or userspace apps should decide which vector to use,
   and hence no progress has been made since we deferred this during
   the 2.6.23 merge window.

 - XRC.  Given the length of the backlog above and the fact that a
   first draft of this code has not been posted yet, I don't see any
   way that we could have something this major ready in time.

BOILERPLATE
===========

I keep patches in a git tree, available from

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

There are several branches of interest in this tree:

    for-2.6.23 - changes queued for merging into the current kernel release
    for-2.6.24 - changes queued for the next merge window
    for-linus  - changes I have asked Linus to pull upstream
    for-mm     - pulled by Andrew for inclusion in -mm

I frequently rewrite history and rebase my tree, so the best way to
track it is to keep a clone of Linus's tree around and then pull a
fresh copy of my tree with

    git clone --reference /path/to/linus/tree \
        git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git

If you would like me to merge a patch, please send it to me as soon as
it is ready.  Do NOT wait for the merge window to open; if your change
is not strictly a fix and you send it to me after the merge window
opens, then it will likely have to wait for the next merge window.
Please let me know if your patch is a fix that should go into the
current release or if it can wait for the next merge window; if it is
a fix, please describe the severity of the issue your are fixing, so I
can make a good judgement about which release it should go into.

Including a good changelog entry that explains what you are changing,
why you are changing it, and how your change accomplishes your goal
will greatly increase the chance of your patch being merged promptly.
Getting an independent review and a Reviewed-by: line also helps a lot.

The files Documentation/SubmittingPatches and Documentation/SubmitChecklist
in kernel source tree also have a lot of good advice that makes it
easier for me to handle your patches.

DETAILS
=======

Here is the complete list of patches I have in my for-2.6.24 branch:

Ali Ayoub (1):
      IB/sa: Error handling thinko fix

Anton Blanchard (3):
      IB/fmr_pool: Clean up some error messages in fmr_pool.c
      IB/ehca: Make output clearer by removing some debug messages
      IB/ehca: Export module parameters in sysfs

Dotan Barak (1):
      mlx4_core: Use enum value GO_BIT_TIMEOUT_MSECS

Eli Cohen (2):
      IPoIB: Fix typo to end statement with ';' instead of ','
      IPoIB: Fix error path memory leak

Hoang-Nam Nguyen (4):
      IB/ehca: Use remap_4k_pfn() to map firmware contexts to user space
      IB/ehca: Fix large page HW cap defines
      IB/ehca: Fix mem leak of firmware ctrlblock in ehca_create_srq()
      IB/ehca: Adjust 64-bit alignment of create QP response for userspace

Jack Morgenstein (5):
      IB/mlx4:  Display misc device information under /sys/class/infiniband/
      mlx4_core: Support ICM tables in coherent memory
      mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command
      IB/mlx4: Implement FMRs
      mlx4_core: Increase max number of QPs per multicast group to 56

Joachim Fenkes (11):
      IB/ehca: Refactor hvcall tracing
      IB/ehca: Print return codes as signed decimal integers
      IB/ehca: ehca_gen_warn() should always print
      IB/ehca: Add check for max #SGE to create_qp()
      IB/ehca: Path migration support
      IB/ehca: Serialize MR alloc and MR free hvCalls
      IB/ehca: Replace get_paca()->paca_index by the more portable raw_smp_processor_id()
      IB/ehca: Bump version number and change its format
      IB/umem: Add hugetlb flag to struct ib_umem
      IB/ehca: Only use MR large pages for hugetlb regions
      IB/ehca: Return srq_attr->max_sge in ehca_query_srq()

Michael S. Tsirkin (2):
      mlx4_core: Enable MSI-X by default
      IB/mthca: Enable MSI-X by default

Peter Oruba (1):
      IB/mthca: Use PCI-X/PCI-Express read control interfaces

Ralph Campbell (1):
      IB/core: Fix handling of multicast response failures

Roland Dreier (14):
      IPoIB: Make sure no receives are handled when stopping device
      IB: find_first_zero_bit() takes unsigned pointer
      mlx4_core: Don't free special QPs in QP number bitmap
      IB/mlx4: Use __set_data_seg() in mlx4_ib_post_recv()
      IB/ehca: Include <linux/mutex.h> from ehca_classes.h
      IB/mlx4: Fix up SRQ limit_watermark endianness
      IB/iser: Remove unnecessary includes
      mlx4_core: Change capability decoding: SRC->XRC
      IB/umad: Add P_Key index support
      IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems
      IB/uverbs: Make ib_uverbs_release_event_file() static
      mlx4_core: Reserve the correct number of MTT segments
      mlx4_core: Fix meaning of dev->caps.reserved_mtts
      IB/mthca: Increase max number of QPs per multicast group to 56

Satyam Sharma (1):
      IB/ehca: Misc cpuinit section annotations and #ifdef cleanups

Sean Hefty (7):
      IPoIB: Specify Traffic Class with path record queries for QoS support
      IB/sa: Add new QoS fields to path record
      RDMA/cma: Add ability to specify type of service
      RDMA/ucma: Allow user space to set service type
      IB/srp: Add QoS support through service ID
      IB/cm: Modify interface to send MRAs in response to duplicate messages
      RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retries

Stefan Roscher (2):
      IB/ehca: Small QP userspace support
      IB/ehca: Support more than 4k QPs for userspace and kernelspace

Steve Wise (2):
      RDMA/cxgb3: Make the iw_cxgb3 module parameters writable
      RDMA/cma: Use neigh_event_send() to start neighbour discovery

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Pradeep Satyanarayana]

> 
> ULPs:
> 
>  - Pradeep's IPoIB CM support for devices that don't have SRQs.  Sean
>    started reviewing but I didn't see any updated patches.
> 
Roland, I submitted an updated patch incorporating some of Sean's comments within
a day or two. Rest of comments pertained to restructuring the code and adding 
some additional module parameters.

This would require more discussions since some of these had been already discussed
previously. We had decided upon this code structure after a lot of discussions and
incorporating these would be undoing some of that.

We can discuss and revisit the comments after the merge.

Pradeep

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Steve Wise]

No mention about the iwarp port space issue?

Here is the status of the current proposed patch:

- needs another round of changes based on Sean's feedback

- Arkady raised issues about the pain this puts on admins

- it forces services like nfs-rdma, which already separates the nfs-rdma 
server by port number, to needlessly use a separate subnet for the rdma 
service.

I'm at a loss as to how to proceed.

Any ideas?

Steve.


Roland Dreier wrote:
> Since 2.6.23 still isn't out, and I've managed to reduce my patch
> review backlog a bit, it's probably a good idea to give another update
> about what I have queued for 2.6.24 already and what I hope to get to
> before the merge window opens.
> 
> Core:
> 
>  - My user_mad P_Key index support patch.  Merged this, although I
>    still owe Sasha a patch to update libraries to use this.
> 
>  - A fix to the user_mad 32-bit big-endian userspace 64/32 problem
>    with the method_mask when registering agents.  Merged.
> 
>  - Sean's QoS changes.  Merged.
> 
>  - Sean's IB CM MRA interface changes.  I merged these -- what the
>    heck, if it breaks we can back them out.
> 
> ULPs:
> 
>  - Pradeep's IPoIB CM support for devices that don't have SRQs.  Sean
>    started reviewing but I didn't see any updated patches.
> 
>  - Moni's IPoIB bonding support.  Seems like we found a clean set of
>    changes, and these will go in via another (Jeff Garzik's?) tree.
> 
>  - Rolf's IPoIB MGID scope changes.  No review progress here.
> 
>  - Eli and Michael's IPoIB stateless offload (checksum offload, LSO,
>    LRO, etc).  Not much review progress here; I'll try to chip away at
>    the series and see what we can get into 2.6.24.
> 
>  - Or's IPoIB/userspace multicast coexistence stuff.  I think we've
>    converged on this; I'll merge this once a final version of the
>    patch appears.
> 
> HW specific:
> 
>  - I already merged patches to enable MSI-X by default for mthca and
>    mlx4.  I hope there aren't too many systems that get hosed if a
>    MSI-X interrupt is generated.
> 
>  - Jack and Michael's mlx4 FMR support.  Merged.  I guess the fix for
>    running in Xen domU may need to wait for 2.6.25, but I'll see what
>    I can do.
> 
>  - ehca patch queue.  Merged everything I think.
> 
>  - Steve's mthca router mode support.  No one looked at it, seems like
>    it's at risk of missing the window.
> 
>  - Arthur's mthca doorbell alignment fixes.  I still need to check
>    various approaches;  I'll definitely merge something for 2.6.24.
> 
>  - Michael's mlx4 WQE shrinking patch.  May miss the window and go for
>    2.6.25, I'll see if I can get to it.
> 
> Here are a few topics that I believe will not be ready in time for the
> 2.6.24 window and will need to wait for 2.6.25:
> 
>  - Multiple CQ event vector support.  I haven't seen any discussions
>    about how ULPs or userspace apps should decide which vector to use,
>    and hence no progress has been made since we deferred this during
>    the 2.6.23 merge window.
> 
>  - XRC.  Given the length of the backlog above and the fact that a
>    first draft of this code has not been posted yet, I don't see any
>    way that we could have something this major ready in time.
> 
> BOILERPLATE
> ===========
> 
> I keep patches in a git tree, available from
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git
> 
> There are several branches of interest in this tree:
> 
>     for-2.6.23 - changes queued for merging into the current kernel release
>     for-2.6.24 - changes queued for the next merge window
>     for-linus  - changes I have asked Linus to pull upstream
>     for-mm     - pulled by Andrew for inclusion in -mm
> 
> I frequently rewrite history and rebase my tree, so the best way to
> track it is to keep a clone of Linus's tree around and then pull a
> fresh copy of my tree with
> 
>     git clone --reference /path/to/linus/tree \
>         git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git
> 
> If you would like me to merge a patch, please send it to me as soon as
> it is ready.  Do NOT wait for the merge window to open; if your change
> is not strictly a fix and you send it to me after the merge window
> opens, then it will likely have to wait for the next merge window.
> Please let me know if your patch is a fix that should go into the
> current release or if it can wait for the next merge window; if it is
> a fix, please describe the severity of the issue your are fixing, so I
> can make a good judgement about which release it should go into.
> 
> Including a good changelog entry that explains what you are changing,
> why you are changing it, and how your change accomplishes your goal
> will greatly increase the chance of your patch being merged promptly.
> Getting an independent review and a Reviewed-by: line also helps a lot.
> 
> The files Documentation/SubmittingPatches and Documentation/SubmitChecklist
> in kernel source tree also have a lot of good advice that makes it
> easier for me to handle your patches.
> 
> DETAILS
> =======
> 
> Here is the complete list of patches I have in my for-2.6.24 branch:
> 
> Ali Ayoub (1):
>       IB/sa: Error handling thinko fix
> 
> Anton Blanchard (3):
>       IB/fmr_pool: Clean up some error messages in fmr_pool.c
>       IB/ehca: Make output clearer by removing some debug messages
>       IB/ehca: Export module parameters in sysfs
> 
> Dotan Barak (1):
>       mlx4_core: Use enum value GO_BIT_TIMEOUT_MSECS
> 
> Eli Cohen (2):
>       IPoIB: Fix typo to end statement with ';' instead of ','
>       IPoIB: Fix error path memory leak
> 
> Hoang-Nam Nguyen (4):
>       IB/ehca: Use remap_4k_pfn() to map firmware contexts to user space
>       IB/ehca: Fix large page HW cap defines
>       IB/ehca: Fix mem leak of firmware ctrlblock in ehca_create_srq()
>       IB/ehca: Adjust 64-bit alignment of create QP response for userspace
> 
> Jack Morgenstein (5):
>       IB/mlx4:  Display misc device information under /sys/class/infiniband/
>       mlx4_core: Support ICM tables in coherent memory
>       mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command
>       IB/mlx4: Implement FMRs
>       mlx4_core: Increase max number of QPs per multicast group to 56
> 
> Joachim Fenkes (11):
>       IB/ehca: Refactor hvcall tracing
>       IB/ehca: Print return codes as signed decimal integers
>       IB/ehca: ehca_gen_warn() should always print
>       IB/ehca: Add check for max #SGE to create_qp()
>       IB/ehca: Path migration support
>       IB/ehca: Serialize MR alloc and MR free hvCalls
>       IB/ehca: Replace get_paca()->paca_index by the more portable raw_smp_processor_id()
>       IB/ehca: Bump version number and change its format
>       IB/umem: Add hugetlb flag to struct ib_umem
>       IB/ehca: Only use MR large pages for hugetlb regions
>       IB/ehca: Return srq_attr->max_sge in ehca_query_srq()
> 
> Michael S. Tsirkin (2):
>       mlx4_core: Enable MSI-X by default
>       IB/mthca: Enable MSI-X by default
> 
> Peter Oruba (1):
>       IB/mthca: Use PCI-X/PCI-Express read control interfaces
> 
> Ralph Campbell (1):
>       IB/core: Fix handling of multicast response failures
> 
> Roland Dreier (14):
>       IPoIB: Make sure no receives are handled when stopping device
>       IB: find_first_zero_bit() takes unsigned pointer
>       mlx4_core: Don't free special QPs in QP number bitmap
>       IB/mlx4: Use __set_data_seg() in mlx4_ib_post_recv()
>       IB/ehca: Include <linux/mutex.h> from ehca_classes.h
>       IB/mlx4: Fix up SRQ limit_watermark endianness
>       IB/iser: Remove unnecessary includes
>       mlx4_core: Change capability decoding: SRC->XRC
>       IB/umad: Add P_Key index support
>       IB/umad: Fix bit ordering and 32-on-64 problems on big endian systems
>       IB/uverbs: Make ib_uverbs_release_event_file() static
>       mlx4_core: Reserve the correct number of MTT segments
>       mlx4_core: Fix meaning of dev->caps.reserved_mtts
>       IB/mthca: Increase max number of QPs per multicast group to 56
> 
> Satyam Sharma (1):
>       IB/ehca: Misc cpuinit section annotations and #ifdef cleanups
> 
> Sean Hefty (7):
>       IPoIB: Specify Traffic Class with path record queries for QoS support
>       IB/sa: Add new QoS fields to path record
>       RDMA/cma: Add ability to specify type of service
>       RDMA/ucma: Allow user space to set service type
>       IB/srp: Add QoS support through service ID
>       IB/cm: Modify interface to send MRAs in response to duplicate messages
>       RDMA/cma: Queue IB CM MRAs to avoid unnecessary remote retries
> 
> Stefan Roscher (2):
>       IB/ehca: Small QP userspace support
>       IB/ehca: Support more than 4k QPs for userspace and kernelspace
> 
> Steve Wise (2):
>       RDMA/cxgb3: Make the iw_cxgb3 module parameters writable
>       RDMA/cma: Use neigh_event_send() to start neighbour discovery
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Jack Morgenstein]

On Saturday 06 October 2007 01:18, Roland Dreier wrote:
>  - XRC.  Given the length of the backlog above and the fact that a
>    first draft of this code has not been posted yet, I don't see any
>    way that we could have something this major ready in time.
> 
I posted the first draft patch set to the OpenFabrics list on September 18:

[ofa-general] [PATCH 0 of 5] XRC implementation patches (libibverbs, libmlx4, core, mlx4)   Jack Morgenstein
[ofa-general] [PATCH 1 of 5] libibverbs: XRC implementation   Jack Morgenstein
[ofa-general] [PATCH 2 of 5] libmlx4: XRC implementation   Jack Morgenstein
[ofa-general] [PATCH 3 of 5] core: XRC implementation for fd = -1 when opening an xrc domain   Jack Morgenstein
[ofa-general] [PATCH 4 of 5] core: XRC implementation -- add support for working with file descriptors   Jack Morgenstein
[ofa-general] [PATCH 5 of 5] mlx4: XRC implementation   Jack Morgenstein

The above patch set implements XRC for userspace verbs layer and below.
The Kernel-space verbs implementation (1 more patch) is finished,
but as yet untested.

- Jack

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Roland Dreier]

 > >  - XRC.  Given the length of the backlog above and the fact that a
 > >    first draft of this code has not been posted yet, I don't see any
 > >    way that we could have something this major ready in time.
 > > 
 > I posted the first draft patch set to the OpenFabrics list on September 18:

Sorry, I didn't update that text.  But my backlog is still too big to
get XRC into 2.6.24.

 - R.

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Roland Dreier]

 > No mention about the iwarp port space issue?

I don't think we're at a stage where I'm prepared to merge something--
we all agree the latest patch has serious drawbacks, and it commits us
to a suboptimal interface that is userspace-visible.

 > I'm at a loss as to how to proceed.

Could we try to do some cleanups to the net core to make the alias
stuff less painful?  eg is there any sane way to make it possible for
a device that creates 'eth0' to also create an 'iw0' alias without an
assigning an address?

 - R.

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Steve Wise]

Roland Dreier wrote:
>  > No mention about the iwarp port space issue?
> 
> I don't think we're at a stage where I'm prepared to merge something--
> we all agree the latest patch has serious drawbacks, and it commits us
> to a suboptimal interface that is userspace-visible.
> 

Fair enough.

>  > I'm at a loss as to how to proceed.
> 
> Could we try to do some cleanups to the net core to make the alias
> stuff less painful?  eg is there any sane way to make it possible for
> a device that creates 'eth0' to also create an 'iw0' alias without an
> assigning an address?
> 

Well, "alias" interfaces really don't exist.  ethX:iw is really just 
adding a address record (struct in_ifaddr) to ethX.  So in the current 
core design, adding an alias without an address is really adding the 
alias with address 0.0.0.0.  And I think the core net code assumes if an 
in_ifaddr struct exists for a device, then its IP address is indeed valid.

So I think the changes wouldn't be small to enhance the design to add a 
concept of an alias interface.

I'll look into this more though.

Steve.

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Roland Dreier]

 > Roland, I submitted an updated patch incorporating some of Sean's comments within
 > a day or two. Rest of comments pertained to restructuring the code and adding 
 > some additional module parameters.
 > 
 > This would require more discussions since some of these had been already discussed
 > previously. We had decided upon this code structure after a lot of discussions and
 > incorporating these would be undoing some of that.

Can you give a link to your current final version of the patch?

Sean, what's your opinion of where we stand?

Since module parameters create a userspace-visible interface that we
are stuck with for a long time, we definitely have to get at least
that much right before merging.

 - R.

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Sean Hefty]

> Can you give a link to your current final version of the patch?
> 
> Sean, what's your opinion of where we stand?

Let me look back over the last version that was sent and reply back 
later today or tomorrow.  Several of my initial comments were on code 
structure.

> Since module parameters create a userspace-visible interface that we
> are stuck with for a long time, we definitely have to get at least
> that much right before merging.

I was taking a slightly different view of the design.  It would be nice 
to agree on whether SRQ should be separated from the QP type before 
merging upstream, even if the implementation doesn't immediately support 
all available options.

- Sean

Re: [ofa-general] Updated InfiniBand/RDMA merge plans for 2.6.24

[Pradeep Satyanarayana]

Roland Dreier wrote:
>  > Roland, I submitted an updated patch incorporating some of Sean's comments within
>  > a day or two. Rest of comments pertained to restructuring the code and adding 
>  > some additional module parameters.
>  > 
>  > This would require more discussions since some of these had been already discussed
>  > previously. We had decided upon this code structure after a lot of discussions and
>  > incorporating these would be undoing some of that.
> 
> Can you give a link to your current final version of the patch?
> 
Roland,

This is the link to the last one that I submitted on 09/18.

http://lists.openfabrics.org/pipermail/general/2007-September/040917.html

Pradeep