InfiniBand 2.6.17 merge plans

InfiniBand 2.6.17 merge plans

[Roland Dreier]

As we enter the second week of the 2.6.17 merge window, I thought it
might be helpful to give an update on what I'm thinking about the
remaining pending pieces.  This is really to stimulate discussion --
if there's something that you think really should (or shouldn't) be
merged, let me know.

Of course bug fixes are always welcome and can be merged pretty much
any time -- that's the whole point of the stabilization period after
the merge window.

 * PathScale ipath driver.  In my git tree at

   git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath

   There is still ia64 build breakage and a lot of sparse endian
   annotations to clean up before it's mergeable.  Also, there are two
   issues I still see:

   - Working around the IB midlayer SMA (subnet management agent) /
     implementation with a character device when ib_mad isn't loaded.
     Maybe I'm off-base here objecting to this.  Hal and Sean, as the
     ib_mad guys, I'd be especially interested in your opinion of this.

   - The driver depends on 64BIT && PCI_MSI, and is basically
     x86_64-only for practical purposes.  I think this is OK as far as
     a merge goes, but it would be nice to be able to use a PCIe
     device on any system with PCIe slots...

 * RDMA CM.  In my git tree at

   git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

   I think this is OK to merge, but I don't see much pull to get it in
   right now.  There are three consumers on the horizon:

   - userspace RDMA CM, which exports the abstraction to userspace.
     The feeling is that this interface needs more time to mature.

   - NFS/RDMA.  Not ready to merge right now.

   - iSER.  Maybe ready to merge -- I haven't heard anything recently.

   My feeling is that without someone making a case for why this
   should go in now, I'm going to hold off.

 * IPoIB tunables.  This is probably OK but I haven't seen any patches
   yet, so we're running out time.

 * SRP FMRs.  I have a patch that I'm pretty happy with now, so I
   think this will go in.  This will need a lot of testing, or we may
   have to turn it off by default for 2.6.17 final

So if you care about any of this, let me know what you think.  And if
there's something not on this list and not in

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-2.6.17

please make sure I know about it, or it won't get merged.

Thanks,
  Roland

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Sean Hefty]

Roland Dreier wrote:
>    - Working around the IB midlayer SMA (subnet management agent) /
>      implementation with a character device when ib_mad isn't loaded.
>      Maybe I'm off-base here objecting to this.  Hal and Sean, as the
>      ib_mad guys, I'd be especially interested in your opinion of this.

I will need to spend some time reading through the PathScale code on this.  I've 
  followed most of the discussion, though not in detail.  Right now, I don't see 
what's being gained by not loading ib_mad.  For instance, how does the driver 
handle loading ib_mad later, say when another IB device is added to the system 
that requires it?

>  * RDMA CM.  In my git tree at
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm
> 
>    I think this is OK to merge, but I don't see much pull to get it in
>    right now.  There are three consumers on the horizon:
> 
>    - userspace RDMA CM, which exports the abstraction to userspace.
>      The feeling is that this interface needs more time to mature.
> 
>    - NFS/RDMA.  Not ready to merge right now.
> 
>    - iSER.  Maybe ready to merge -- I haven't heard anything recently.

I agree that we need to let the userspace interface mature.  And even the kernel 
interface could benefit from having some real users.  The code was added to the 
-mm branch, correct?

The main drawback that we have not merging the RDMA CM with userspace support, 
is that OpenIB release 1.0 will have very poor connection management support. 
There's no easy way for userspace applications to obtain path records.

- Sean

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Sean> I agree that we need to let the userspace interface mature.
    Sean> And even the kernel interface could benefit from having some
    Sean> real users.  The code was added to the -mm branch, correct?

Yes, I've put both the rdma_cm and ipath branches into my for-mm
branch, so Andrew will pick them up automatically.

 - R.

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Shirley> How soon do you need this patch? I will try my best to
    Shirley> catch up the schedule.

Theoretically Linus's merge window ends this weekend.  So, depending
on how invasive this ends up being, it probably needs to happen in the
next few days to have a chance at getting merged.

 - R.

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

BTW, what do you think of changing rdma_wq to be a GPL export, to give
a hint that this is an internal symbol not really for general use?

In other words:

diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index 6ebd1d1..cdf4888 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -56,7 +56,7 @@ static DEFINE_MUTEX(lock);
 static LIST_HEAD(req_list);
 static DECLARE_WORK(work, process_req, NULL);
 struct workqueue_struct *rdma_wq;
-EXPORT_SYMBOL(rdma_wq);
+EXPORT_SYMBOL_GPL(rdma_wq);
 
 static int copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
 		     unsigned char *dst_dev_addr)

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Sean Hefty]

Roland Dreier wrote:
> BTW, what do you think of changing rdma_wq to be a GPL export, to give
> a hint that this is an internal symbol not really for general use?

I'm actually testing a patch set now that moves rdma_wq internal to ib_addr, and 
  gives the RDMA CM its own WQ.

- Sean

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Sean> I'm actually testing a patch set now that moves rdma_wq
    Sean> internal to ib_addr, and gives the RDMA CM its own WQ.

OK, that's better still.

BTW ib_local_sa also uses rdma_wq ... it might be worth fixing that,
because it actually makes ib_local_sa depend on CONFIG_NET right now
(since ib_addr can't build without CONFIG_NET).

Not that anyone is building kernels without CONFIG_NET...

 - R.

RE: [openib-general] InfiniBand 2.6.17 merge plans

[Sean Hefty]

>BTW ib_local_sa also uses rdma_wq ... it might be worth fixing that,
>because it actually makes ib_local_sa depend on CONFIG_NET right now
>(since ib_addr can't build without CONFIG_NET).

Yes - I've fixed that as well.

- Sean

RE: [openib-general] InfiniBand 2.6.17 merge plans

[Bob Woodruff]

Roland wrote, 
>   - userspace RDMA CM, which exports the abstraction to userspace.
>     The feeling is that this interface needs more time to mature.

The userspace RDMA-CM is currently being used by uDAPL and seems to 
work fine for that application. Our testing so far shows that the code is
stable. However, uDAPL also has a socket based mechanism that can used until
the CMA 
is merged. If others think it still needs more time to mature, then uDAPL
has an interim solution.

my 2 cents on this one.

woody

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Dan Bar Dov]

On 3/27/06, Roland Dreier <rdreier@cisco.com> wrote:
> As we enter the second week of the 2.6.17 merge window, I thought it
> might be helpful to give an update on what I'm thinking about the
> remaining pending pieces.  This is really to stimulate discussion --
> if there's something that you think really should (or shouldn't) be
> merged, let me know.
<snip>
>    - iSER.  Maybe ready to merge -- I haven't heard anything recently.
>

iSER needs to complete the removal of the dummy socket provider
and change the open-iscsi user-space interface.
We are coordinating this effort with open-iscsi, and rely
on some open-iscsi changes that are planned to be pushed to 2.6.18.

Therefore the plan for iSER is to push it into 2.6.18

Dan

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Dan> Therefore the plan for iSER is to push it into 2.6.18

OK, thanks.

Based on all of this I'm thinking that it's better to hold back the
RDMA CM as well, since there will be no code using it ready to merge
for 2.6.17.

 - R.

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Bryan O'Sullivan]

On Mon, 2006-03-27 at 11:56 -0800, Roland Dreier wrote:

>  * PathScale ipath driver.  In my git tree at
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath
> 
>    There is still ia64 build breakage and a lot of sparse endian
>    annotations to clean up before it's mergeable.

These are all fixed now.

>    - Working around the IB midlayer SMA (subnet management agent) /
>      implementation with a character device when ib_mad isn't loaded.

We dropped this in the round of changes I just submitted.

>    - The driver depends on 64BIT && PCI_MSI, and is basically
>      x86_64-only for practical purposes.  I think this is OK as far as
>      a merge goes, but it would be nice to be able to use a PCIe
>      device on any system with PCIe slots...

We'll see what we can do about this at some point, but I don't think it
should block the merge.

	<b

updated InfiniBand 2.6.17 merge plans

[Roland Dreier]

OK, here's a quick update on 2.6.17 merge plans:

 * PathScale ipath driver.  In my git tree at

   git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git ipath

   The new version looks good to me.  I'll merge it unless I hear an
   objection to the latest code.

 * RDMA CM.  In my git tree at

   git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm

   None of the users of this code look are to merge, and it looks like
   there's some changes in the design happening now.  Seems like this
   can and should wait for 2.6.18.

 * IPoIB tunable ring sizes.  Still no patch yet.

 * SRP FMRs.  I have a patch that I like, but it's not totally stable.
   I may be hitting target bugs (ie in someone else's code).  On the
   other hand I don't have any numbers showing a benefit, so I'm not
   sure if it's worth merging this.

 * (new since last time) Improved static rate handling.  I will get
   this in.

As before, if you care about any of this, let me know what you think.
And if there's something not on this list and not in

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-2.6.17

please make sure I know about it, or it won't get merged.

Thanks,
  Roland

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

[Roland Dreier]

Oh yeah, one other thing I plan to merge unless someone objects:

 * Get rid of option for building IPoIB and mthca debug output unless EMBEDDED=y

 - R.

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

[Sean Hefty]

Roland Dreier wrote:
>  * RDMA CM.  In my git tree at
> 
>    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git rdma_cm
> 
>    None of the users of this code look are to merge, and it looks like
>    there's some changes in the design happening now.  Seems like this
>    can and should wait for 2.6.18.

Does it make sense to merge the first two patches of that patch series?

1. Provide common handling for marshalling data between userspace clients and
kernel mode Infiniband drivers.

2. Extend the Infiniband CM to include private data comparisons as part of its
connection request matching process.

This would make it easier to keep the openib tree in sync with the kernel.

- Sean

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 299 bytes --]

On Thu, Mar 30, 2006 at 03:11:17PM -0800, Roland Dreier wrote:
> Oh yeah, one other thing I plan to merge unless someone objects:
> 
>  * Get rid of option for building IPoIB and mthca debug output unless EMBEDDED=y

NACK.  Just add a FOO_DEBUG config option, this has noþhing to do with
EMBEDDED.

Re: [openib-general] updated InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Roland> * Get rid of option for building IPoIB and mthca debug
    Roland> output unless EMBEDDED=y

    Christoph> NACK.  Just add a FOO_DEBUG config option, this has
    Christoph> noþhing to do with EMBEDDED.

I think you misunderstood (probably because what I wrote was very
unclear).  What I meant to say was that I wanted to make
CONFIG_INFINIBAND_IPOIB_DEBUG=y, without the option to disable it,
unless CONFIG_EMBEDDED is set.  In Kconfig language, change it to:

	config INFINIBAND_IPOIB_DEBUG
		bool "IP-over-InfiniBand debugging" if EMBEDDED
		depends on INFINIBAND_IPOIB
		default y

Do you still have a problem with that?  The rationale is that we want
the debugging output compiled in (it still needs to be enabled at
runtime) unless the user really really knows they don't want it,
because by the time we know we want the output, it's too late to
recompile to kernel.

 - R.

Re: updated InfiniBand 2.6.17 merge plans

[Michael S. Tsirkin]

Quoting r. Roland Dreier <rdreier@cisco.com>:
> Subject: updated InfiniBand 2.6.17 merge plans
> 
> OK, here's a quick update on 2.6.17 merge plans:

These are just the features, right? There's as the usual
bugfixes which aren't listed here.

-- 
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies

Re: updated InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Michael> These are just the features, right? There's as the usual
    Michael> bugfixes which aren't listed here.

Yes, of course.  We can always merge bugfixes.  Although I don't know
of any pending bugfixes that are not at least in my for-2.6.17 branch...

 - R.

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Dan Bar Dov]

On the other hand, there are user-space consumers for CMA,
such as udapl, so although ISER will attempt only at 2.6.18,
maybe it is a good idea to get CMA into 2.6.17

Dan

On 3/29/06, Roland Dreier <rdreier@cisco.com> wrote:
>     Dan> Therefore the plan for iSER is to push it into 2.6.18
>
> OK, thanks.
>
> Based on all of this I'm thinking that it's better to hold back the
> RDMA CM as well, since there will be no code using it ready to merge
> for 2.6.17.
>
>  - R.
>

Re: [openib-general] InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Dan> On the other hand, there are user-space consumers for CMA,
    Dan> such as udapl, so although ISER will attempt only at 2.6.18,
    Dan> maybe it is a good idea to get CMA into 2.6.17

Well, too late now anyway.

But Sean and I agreed that the userspace interface to CMA was not
mature enough to merge for 2.6.17.

 - R.

Re: updated InfiniBand 2.6.17 merge plans

[Michael S. Tsirkin]

Quoting r. Roland Dreier <rdreier@cisco.com>:
> Subject: Re: updated InfiniBand 2.6.17 merge plans
> 
>     Michael> These are just the features, right? There's as the usual
>     Michael> bugfixes which aren't listed here.
> 
> Yes, of course.  We can always merge bugfixes.  Although I don't know
> of any pending bugfixes that are not at least in my for-2.6.17 branch...

One thing I wanted to do was kill the global mutexes in
ipoib, replacing all uses with priv->lock: we never do anything blocking
there. Is this 2.6.17 material?

-- 
Michael S. Tsirkin
Staff Engineer, Mellanox Technologies

Re: updated InfiniBand 2.6.17 merge plans

[Roland Dreier]

    Michael> One thing I wanted to do was kill the global mutexes in
    Michael> ipoib, replacing all uses with priv->lock: we never do
    Michael> anything blocking there. Is this 2.6.17 material?

I don't have a strong feeling either way.  It doesn't fix anything,
but on the other hand it's a small low-risk cleanup.

 - R.