[PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Chien Tung]

Current iWARP implementations make use of the TCP port space for iWARP
connections without synchornizing with the host TCP stack.  This can
lead to both host TCP and offloaded iWARP connections for the iWARP
interface to use the same TCP port.  The combined traffic will result
in undesirable behaviors from connection failures to resets.

This patch resolves the issue by allocating a kernel socket consuming
the TCP port for the iWARP connection upon connection creation and
freeing the socket once the iWARP connection is destroyed.

This functionality is needed for iWARP adapters that are both a L2 and
offload iWARP device.  Applications can create and use host TCP and
iWARP connections over the same interface without additional
administrative overhead and complication needed to manage TCP port space
manually.

Signed-off-by: Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Chien Tung <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
V2 changes:

*  Fix compilation warning in cma_0100_unified_tcp_ports.patch

   Signed-off-by: Vladimir Sokolovsky <vlad-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>

  -			ip_addr_size(&id_priv->id.route.addr.src_addr));
  +			ip_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr));


*  IB/core: fix build warning

   Eliminates the following build warning:

      CC [M]  ./drivers/infiniband/core/cma.o
    ./drivers/infiniband/core/cma.c: In function 'cma_get_tcp_port':
    ./drivers/infiniband/core/cma.c:2076: warning: passing argument 1 of 'ip_addr_size' from incompatible pointer type

    Signed-off-by: John Gregor <john.gregor-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org>
    Signed-off-by: Yannick Cote <yannick.cote-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org>

  -	size = ip_addr_size(&id_priv->id.route.addr.src_addr);
  +	size = ip_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr);

* Add getname

  Signed-off-by: Faisal Latif <faisal.latif-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

* Change default unify_tcp_port_space to 1

  Signed-off-by: Chien Tung <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

 drivers/infiniband/core/cma.c |   46 ++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 45 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index b930b81..d5855f9 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -56,6 +56,11 @@ MODULE_AUTHOR("Sean Hefty");
 MODULE_DESCRIPTION("Generic RDMA CM Agent");
 MODULE_LICENSE("Dual BSD/GPL");
 
+int unify_tcp_port_space = 1;
+module_param(unify_tcp_port_space, int, 0644);
+MODULE_PARM_DESC(unify_tcp_port_space, "Unify the host TCP and RDMA port "
+		 "space allocation (default=1)");
+
 #define CMA_CM_RESPONSE_TIMEOUT 20
 #define CMA_MAX_CM_RETRIES 15
 #define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24)
@@ -118,6 +123,7 @@ struct rdma_id_private {
 	struct rdma_cm_id	id;
 
 	struct rdma_bind_list	*bind_list;
+	struct socket		*sock;
 	struct hlist_node	node;
 	struct list_head	list; /* listen_any_list or cma_device.list */
 	struct list_head	listen_list; /* per device listens */
@@ -806,6 +812,8 @@ static void cma_release_port(struct rdma_id_private *id_priv)
 		kfree(bind_list);
 	}
 	mutex_unlock(&lock);
+	if (id_priv->sock)
+		sock_release(id_priv->sock);
 }
 
 static void cma_leave_mc_groups(struct rdma_id_private *id_priv)
@@ -2035,6 +2043,37 @@ static int cma_use_port(struct idr *ps, struct rdma_id_private *id_priv)
 	return 0;
 }
 
+static int cma_get_tcp_port(struct rdma_id_private *id_priv)
+{
+	int ret;
+	int size;
+	struct socket *sock;
+
+	ret = sock_create_kern(AF_INET, SOCK_STREAM, IPPROTO_TCP, &sock);
+	if (ret)
+		return ret;
+
+	size = ip_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr);
+
+	ret = sock->ops->bind(sock,
+			(struct sockaddr *) &id_priv->id.route.addr.src_addr,
+			size);
+	if (ret)
+		goto err;
+
+	ret = sock->ops->getname(sock,
+			(struct sockaddr *) &id_priv->id.route.addr.src_addr,
+			&size, 0);
+	if (ret)
+		goto err;
+
+	id_priv->sock = sock;
+	return 0;
+err:
+	sock_release(sock);
+	return ret;
+}
+
 static int cma_get_port(struct rdma_id_private *id_priv)
 {
 	struct idr *ps;
@@ -2046,6 +2085,11 @@ static int cma_get_port(struct rdma_id_private *id_priv)
 		break;
 	case RDMA_PS_TCP:
 		ps = &tcp_ps;
+		if (unify_tcp_port_space) {
+			ret = cma_get_tcp_port(id_priv);
+			if (ret)
+				goto out;
+		}
 		break;
 	case RDMA_PS_UDP:
 		ps = &udp_ps;
@@ -2063,7 +2107,7 @@ static int cma_get_port(struct rdma_id_private *id_priv)
 	else
 		ret = cma_use_port(ps, id_priv);
 	mutex_unlock(&lock);
-
+out:
 	return ret;
 }
 
-- 
1.6.4.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > This patch resolves the issue by allocating a kernel socket consuming
 > the TCP port for the iWARP connection upon connection creation and
 > freeing the socket once the iWARP connection is destroyed.

Hasn't this exact approach been explicitly NAK'ed by David Miller?

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Or Gerlitz]

On Fri, Jun 11, 2010 at 3:47 PM, Chien Tung <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> V2 changes:

What you consider to be V1, this thread from 2007?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

>  > This patch resolves the issue by allocating a kernel socket consuming
>  > the TCP port for the iWARP connection upon connection creation and
>  > freeing the socket once the iWARP connection is destroyed.
> 
> Hasn't this exact approach been explicitly NAK'ed by David Miller?

Yes but this approach still is the simplest and cleanest way to solve this
problem that I am aware of.

I do want to make clear on the network resource usage.  For example, if
you run an MPI job over host TCP stack and the job creates 4 connections
thus consuming 4 sockets.  With this patch, the same job over iWARP will
consume 4 sockets.  This patch has zero impact on non-iWARP applications
and only consumes equal amount of network resources.

I would like this patch to be reconsidered for inclusion into the kernel.

Chien



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> > V2 changes:
> 
> What you consider to be V1, this thread from 2007?

Yes.  V2 has changes from OFED plus a few more things.

Chien


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Tung, Chien Tin wrote:
>>  > This patch resolves the issue by allocating a kernel socket consuming
>>  > the TCP port for the iWARP connection upon connection creation and
>>  > freeing the socket once the iWARP connection is destroyed.
>>
>> Hasn't this exact approach been explicitly NAK'ed by David Miller?
>>     
>
> Yes but this approach still is the simplest and cleanest way to solve this
> problem that I am aware of.
>
> I do want to make clear on the network resource usage.  For example, if
> you run an MPI job over host TCP stack and the job creates 4 connections
> thus consuming 4 sockets.  With this patch, the same job over iWARP will
> consume 4 sockets.  This patch has zero impact on non-iWARP applications
> and only consumes equal amount of network resources.
>
> I would like this patch to be reconsidered for inclusion into the kernel.
>
> Chien
>
>   

You can peruse this thread from 2007:

http://lkml.org/lkml/2007/8/15/174

In that thread David Miller said he would NAK this solution and even NAK 
a solution where we expose the low level port allocation services and 
allow RDMA/iWARP and TCP to share the port space.  Thus I see no way to 
resolve this issue with Dave's ACK, and maybe the RDMA contributors and 
maintainers should move forward without it?


My 2 cents.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > You can peruse this thread from 2007:
 > 
 > http://lkml.org/lkml/2007/8/15/174
 > 
 > In that thread David Miller said he would NAK this solution and even
 > NAK a solution where we expose the low level port allocation services
 > and allow RDMA/iWARP and TCP to share the port space.  Thus I see no
 > way to resolve this issue with Dave's ACK, and maybe the RDMA
 > contributors and maintainers should move forward without it?

If Dave Miller says this is an unacceptable thing to merge, then I'm not
going to merge it over his explicit and unequivocal NAK.  Trying sneak
this in by hiding this patch on linux-rdma without cc-ing netdev is the
wrong approach... the right approach is to get the hordes of users being
hurt by this issue (if there are such hordes), make a big stink, and
force a solution that everyone agrees on.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Fri, Jun 11, 2010 at 03:08:20PM -0700, Roland Dreier wrote:

> If Dave Miller says this is an unacceptable thing to merge, then I'm not
> going to merge it over his explicit and unequivocal NAK.  Trying sneak
> this in by hiding this patch on linux-rdma without cc-ing netdev is the
> wrong approach... the right approach is to get the hordes of users being
> hurt by this issue (if there are such hordes), make a big stink, and
> force a solution that everyone agrees on.

Right..

What exactly is the end-user harm anyhow? Everything works 100%
correctly without this patch, and your arguments all revolved around
diagnostic capability..

It is a known weakness that the RDMA stack has crappy visibility. That
has been talked about before. If you want better visibility then fix
that by adding netlink stuff to dump out all the QP states, PDs, etc,
similar to what TCP does.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Hefty, Sean]

> What exactly is the end-user harm anyhow? Everything works 100%
> correctly without this patch, and your arguments all revolved around
> diagnostic capability..

OFED includes this patch, which was picked up by the distros.  I'm not sure iWarp customers are running without it.

> It is a known weakness that the RDMA stack has crappy visibility. That
> has been talked about before. If you want better visibility then fix
> that by adding netlink stuff to dump out all the QP states, PDs, etc,
> similar to what TCP does.

This is a separate topic from what the patch is trying to address.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Fri, Jun 11, 2010 at 03:47:55PM -0700, Hefty, Sean wrote:
> > What exactly is the end-user harm anyhow? Everything works 100%
> > correctly without this patch, and your arguments all revolved around
> > diagnostic capability..
> 
> OFED includes this patch, which was picked up by the distros.  I'm
> not sure iWarp customers are running without it.

Sigh. That's unconscionable. :(
 
> > It is a known weakness that the RDMA stack has crappy visibility. That
> > has been talked about before. If you want better visibility then fix
> > that by adding netlink stuff to dump out all the QP states, PDs, etc,
> > similar to what TCP does.
> 
> This is a separate topic from what the patch is trying to address.

It isn't if that is all the patch is trying to address...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Fri, Jun 11, 2010 at 03:08:20PM -0700, Roland Dreier wrote:
>
>   
>> If Dave Miller says this is an unacceptable thing to merge, then I'm not
>> going to merge it over his explicit and unequivocal NAK.  Trying sneak
>> this in by hiding this patch on linux-rdma without cc-ing netdev is the
>> wrong approach... the right approach is to get the hordes of users being
>> hurt by this issue (if there are such hordes), make a big stink, and
>> force a solution that everyone agrees on.
>>     
>
> Right..
>
> What exactly is the end-user harm anyhow? Everything works 100%
> correctly without this patch, and your arguments all revolved around
> diagnostic capability..
>
>   

No, everything does not work without this patch.  Go read the thread 
from 2007 (I posted the URL to it on this thread).   I describe exactly 
how an application like MPI will get incorrect behavior due to port 
space collisions on iWARP HW.


> It is a known weakness that the RDMA stack has crappy visibility. That
> has been talked about before. If you want better visibility then fix
> that by adding netlink stuff to dump out all the QP states, PDs, etc,
> similar to what TCP does.
>   

What does this have to do with port space allocation?  I agree the rdma 
stacks should have tools to dump all that good stuff out for 
admins/debugging.  But its really not part of this issue IMO.

Steve.


> Jason
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > You can peruse this thread from 2007:
>  > 
>  > http://lkml.org/lkml/2007/8/15/174
>  > 
>  > In that thread David Miller said he would NAK this solution and even
>  > NAK a solution where we expose the low level port allocation services
>  > and allow RDMA/iWARP and TCP to share the port space.  Thus I see no
>  > way to resolve this issue with Dave's ACK, and maybe the RDMA
>  > contributors and maintainers should move forward without it?
>
> If Dave Miller says this is an unacceptable thing to merge, then I'm not
> going to merge it over his explicit and unequivocal NAK.  Trying sneak
> this in by hiding this patch on linux-rdma without cc-ing netdev is the
> wrong approach... the right approach is to get the hordes of users being
> hurt by this issue (if there are such hordes), make a big stink, and
> force a solution that everyone agrees on.
>
>   

I hear ya.

It seems to be difficult to get end users to horde up and stink about 
these sorts of technical issues, because they don't have the background 
to argue their case.   And technical merit alone isn't working here.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Fri, Jun 11, 2010 at 06:40:33PM -0500, Steve Wise wrote:

> No, everything does not work without this patch.  Go read the thread  
> from 2007 (I posted the URL to it on this thread).   I describe exactly  
> how an application like MPI will get incorrect behavior due to port  
> space collisions on iWARP HW.

I guess I missed the fact that OFED included this patch, rather than
relying on the other administratively configured schemes you
originally outlined. I was opining that nobody was complaining about
the administrative stuff.. but of course nobody complains, they don't
have to do it :(

A seperate IP for iWarp always seemed to me to be the way forward on
this. And I do mean seperate, as in TCP packets for this IP are never
sent into the net stack seperate :|

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Fri, Jun 11, 2010 at 06:40:33PM -0500, Steve Wise wrote:
>
>   
>> No, everything does not work without this patch.  Go read the thread  
>> from 2007 (I posted the URL to it on this thread).   I describe exactly  
>> how an application like MPI will get incorrect behavior due to port  
>> space collisions on iWARP HW.
>>     
>
> I guess I missed the fact that OFED included this patch, rather than
> relying on the other administratively configured schemes you
> originally outlined. I was opining that nobody was complaining about
> the administrative stuff.. but of course nobody complains, they don't
> have to do it :(
>
> A seperate IP for iWarp always seemed to me to be the way forward on
> this. And I do mean seperate, as in TCP packets for this IP are never
> sent into the net stack seperate :|
>
>   
 From a user perspective, separate IP addresses for iWARP is terrible.  
Further, its not even needed for IB, just iWARP.  That's an unnecessary 
admin pain IMO.  iWARP should and can easily co-exist with the host TCP 
by sharing the port space.  But, as Roland stated already, maybe the 
only way forward it to get end-user pressure applied at the appropriate 
places! :)

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Fri, Jun 11, 2010 at 08:18:43PM -0500, Steve Wise wrote:

> Further, its not even needed for IB, just iWARP.  That's an unnecessary
> admin pain IMO.

IB already completely seperates from the host stack, that is why it
isn't effected by this problem. It has both a seperate port numbering
space (in rdma cm) and a separate addressing space (GID).

The entire problem with iWARP is that it is trying to not be
seperate, unlike IB. So.. simple answer: use a seperate IP, or use a
seperate port space (ie don't use the TCP protocol number).

ROCEE won't have this problem either..

> iWARP should and can easily co-exist with the host TCP by sharing
> the port space.  But, as Roland stated already, maybe the only way
> forward it to get end-user pressure applied at the appropriate
> places! :)

*shrug* This isn't going to happen until netdev decides to design-in
statefull offload. I doubt that is going to happen any time
soon. I've already seen Linux max out 40GE on benchmarks, so it is
hard to see what the driver would be.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Fri, Jun 11, 2010 at 08:18:43PM -0500, Steve Wise wrote:
>
>   
>> Further, its not even needed for IB, just iWARP.  That's an unnecessary
>> admin pain IMO.
>>     
>
> IB already completely seperates from the host stack, that is why it
> isn't effected by this problem. It has both a seperate port numbering
> space (in rdma cm) and a separate addressing space (GID).
>
>   

By virtue of being a different L4 transport.   iWARP uses TCP as the L4 
and thus has these issues.

Also, from an application perspective,  IB has IP addresses that are 
shared with the TCP stack and the RDMA stack.  So it still appears as 
integrated.  At least with librdmacm applications...

> The entire problem with iWARP is that it is trying to not be
> seperate, unlike IB. So.. simple answer: use a seperate IP, or use a
> seperate port space (ie don't use the TCP protocol number).
>   


Change the iWARP specification?  I don't think that's a simple answer. :)


> ROCEE won't have this problem either..
>
>   
>> iWARP should and can easily co-exist with the host TCP by sharing
>> the port space.  But, as Roland stated already, maybe the only way
>> forward it to get end-user pressure applied at the appropriate
>> places! :)
>>     
>
> *shrug* This isn't going to happen until netdev decides to design-in
> statefull offload. I doubt that is going to happen any time
> soon. I've already seen Linux max out 40GE on benchmarks, so it is
> hard to see what the driver would be.
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Peter P Waskiewicz Jr]

On Fri, 11 Jun 2010, Steve Wise wrote:

> Jason Gunthorpe wrote:
>> On Fri, Jun 11, 2010 at 08:18:43PM -0500, Steve Wise wrote:
>>
>>
>>> Further, its not even needed for IB, just iWARP.  That's an unnecessary
>>> admin pain IMO.
>>>
>>
>> IB already completely seperates from the host stack, that is why it
>> isn't effected by this problem. It has both a seperate port numbering
>> space (in rdma cm) and a separate addressing space (GID).
>>
>>
>
> By virtue of being a different L4 transport.   iWARP uses TCP as the L4
> and thus has these issues.
>
> Also, from an application perspective,  IB has IP addresses that are
> shared with the TCP stack and the RDMA stack.  So it still appears as
> integrated.  At least with librdmacm applications...
>
>> The entire problem with iWARP is that it is trying to not be
>> seperate, unlike IB. So.. simple answer: use a seperate IP, or use a
>> seperate port space (ie don't use the TCP protocol number).
>>
>

That's right.  iWARP isn't trying to be seperate.  It's trying to converge 
traffic on a single link.  People are doing this all over the place with 
Data Center Bridging (IEEE 802.1Qaz).  It results in easier data center 
deployments, less switches, less cables, less cost, etc., etc.

The fundamental problem here is certain people in the networking community 
don't believe in RDMA or iWARP.  This leads to exactly what's going on 
here, which is the lack of discussion to come up with a better approach if 
people are opposed to this one.  The iWARP community is highlighting a 
limitation with how the protocol is designed, they're trying to fix it, 
and the response is "don't do that."  We need technical guidance if people 
are opposed.

Other protocols are also running over networking today, such as iSCSI and 
FCoE.  These happily co-exist with other L2->L4 protocols in the stack. 
This iWARP patch allows iWARP to happily co-exist on a TCP connection, and 
does *not* negatively affect the networking stack at all.

> Change the iWARP specification?  I don't think that's a simple answer. :)
>
>
>> ROCEE won't have this problem either..

Obviously...it's running over Ethernet.  This doesn't help the discussion 
at hand one bit.  It's comparing apples to oranges.

>>> iWARP should and can easily co-exist with the host TCP by sharing
>>> the port space.  But, as Roland stated already, maybe the only way
>>> forward it to get end-user pressure applied at the appropriate
>>> places! :)
>>>
>>
>> *shrug* This isn't going to happen until netdev decides to design-in
>> statefull offload. I doubt that is going to happen any time
>> soon. I've already seen Linux max out 40GE on benchmarks, so it is
>> hard to see what the driver would be.

I beg to differ.  I work in the Ethernet division at Intel, and have been 
working with 10GbE devices for the past few years.  I can easily scale 
Nehalem-EX systems beyond 100 GbE with hand-tuning of NUMA alignment, 
interrupt affinity, and flow/application affinity.  Linux is more than 
capable of scaling.  The question isn't about netdev adding stateful 
offloads; this patch isn't doing anything like that one bit.  The iWARP 
hardware knows which ports are in use for iWARP connections, and delivers 
those packets to the QP's that need it.  Those then interact directly with 
the RDMA stack.  No hooks in netdev needed.

The base argument is Dave Miller doesn't like RDMA or iWARP.  I completely 
respect Dave's opinions, but in this case, I don't think it's his call, 
since this patch is completely isolated to the IB tree and the RDMA stack. 
Not including this will directly affect anyone using iWARP with Linux. 
Hopefully those people will chime in at some point, but I'm not sure that 
will help because the arguments so far against this have been somewhat 
irrational, and nothing technical-oriented.

Cheers,
-PJ Waskiewicz
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > Other protocols are also running over networking today, such as iSCSI
 > and FCoE.  These happily co-exist with other L2->L4 protocols in the
 > stack. This iWARP patch allows iWARP to happily co-exist on a TCP
 > connection, and does *not* negatively affect the networking stack at
 > all.

How do iSCSI offload HBAs coexist?  As I understand it, they typically
just choose a separate IP address.

In any case I'm not going to slip in a patch that another maintainer has
explicitly NAKed.  Maybe one way to force things forward would be to
write up an exhaustive explanation of the underlying problem and the
impact on end users, include this patch, explain that it touches only
RDMA code, and point out that most end users are already using this
patch since it's shipped in OFED.  Then send the whole thing to Linus
and Andrew Morton, making sure to cc Dave Miller, netdev, and
linux-rdma.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Fri, Jun 11, 2010 at 09:34:44PM -0700, Peter P Waskiewicz Jr wrote:

> The base argument is Dave Miller doesn't like RDMA or iWARP.  I completely 
> respect Dave's opinions, but in this case, I don't think it's his call, 
> since this patch is completely isolated to the IB tree and the RDMA stack. 
> Not including this will directly affect anyone using iWARP with Linux.
> Hopefully those people will chime in at some point, but I'm not sure that 
> will help because the arguments so far against this have been somewhat 
> irrational, and nothing technical-oriented.

My impression of the general netdev argument is that if it touches
sockets it has to have 100% of the behavior and functionality of the
Linux stack. iwarp fails that test.. There is absolutely no way to fix
that - that is rational, technical and not at all 'solution oriented' :)

The only reason iwarp has got as far as it has in the offical tree is
because it piggy backed on IB's 'sandbox' approach to the RDMA
layer. So, if you want to play in that area you have to play by those
rules, and that means staying in the sandbox ..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Pradeep Satyanarayana]

Roland Dreier wrote:
>  > Other protocols are also running over networking today, such as iSCSI
>  > and FCoE.  These happily co-exist with other L2->L4 protocols in the
>  > stack. This iWARP patch allows iWARP to happily co-exist on a TCP
>  > connection, and does *not* negatively affect the networking stack at
>  > all.
> 
> How do iSCSI offload HBAs coexist?  As I understand it, they typically
> just choose a separate IP address.
> 
> In any case I'm not going to slip in a patch that another maintainer has
> explicitly NAKed.  Maybe one way to force things forward would be to
> write up an exhaustive explanation of the underlying problem and the
> impact on end users, include this patch, explain that it touches only
> RDMA code, and point out that most end users are already using this
> patch since it's shipped in OFED.  Then send the whole thing to Linus
> and Andrew Morton, making sure to cc Dave Miller, netdev, and
> linux-rdma.

Would it help to weigh in on the usage of RDMA, not so much a niche these days.
The latest top500 list shows 207 systems (i.e. 41%) using RDMA (read IB)?

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> > The base argument is Dave Miller doesn't like RDMA or iWARP.  I completely
> > respect Dave's opinions, but in this case, I don't think it's his call,
> > since this patch is completely isolated to the IB tree and the RDMA stack.
> > Not including this will directly affect anyone using iWARP with Linux.
> > Hopefully those people will chime in at some point, but I'm not sure that
> > will help because the arguments so far against this have been somewhat
> > irrational, and nothing technical-oriented.
> 
> My impression of the general netdev argument is that if it touches
> sockets it has to have 100% of the behavior and functionality of the
> Linux stack. iwarp fails that test.. There is absolutely no way to fix
> that - that is rational, technical and not at all 'solution oriented' :)

The socket is behaving 100% as in the Linux stack.  It is not even used for
sending or receiving traffic.  Allocating a socket is the secondary solution.
The problem is we need a way to reserve a TCP port.  We are not interested in
mucking with networking resources anymore than we have to.

>The only reason iwarp has got as far as it has in the offical tree is
>because it piggy backed on IB's 'sandbox' approach to the RDMA
>layer. So, if you want to play in that area you have to play by those
>rules, and that means staying in the sandbox ..

Can someone give me a history lesson here?  When/where is the "sandbox" agreement?
I am still confused as to why a patch to an infiniband module has to be sent
to netdev for approval instead of openly to linux-rdma!?

Chien


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > Other protocols are also running over networking today, such as iSCSI
>  > and FCoE.  These happily co-exist with other L2->L4 protocols in the
>  > stack. This iWARP patch allows iWARP to happily co-exist on a TCP
>  > connection, and does *not* negatively affect the networking stack at
>  > all.
>
> How do iSCSI offload HBAs coexist?  As I understand it, they typically
> just choose a separate IP address.
>
> In any case I'm not going to slip in a patch that another maintainer has
> explicitly NAKed.  Maybe one way to force things forward would be to
> write up an exhaustive explanation of the underlying problem and the
> impact on end users, include this patch, explain that it touches only
> RDMA code, and point out that most end users are already using this
> patch since it's shipped in OFED.  Then send the whole thing to Linus
> and Andrew Morton, making sure to cc Dave Miller, netdev, and
> linux-rdma.
>
>  - R.
>   

My 2007 thread does this basically, but posted it to lkml and David 
Miller.  But the rationale for why we need it as well as other possible 
solutions is included in that thread.  We could re-package it and send 
it on as you suggest.  It might carry more weight coming from the linux 
rdma maintainer though. :)





--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Bernard Metzler]

I agree that the issue must get solved and its good that it has
been brought up again. I agree with Chien that the
solution should respect and interface to a single in kernel instance
maintaining host global TCP port space. iWARP is just another
protocol on top of TCP - like iSCSI. There is no good reason to
invent another TCP port maintainer per TCP user type trying to
synchonize with the kernel if the resource is host global and
already maintained by the kernel.

Since we are developing and already open sourced a full software
implementation (SoftiWARP) of RDMA, our view on the optimal solution
must be different. Like kernel iSCSI, we are running on top of regular
kernel sockets. With that, there is no point having a connection manager
blocking just the port we wanted to use for communication - SoftiWARP
uses kernel sockets for data communication.

Therefore, I propose pushing back responsibility to the RDMA device driver,
where the actual connection setup is initiated (RNIC) or takes place
(software RMDA stack). I think, it is not the job of the RDMA connection
manager to maintain TCP port space at all. It should be up to the driver
to do the appropriate steps. Due to the lack of another interface, an
RNIC driver would create and bind a kernel socket to get hold of
the TCP port it is intending to use for offloaded communication,
while a software RDMA stack just goes forward doing communication on
that socket. For the future it might be a good idea to approach the
netdev folks kindly asking for a neat interface for just TCP port
maintainance without the need to create and bind an otherwise
useless socket.

Of course, the RNIC driver must restrict its activities to local
IP adresses on its cards (or, for SoftiWARP, to IP adresses of interfaces
it is bound to). For example, a wildcard listen must get translated
into a listen restricted to the interface(s) under local control.

With that, the RDMA connection manager should simply be aware of
the possibility that a listen or connect call may fail for one
more reason. From using SoftiWARP in that environment I know,
that's already the case (-EADDRINUSE is always an acceptable
return value).

Thanks,
Bernard.

linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org wrote on 06/12/2010 05:17:58 PM:

> Roland Dreier wrote:
> >  > Other protocols are also running over networking today, such as
iSCSI
> >  > and FCoE.  These happily co-exist with other L2->L4 protocols in the
> >  > stack. This iWARP patch allows iWARP to happily co-exist on a TCP
> >  > connection, and does *not* negatively affect the networking stack at
> >  > all.
> >
> > How do iSCSI offload HBAs coexist?  As I understand it, they typically
> > just choose a separate IP address.
> >
> > In any case I'm not going to slip in a patch that another maintainer
has
> > explicitly NAKed.  Maybe one way to force things forward would be to
> > write up an exhaustive explanation of the underlying problem and the
> > impact on end users, include this patch, explain that it touches only
> > RDMA code, and point out that most end users are already using this
> > patch since it's shipped in OFED.  Then send the whole thing to Linus
> > and Andrew Morton, making sure to cc Dave Miller, netdev, and
> > linux-rdma.
> >
> >  - R.
> >
>
> My 2007 thread does this basically, but posted it to lkml and David
> Miller.  But the rationale for why we need it as well as other possible
> solutions is included in that thread.  We could re-package it and send
> it on as you suggest.  It might carry more weight coming from the linux
> rdma maintainer though. :)
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Bernard Metzler wrote:
> I agree that the issue must get solved and its good that it has
> been brought up again. I agree with Chien that the
> solution should respect and interface to a single in kernel instance
> maintaining host global TCP port space. iWARP is just another
> protocol on top of TCP - like iSCSI. There is no good reason to
> invent another TCP port maintainer per TCP user type trying to
> synchonize with the kernel if the resource is host global and
> already maintained by the kernel.
>
> Since we are developing and already open sourced a full software
> implementation (SoftiWARP) of RDMA, our view on the optimal solution
> must be different. Like kernel iSCSI, we are running on top of regular
> kernel sockets. With that, there is no point having a connection manager
> blocking just the port we wanted to use for communication - SoftiWARP
> uses kernel sockets for data communication.
>
>   


Hey Bernard,

Has SoftiWARP been submitted upstream yet?


> Therefore, I propose pushing back responsibility to the RDMA device driver,
> where the actual connection setup is initiated (RNIC) or takes place
> (software RMDA stack). I think, it is not the job of the RDMA connection
> manager to maintain TCP port space at all. It should be up to the driver
> to do the appropriate steps. Due to the lack of another interface, an
> RNIC driver would create and bind a kernel socket to get hold of
> the TCP port it is intending to use for offloaded communication,
> while a software RDMA stack just goes forward doing communication on
> that socket. For the future it might be a good idea to approach the
> netdev folks kindly asking for a neat interface for just TCP port
> maintainance without the need to create and bind an otherwise
> useless socket.
>   


I proposed this design in 2007. It was NAK'd. Read the tail end of this 
email where I describe such a solution and indicate that Miller already 
NAK'd it. Now we could try again with this solution, but unless we have 
end users backing us and showing how much demand there is for this, it 
won't fly IMO.


http://lkml.org/lkml/2007/8/15/174



> Of course, the RNIC driver must restrict its activities to local
> IP adresses on its cards (or, for SoftiWARP, to IP adresses of interfaces
> it is bound to). For example, a wildcard listen must get translated
> into a listen restricted to the interface(s) under local control.
>
>   


I implemented and submitted this type of solution for cxgb3 in 2007 as 
well.


http://lkml.org/lkml/2007/9/13/268


Roland didn't like it, I think, because it used well known tokens in the 
interface name to designate iwarp ip addresses via ifconfig. Like 
"eth0:iw1". So the solution really required the admin to setup these 
iwarp-only subnets/interfaces. There was nothing that prevented non 
iwarp traffic to arrive on these ip addresses other than admin policy. I 
think that was another reason Roland didn't like this solution. Anyway, 
you can peruse that thread and maybe its a starting point for some 
"separate iwarp ipaddresses" solution....



Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Bernard Metzler]

Hi Steve,

> Bernard Metzler wrote:
> > I agree that the issue must get solved and its good that it has
> > been brought up again. I agree with Chien that the
> > solution should respect and interface to a single in kernel instance
> > maintaining host global TCP port space. iWARP is just another
> > protocol on top of TCP - like iSCSI. There is no good reason to
> > invent another TCP port maintainer per TCP user type trying to
> > synchonize with the kernel if the resource is host global and
> > already maintained by the kernel.
> >
> > Since we are developing and already open sourced a full software
> > implementation (SoftiWARP) of RDMA, our view on the optimal solution
> > must be different. Like kernel iSCSI, we are running on top of regular
> > kernel sockets. With that, there is no point having a connection
manager
> > blocking just the port we wanted to use for communication - SoftiWARP
> > uses kernel sockets for data communication.
> >
> >
>
>
> Hey Bernard,
>
> Has SoftiWARP been submitted upstream yet?

It's at gitorious.org/softiwarp, but not pushed upstream.
After doing some performance measurements and further stability
testing, we decided to improve things (carefully reading Rolands recent
email on code quality confirmed me ;). After an update of the sources
in the next days, we will try harder getting recognized by netdev.

>
>
> > Therefore, I propose pushing back responsibility to the RDMA device
driver,
> > where the actual connection setup is initiated (RNIC) or takes place
> > (software RMDA stack). I think, it is not the job of the RDMA
connection
> > manager to maintain TCP port space at all. It should be up to the
driver
> > to do the appropriate steps. Due to the lack of another interface, an
> > RNIC driver would create and bind a kernel socket to get hold of
> > the TCP port it is intending to use for offloaded communication,
> > while a software RDMA stack just goes forward doing communication on
> > that socket. For the future it might be a good idea to approach the
> > netdev folks kindly asking for a neat interface for just TCP port
> > maintainance without the need to create and bind an otherwise
> > useless socket.
> >
>
>
> I proposed this design in 2007. It was NAK'd. Read the tail end of this
> email where I describe such a solution and indicate that Miller already
> NAK'd it. Now we could try again with this solution, but unless we have
> end users backing us and showing how much demand there is for this, it
> won't fly IMO.
>
>
> http://lkml.org/lkml/2007/8/15/174
>
>
>
> > Of course, the RNIC driver must restrict its activities to local
> > IP adresses on its cards (or, for SoftiWARP, to IP adresses of
interfaces
> > it is bound to). For example, a wildcard listen must get translated
> > into a listen restricted to the interface(s) under local control.
> >
> >
>
>
> I implemented and submitted this type of solution for cxgb3 in 2007 as
> well.
>
>
> http://lkml.org/lkml/2007/9/13/268
>
>
> Roland didn't like it, I think, because it used well known tokens in the
> interface name to designate iwarp ip addresses via ifconfig. Like
> "eth0:iw1". So the solution really required the admin to setup these
> iwarp-only subnets/interfaces. There was nothing that prevented non
> iwarp traffic to arrive on these ip addresses other than admin policy. I
> think that was another reason Roland didn't like this solution. Anyway,
> you can peruse that thread and maybe its a starting point for some
> "separate iwarp ipaddresses" solution....
>
>

I don't find it the appropriate solution to establish 'iwarp-only'
interfaces
and subnets. iWARP is just a TCP application as iSCSI is one. There is no
technical need to establish 'iSCSI-only' subnets to run iSCSI. Having
separate subnets and interfaces is InfiniBand. iWARP does not need
that, it even does not need dedicated hardware to run the protocol.
As Peter already pointed out - its trying to converge
traffic on a single link. I hope. (Yes, any RNIC should also be an
Ethernet device at the same time, for all the ports not used for RDMA
communication, and the vendor of that RNIC should maintain correct
behaviour.)

So, Steve, I would go for something like your early driver patch...

Bernard.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > iWARP is just another protocol on top of TCP - like iSCSI. There is
 > no good reason to invent another TCP port maintainer per TCP user
 > type trying to synchonize with the kernel if the resource is host
 > global and already maintained by the kernel.

I think the counter-argument to this is than an iWARP offload NIC is an
independent TCP stack and hence should not be tied into the host stack.
It's interesting that you bring up iSCSI -- as I understand things,
iSCSI offload HBAs are typically configured with their own IP, through a
separate mechanism.  (The port collision problem is not likely to be hit
with iSCSI, since the HBA is an initiator and hence does only active
connections, and a 4-tuple collision between connections to the iSCSI
target is not likely and other host stack traffic is extremely unlikely)

 > Since we are developing and already open sourced a full software
 > implementation (SoftiWARP) of RDMA, our view on the optimal solution
 > must be different. Like kernel iSCSI, we are running on top of regular
 > kernel sockets. With that, there is no point having a connection manager
 > blocking just the port we wanted to use for communication - SoftiWARP
 > uses kernel sockets for data communication.

I think this is an extremely strong argument against the patch that
started the thread.  Breaking soft iWARP seems a fatal flaw.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > Since we are developing and already open sourced a full software
>  > implementation (SoftiWARP) of RDMA, our view on the optimal solution
>  > must be different. Like kernel iSCSI, we are running on top of regular
>  > kernel sockets. With that, there is no point having a connection manager
>  > blocking just the port we wanted to use for communication - SoftiWARP
>  > uses kernel sockets for data communication.
>
> I think this is an extremely strong argument against the patch that
> started the thread.  Breaking soft iWARP seems a fatal flaw.
>
>  - R.
>   

I agree.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > iWARP is just another protocol on top of TCP - like iSCSI. There is
>  > no good reason to invent another TCP port maintainer per TCP user
>  > type trying to synchonize with the kernel if the resource is host
>  > global and already maintained by the kernel.
>
> I think the counter-argument to this is than an iWARP offload NIC is an
> independent TCP stack and hence should not be tied into the host stack.
> It's interesting that you bring up iSCSI -- as I understand things,
> iSCSI offload HBAs are typically configured with their own IP, through a
> separate mechanism.  (The port collision problem is not likely to be hit
> with iSCSI, since the HBA is an initiator and hence does only active
> connections, and a 4-tuple collision between connections to the iSCSI
> target is not likely and other host stack traffic is extremely unlikely)
>
>   

Roland, do you think the iSCSI approach is a "good design" for iWARP 
devices?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > Roland, do you think the iSCSI approach is a "good design" for iWARP
 > devices?

Well, it's a different problem since as I said the port collision
problem is a non-issue for iSCSI anyway.  But yes having a separate
interface to assign an iWARP IP address to an RNIC does seem to avoid
the immediate problem.

I actually don't know what the right answer is -- having a separate IP
address for iWARP does seem to lead to having to duplicate everything
for configuring it.  (And this is the approach for the cxgb[34] iSCSI
drivers, right?)

On the other hand trying to hook offloaded iWARP into the normal stack
does seem to lead to a mess.  I see DaveM's point: TCP port space is
just the beginning -- filtering, queueing, etc also have config that
ultimately an offload device would want to hook too.

Maybe the sanest out of a bad set of options would be to come up with a
standard way to configure independent TCP/IP stacks that share a link.

really, dunno.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> > I think this is an extremely strong argument against the patch that
> > started the thread.  Breaking soft iWARP seems a fatal flaw.
> >
> >  - R.
> >
> 
> I agree.

The patch or SoftiWARP can be reworked to allow the whole iWARP family
to coexist.  It is a matter of agreeing on which path to take.

Chien


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > I just think the customer looses when we add iwarp-specific tools,
 > ipaddrs, subnets, etc etc.  And what about software iwarp?  Will it
 > use the host stack tools and not these new tools?  So then we end up
 > with 2 sets of tools for iwarp devices. :(

Agree -- but same prob with current iSCSI offload stuff...
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > Roland, do you think the iSCSI approach is a "good design" for iWARP
>  > devices?
>
> Well, it's a different problem since as I said the port collision
> problem is a non-issue for iSCSI anyway.  But yes having a separate
> interface to assign an iWARP IP address to an RNIC does seem to avoid
> the immediate problem.
>
> I actually don't know what the right answer is -- having a separate IP
> address for iWARP does seem to lead to having to duplicate everything
> for configuring it.  (And this is the approach for the cxgb[34] iSCSI
> drivers, right?)
>
>   

Yes.  Perusing the drivers/scsi/cxgb3i code I see the iscsi ipaddr is 
actually stored in the port_info struct which is hung of the netdev_priv 
of the cxgb3 device.  It is set by cxgb3i_host_set_param() which is part 
of the iscsi transport interface.

I just think the customer looses when we add iwarp-specific tools, 
ipaddrs, subnets, etc etc.  And what about software iwarp?  Will it use 
the host stack tools and not these new tools?  So then we end up with 2 
sets of tools for iwarp devices. :(


> On the other hand trying to hook offloaded iWARP into the normal stack
> does seem to lead to a mess.  I see DaveM's point: TCP port space is
> just the beginning -- filtering, queueing, etc also have config that
> ultimately an offload device would want to hook too.
>
> Maybe the sanest out of a bad set of options would be to come up with a
> standard way to configure independent TCP/IP stacks that share a link.
>
> really, dunno.
>
>  - R.
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Tung, Chien Tin wrote:
>>> I think this is an extremely strong argument against the patch that
>>> started the thread.  Breaking soft iWARP seems a fatal flaw.
>>>
>>>  - R.
>>>
>>>       
>> I agree.
>>     
>
> The patch or SoftiWARP can be reworked to allow the whole iWARP family
> to coexist.  It is a matter of agreeing on which path to take.
>
>   

I agree with this too! :)  My only reason for stating I agree with 
Roland/Bernard is that reserving a port in the rdma-cm definitely breaks 
software iwarp, so we need to rethink this whole thing in light of 
software iwarp.


Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> On the other hand trying to hook offloaded iWARP into the normal stack
> does seem to lead to a mess.  I see DaveM's point: TCP port space is
> just the beginning -- filtering, queueing, etc also have config that
> ultimately an offload device would want to hook too.

TCP port space is just the beginning but then these features
didn't show up all at once in the kernel either.  Instead of
evolving iWARP implementation, we can't even take a baby step
and fix a flaw that exists in the current kernel.  Why are we
"replicating" everything offered by the host stack instead of
hooking in?  It does not sound like good engineering to me.

Chien

 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Wed, Jun 23, 2010 at 01:46:47PM -0500, Steve Wise wrote:

> Yes.  Perusing the drivers/scsi/cxgb3i code I see the iscsi ipaddr is  
> actually stored in the port_info struct which is hung of the netdev_priv  
> of the cxgb3 device.  It is set by cxgb3i_host_set_param() which is part  
> of the iscsi transport interface.

I wonder how does neighbor discovery, routing, etc work with iscsi?

> I just think the customer looses when we add iwarp-specific tools,  
> ipaddrs, subnets, etc etc.  And what about software iwarp?  Will it use  
> the host stack tools and not these new tools?  So then we end up with 2  
> sets of tools for iwarp devices. :(

Well, maybe you can get netdev to agree on some way to create an
interface that has all the IP services, but no TCP protocol binding?
Then the configuration could be largely the same. If you could share
that with the iscsi world then maybe it isn't so bad?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Wed, Jun 23, 2010 at 01:46:47PM -0500, Steve Wise wrote:
>
>   
>> Yes.  Perusing the drivers/scsi/cxgb3i code I see the iscsi ipaddr is  
>> actually stored in the port_info struct which is hung of the netdev_priv  
>> of the cxgb3 device.  It is set by cxgb3i_host_set_param() which is part  
>> of the iscsi transport interface.
>>     
>
> I wonder how does neighbor discovery, routing, etc work with iscsi?
>
>   

For cxgb3i:

ND is handled by initiating ND via exported kernel services 
(neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net 
events to get updated neigh entries.

The host routing table is consulted via ip_route_output_flow() to map a 
destination ip address to a local netdev, and then if that device is T3, 
it will do the iscsi offload.



>> I just think the customer looses when we add iwarp-specific tools,  
>> ipaddrs, subnets, etc etc.  And what about software iwarp?  Will it use  
>> the host stack tools and not these new tools?  So then we end up with 2  
>> sets of tools for iwarp devices. :(
>>     
>
> Well, maybe you can get netdev to agree on some way to create an
> interface that has all the IP services, but no TCP protocol binding?
> Then the configuration could be largely the same. If you could share
> that with the iscsi world then maybe it isn't so bad?
>   


Maybe.  I fear this will meet the same resistance from the netdev folks.


Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

>> I wonder how does neighbor discovery, routing, etc work with iscsi?
>>
>>   
>
> For cxgb3i:
>
> ND is handled by initiating ND via exported kernel services 
> (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net 
> events to get updated neigh entries.
>
> The host routing table is consulted via ip_route_output_flow() to map 
> a destination ip address to a local netdev, and then if that device is 
> T3, it will do the iscsi offload.
>
>



By the way, this is how iWARP works too.    The ND stuff is done by the 
IWCM during RESOLVE_ADDR.  The routing lookups are done by the iWARP 
devices themselves typically.


Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Steve Wise wrote:
>
>>> I wonder how does neighbor discovery, routing, etc work with iscsi?
>>>
>>>   
>>
>> For cxgb3i:
>>
>> ND is handled by initiating ND via exported kernel services 
>> (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net 
>> events to get updated neigh entries.
>>
>> The host routing table is consulted via ip_route_output_flow() to map 
>> a destination ip address to a local netdev, and then if that device 
>> is T3, it will do the iscsi offload.
>>
>>
>
>
>
> By the way, this is how iWARP works too.    The ND stuff is done by 
> the IWCM during RESOLVE_ADDR.  The routing lookups are done by the 
> iWARP devices themselves typically.
>
>

Sorry I meant "by the iWARP device drivers themselves"

Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > > On the other hand trying to hook offloaded iWARP into the normal stack
 > > does seem to lead to a mess.  I see DaveM's point: TCP port space is
 > > just the beginning -- filtering, queueing, etc also have config that
 > > ultimately an offload device would want to hook too.

 > TCP port space is just the beginning but then these features
 > didn't show up all at once in the kernel either.  Instead of
 > evolving iWARP implementation, we can't even take a baby step
 > and fix a flaw that exists in the current kernel.  Why are we
 > "replicating" everything offered by the host stack instead of
 > hooking in?  It does not sound like good engineering to me.

Well as I said I don't particularly see a clean solution.  But the point
I was making was that the net stack is already very complex with many
places where interface configs are controlled -- having to add hooks to
pass that config on to offload devices is going to add even more
complexity and also add constraints to the format of that config
information.  Which is not good.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Wed, Jun 23, 2010 at 02:42:43PM -0500, Steve Wise wrote:
>> I wonder how does neighbor discovery, routing, etc work with iscsi?
>
> For cxgb3i:
>
> ND is handled by initiating ND via exported kernel services  
> (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net  
> events to get updated neigh entries.
>
> The host routing table is consulted via ip_route_output_flow() to map a  
> destination ip address to a local netdev, and then if that device is T3,  
> it will do the iscsi offload.

That is what RDMA does.. So that means that the IP used for iscsi is
actually an IP assigned to the interface? Doesn't that mean the port
collision problem still exits, although probably less likely?

>> Well, maybe you can get netdev to agree on some way to create an
>> interface that has all the IP services, but no TCP protocol binding?
>> Then the configuration could be largely the same. If you could share
>> that with the iscsi world then maybe it isn't so bad?

> Maybe.  I fear this will meet the same resistance from the netdev folks.

Hmm.. It kinds codifies what is already in the kernel, these offload
devices rely on neighbour and routing services from netdev and provide
their own TCP on top of it...

But.. having a device that effectively swaps the entire TCP
implementation for a proprietary version is not going to be popular
either.

At the very least, bringing iSCSI offload NICs into your solution
broadens the applicability.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Roland Dreier]

 > Doesn't that mean the port collision problem still exits [for iSCSI],
 > although probably less likely?

Yes, it's there, but almost impossible to hit: first of all, iSCSI HBAs
never listen on a port, so that can never collide.  Second, iSCSI HBAs
only establish connections to iSCSI targets on the iSCSI port -- so
really your only chance of a problem is if you ran an offloaded and
non-offloaded iSCSI initiator on the same IP to the same target, _and_
you got unlucky on the local ports that you chose.  So in practice no
one will hit this.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Wed, Jun 23, 2010 at 03:11:27PM -0500, Steve Wise wrote:
> The IP address assigned for the cxgb3i iscsi device is _not_ assigned to  
> a netdev interface via ifconfig, as far as I understand it (by looking  
> at the cxgb3i code).  So the host stack doesn't know about this address.  
> There is an administrative requirement, I assume, that the secret iscsi 
> ipaddr is within a subnet that is bound to the T3 ethX interface.  
> Otherwise the routing lookup wouldn't work.

So who responds to neighbor queries, and how do outgoing queries get
sent with the right IP? Sounds odd...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Wed, Jun 23, 2010 at 02:42:43PM -0500, Steve Wise wrote:
>   
>>> I wonder how does neighbor discovery, routing, etc work with iscsi?
>>>       
>> For cxgb3i:
>>
>> ND is handled by initiating ND via exported kernel services  
>> (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net  
>> events to get updated neigh entries.
>>
>> The host routing table is consulted via ip_route_output_flow() to map a  
>> destination ip address to a local netdev, and then if that device is T3,  
>> it will do the iscsi offload.
>>     
>
> That is what RDMA does.. So that means that the IP used for iscsi is
> actually an IP assigned to the interface?
>
>   

The IP address assigned for the cxgb3i iscsi device is _not_ assigned to 
a netdev interface via ifconfig, as far as I understand it (by looking 
at the cxgb3i code).  So the host stack doesn't know about this 
address.  There is an administrative requirement, I assume, that the 
secret iscsi ipaddr is within a subnet that is bound to the T3 ethX 
interface.  Otherwise the routing lookup wouldn't work.


Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Roland Dreier wrote:
>  > Doesn't that mean the port collision problem still exits [for iSCSI],
>  > although probably less likely?
>
> Yes, it's there, but almost impossible to hit: first of all, iSCSI HBAs
> never listen on a port, so that can never collide.  Second, iSCSI HBAs
> only establish connections to iSCSI targets on the iSCSI port -- so
> really your only chance of a problem is if you ran an offloaded and
> non-offloaded iSCSI initiator on the same IP to the same target, _and_
> you got unlucky on the local ports that you chose.  So in practice no
> one will hit this.
>   

I believe, at least for cxgb3i, the ipaddr used is not bound to an ethX 
interface.  So the 4-tuple will never collide with host TCP connections.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Wed, Jun 23, 2010 at 03:11:27PM -0500, Steve Wise wrote:
>   
>> The IP address assigned for the cxgb3i iscsi device is _not_ assigned to  
>> a netdev interface via ifconfig, as far as I understand it (by looking  
>> at the cxgb3i code).  So the host stack doesn't know about this address.  
>> There is an administrative requirement, I assume, that the secret iscsi 
>> ipaddr is within a subnet that is bound to the T3 ethX interface.  
>> Otherwise the routing lookup wouldn't work.
>>     
>
> So who responds to neighbor queries, and how do outgoing queries get
> sent with the right IP? Sounds odd...
>
>   

The iscsi hba is only an initiator, so it doesn't need to respond to arp 
queries.  I guess the Source Protocol Address in the outgoing ARP 
request will be the ipaddr of the outgoing interface.   Its ok though 
because what is needed is the next-hop peer's hwaddr.   So the ARP reply 
comes in, updates the host neigh entry, and a NEIGH_EVENT callout is 
performed to the offload device drivers.  It is a little hackish, but 
that's the only way the netdev maintainers would allow iscsi offload 
in.  They originally tried to use the src address from the ethX 
interface for the offload iscsi connections and that was rejected.


Steve.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Jason Gunthorpe]

On Wed, Jun 23, 2010 at 03:19:31PM -0500, Steve Wise wrote:

>> So who responds to neighbor queries, and how do outgoing queries get
>> sent with the right IP? Sounds odd...
>
> The iscsi hba is only an initiator, so it doesn't need to respond to arp  
> queries.  

Hmm.. The other side could arp you at any time, and if you don't answer
stuff can go bad, so something must be generating the replies.

But I guess that is seperate, sounds like iSCSI is in a similar boat
and they were not able to reconcile either? :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Steve Wise wrote:
> Jason Gunthorpe wrote:
>> On Wed, Jun 23, 2010 at 03:11:27PM -0500, Steve Wise wrote:
>>  
>>> The IP address assigned for the cxgb3i iscsi device is _not_ 
>>> assigned to  a netdev interface via ifconfig, as far as I understand 
>>> it (by looking  at the cxgb3i code).  So the host stack doesn't know 
>>> about this address.  There is an administrative requirement, I 
>>> assume, that the secret iscsi ipaddr is within a subnet that is 
>>> bound to the T3 ethX interface.  Otherwise the routing lookup 
>>> wouldn't work.
>>>     
>>
>> So who responds to neighbor queries, and how do outgoing queries get
>> sent with the right IP? Sounds odd...
>>
>>   
>
> The iscsi hba is only an initiator, so it doesn't need to respond to 
> arp queries.  I guess the Source Protocol Address in the outgoing ARP 
> request will be the ipaddr of the outgoing interface.   Its ok though 
> because what is needed is the next-hop peer's hwaddr.   So the ARP 
> reply comes in, updates the host neigh entry, and a NEIGH_EVENT 
> callout is performed to the offload device drivers.  It is a little 
> hackish, but that's the only way the netdev maintainers would allow 
> iscsi offload in.  They originally tried to use the src address from 
> the ethX interface for the offload iscsi connections and that was 
> rejected.
>
>

In case you're interested...Here is the tail end of the cxgb3i original 
submission thread showing the use of a "private  IP address which is 
unkown to the OS".

http://marc.info/?l=linux-netdev&m=121944339211552


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Jason Gunthorpe wrote:
> On Wed, Jun 23, 2010 at 03:19:31PM -0500, Steve Wise wrote:
>
>   
>>> So who responds to neighbor queries, and how do outgoing queries get
>>> sent with the right IP? Sounds odd...
>>>       
>> The iscsi hba is only an initiator, so it doesn't need to respond to arp  
>> queries.  
>>     
>
> Hmm.. The other side could arp you at any time, and if you don't answer
> stuff can go bad, so something must be generating the replies.
>   


You're right!  The low level driver, cxgb3, handles it.  See 
cxgb3_arp_process().   I missed this change in the original submission 
of cxgb3i...


> But I guess that is seperate, sounds like iSCSI is in a similar boat
> and they were not able to reconcile either? :(
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

>  > > On the other hand trying to hook offloaded iWARP into the normal stack
>  > > does seem to lead to a mess.  I see DaveM's point: TCP port space is
>  > > just the beginning -- filtering, queueing, etc also have config that
>  > > ultimately an offload device would want to hook too.
> 
>  > TCP port space is just the beginning but then these features
>  > didn't show up all at once in the kernel either.  Instead of
>  > evolving iWARP implementation, we can't even take a baby step
>  > and fix a flaw that exists in the current kernel.  Why are we
>  > "replicating" everything offered by the host stack instead of
>  > hooking in?  It does not sound like good engineering to me.
> 
> Well as I said I don't particularly see a clean solution.  But the point
> I was making was that the net stack is already very complex with many
> places where interface configs are controlled -- having to add hooks to
> pass that config on to offload devices is going to add even more
> complexity and also add constraints to the format of that config
> information.  Which is not good.

I don't want separate config file for L2 and iWARP as it adds more
work and complexity for the user.  I want it dead simple.  I can see
extending config format to include information specific for offload
but I don't see how it can limit the format.  That has not been the
case up to this point.  Also, port space patch is totally transparent
to the user and config file.  There is no managing host TCP and iWARP
TCP port space for the user.

I'm not sure about passing config info to offload devices, if the info
is outside of what L2 driver currently picks up then sure some work
needs to be done.  Hopefully everything can be pass-through from L2 to iWARP.


Chien




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Bernard Metzler]

>  > > On the other hand trying to hook offloaded iWARP into the normal
stack
>  > > does seem to lead to a mess.  I see DaveM's point: TCP port space is
>  > > just the beginning -- filtering, queueing, etc also have config that
>  > > ultimately an offload device would want to hook too.
>
>  > TCP port space is just the beginning but then these features
>  > didn't show up all at once in the kernel either.  Instead of
>  > evolving iWARP implementation, we can't even take a baby step
>  > and fix a flaw that exists in the current kernel.  Why are we
>  > "replicating" everything offered by the host stack instead of
>  > hooking in?  It does not sound like good engineering to me.
>
> Well as I said I don't particularly see a clean solution.  But the point
> I was making was that the net stack is already very complex with many
> places where interface configs are controlled -- having to add hooks to
> pass that config on to offload devices is going to add even more
> complexity and also add constraints to the format of that config
> information.  Which is not good.
>
To my understanding, our discussion touches two topics. One is
to solve the TCP port space issue, the other is more general, its about
proper integration of offloaded TCP within Linux. So, the second
topic is a generalization of the first.

Regarding the first topic, what I was about to propose is that the
iWARP kernel driver (software iWARP or RNIC) itself should take care of
port space allocations. Port space maintenance functionality should
be minimized at iWARP CM level. It looks straightforward to me if
during the rdma_connect() call the driver picks a free port using
a socket/bind sequence for its local interface. The same would be possible
for
the passive connection setup, which always involves an rdma_bind_addr()
- we would have to pass the rdma_bind_addr() call down to the driver
and EADDRINUSE would be a reasonable return value.
Here things are getting a little more complicated, if it comes to
INADDR_ANY and port 0 bindings. In private email, Bob Sharp already
suggested it -  the iWARP CM would have to pick a port and
try it on all interfaces....maybe by starting with port 0 binding
on one interface and trying to extend with the returned port on
all remaining interfaces. That introduces an unbind() call if things
fail, too. In any case, the rdma_bind_addr() call would create additional
state
at driver level.

For softiwarp, during bind() or connect(), a TCP socket would be created
and bound, for an RNIC driver (currently) the same would happen. While with
softiwarp this socket would be used for communication later, the RNIC
driver
would simply have to keep it around until the connection endpoint gets
destroyed
or the port gets unbound.

Introducing a new kernel interface to bind a port w/o having to allocate
a socket i would put on the wishlist for netdev.



The more general issue - the proper integration of offloaded TCP with all
the available tools for filtering, queueing ... of the kernel TCP stack, is
the harder nut to crack and we should start discussing it.

I propose to avoid any special treatment of RNIC devices at link or
IP level, but, at least for now, make it visible per
connection (only!) if a TCP connection is offloaded. A simple socket flag
(visible via netstat etc.) could serve that purpose. Architecturally,
network interfaces introduced by RNIC hardware should be able to serve
normal
L2 connectivity (used by any in-kernel connection endpoint) and
offloaded iWARP connections at the same time, while sharing TCP port
space with the kernel. The major argument for iWarp is link unification,
and it should be extensible to flexible RDMA enablement at application
level. And, single homed hosts with an RNIC should have plain
TCP connectivity...
For now and maybe forever, an offloaded connection would not fulfill the
conditions to serve all the good additional features of an in-kernel
connection. It would be up to the user to explicitly decide if he
likes to have offloaded connections anyway.
Some of the features might get supported by additional private
communication
between driver and offloaded connection - but i would restrict that to
supporting functionality which does not impose any further
changes to the kernel network stack (statistics etc. if possible).
All other features would be known to be unavailable.
Of course, a softiwarp connection would be visible as a normal
in-kernel TCP connection.

Maybe, that solution is to simple-minded and I miss some serious
roadblocks. Please let me know. In any case, let's start discussing these
things to come up with a reasonable solution to be further
disussed with the responsible people.



Many thanks,
Bernard.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> To my understanding, our discussion touches two topics. One is
> to solve the TCP port space issue, the other is more general, its about
> proper integration of offloaded TCP within Linux. So, the second
> topic is a generalization of the first.
> 
> Regarding the first topic, what I was about to propose is that the
> iWARP kernel driver (software iWARP or RNIC) itself should take care of
> port space allocations. Port space maintenance functionality should
> be minimized at iWARP CM level. It looks straightforward to me if
> during the rdma_connect() call the driver picks a free port using
> a socket/bind sequence for its local interface. The same would be possible
> for
> the passive connection setup, which always involves an rdma_bind_addr()
> - we would have to pass the rdma_bind_addr() call down to the driver
> and EADDRINUSE would be a reasonable return value.
> Here things are getting a little more complicated, if it comes to
> INADDR_ANY and port 0 bindings. In private email, Bob Sharp already
> suggested it -  the iWARP CM would have to pick a port and
> try it on all interfaces....maybe by starting with port 0 binding
> on one interface and trying to extend with the returned port on
> all remaining interfaces. That introduces an unbind() call if things
> fail, too. In any case, the rdma_bind_addr() call would create additional
> state
> at driver level.

I am okay with adding rdma_bind_addr and rdma_unbind_addr calls.  I won't
speak for Sean and the work that needs to go into the CM.  But this will allow
all known iWARP implementations to work together.
 
> For softiwarp, during bind() or connect(), a TCP socket would be created
> and bound, for an RNIC driver (currently) the same would happen. While with
> softiwarp this socket would be used for communication later, the RNIC
> driver
> would simply have to keep it around until the connection endpoint gets
> destroyed
> or the port gets unbound.

We want to be careful and make sure there is only one iWARP provider per IP address.
If softiWARP binds and surfaces another verbs interface on an existing one,
this scheme will not work.

Chien


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

Steve,

Do you see any issues with Bernard's proposal?  Is this something we can agree on?

Chien

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Tung,
> Chien Tin
> Sent: Friday, June 25, 2010 3:15 PM
> To: Bernard Metzler; Roland Dreier
> Cc: Jason Gunthorpe; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Waskiewicz Jr,
> Peter P; Steve Wise
> Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage
> 
> > To my understanding, our discussion touches two topics. One is
> > to solve the TCP port space issue, the other is more general, its about
> > proper integration of offloaded TCP within Linux. So, the second
> > topic is a generalization of the first.
> >
> > Regarding the first topic, what I was about to propose is that the
> > iWARP kernel driver (software iWARP or RNIC) itself should take care of
> > port space allocations. Port space maintenance functionality should
> > be minimized at iWARP CM level. It looks straightforward to me if
> > during the rdma_connect() call the driver picks a free port using
> > a socket/bind sequence for its local interface. The same would be possible
> > for
> > the passive connection setup, which always involves an rdma_bind_addr()
> > - we would have to pass the rdma_bind_addr() call down to the driver
> > and EADDRINUSE would be a reasonable return value.
> > Here things are getting a little more complicated, if it comes to
> > INADDR_ANY and port 0 bindings. In private email, Bob Sharp already
> > suggested it -  the iWARP CM would have to pick a port and
> > try it on all interfaces....maybe by starting with port 0 binding
> > on one interface and trying to extend with the returned port on
> > all remaining interfaces. That introduces an unbind() call if things
> > fail, too. In any case, the rdma_bind_addr() call would create additional
> > state
> > at driver level.
> 
> I am okay with adding rdma_bind_addr and rdma_unbind_addr calls.  I won't
> speak for Sean and the work that needs to go into the CM.  But this will allow
> all known iWARP implementations to work together.
> 
> > For softiwarp, during bind() or connect(), a TCP socket would be created
> > and bound, for an RNIC driver (currently) the same would happen. While with
> > softiwarp this socket would be used for communication later, the RNIC
> > driver
> > would simply have to keep it around until the connection endpoint gets
> > destroyed
> > or the port gets unbound.
> 
> We want to be careful and make sure there is only one iWARP provider per IP address.
> If softiWARP binds and surfaces another verbs interface on an existing one,
> this scheme will not work.
> 
> Chien
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

I haven't thought through all the details, but in principal this should 
work.  But this isn't just and iWARP issue.  Currently all RDMA-CM users 
share the same port space.   I think we need to maintain this, so a 
transport-independent RDMA app can run over both IB and IW.  This goes 
for server side wrt listen/accept as well.

Steve.


Tung, Chien Tin wrote:
> Steve,
>
> Do you see any issues with Bernard's proposal?  Is this something we can agree on?
>
> Chien
>
>   
>> -----Original Message-----
>> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Tung,
>> Chien Tin
>> Sent: Friday, June 25, 2010 3:15 PM
>> To: Bernard Metzler; Roland Dreier
>> Cc: Jason Gunthorpe; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Waskiewicz Jr,
>> Peter P; Steve Wise
>> Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage
>>
>>     
>>> To my understanding, our discussion touches two topics. One is
>>> to solve the TCP port space issue, the other is more general, its about
>>> proper integration of offloaded TCP within Linux. So, the second
>>> topic is a generalization of the first.
>>>
>>> Regarding the first topic, what I was about to propose is that the
>>> iWARP kernel driver (software iWARP or RNIC) itself should take care of
>>> port space allocations. Port space maintenance functionality should
>>> be minimized at iWARP CM level. It looks straightforward to me if
>>> during the rdma_connect() call the driver picks a free port using
>>> a socket/bind sequence for its local interface. The same would be possible
>>> for
>>> the passive connection setup, which always involves an rdma_bind_addr()
>>> - we would have to pass the rdma_bind_addr() call down to the driver
>>> and EADDRINUSE would be a reasonable return value.
>>> Here things are getting a little more complicated, if it comes to
>>> INADDR_ANY and port 0 bindings. In private email, Bob Sharp already
>>> suggested it -  the iWARP CM would have to pick a port and
>>> try it on all interfaces....maybe by starting with port 0 binding
>>> on one interface and trying to extend with the returned port on
>>> all remaining interfaces. That introduces an unbind() call if things
>>> fail, too. In any case, the rdma_bind_addr() call would create additional
>>> state
>>> at driver level.
>>>       
>> I am okay with adding rdma_bind_addr and rdma_unbind_addr calls.  I won't
>> speak for Sean and the work that needs to go into the CM.  But this will allow
>> all known iWARP implementations to work together.
>>
>>     
>>> For softiwarp, during bind() or connect(), a TCP socket would be created
>>> and bound, for an RNIC driver (currently) the same would happen. While with
>>> softiwarp this socket would be used for communication later, the RNIC
>>> driver
>>> would simply have to keep it around until the connection endpoint gets
>>> destroyed
>>> or the port gets unbound.
>>>       
>> We want to be careful and make sure there is only one iWARP provider per IP address.
>> If softiWARP binds and surfaces another verbs interface on an existing one,
>> this scheme will not work.
>>
>> Chien
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>     
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Bernard Metzler]

Steve,

merging the port space of different transports is very particular
and it would be good if we could avoid it w/o losing on transport
independence. Could we prove a strong need for that short-cut?
If its really needed, one global RDMA_CM instance should be able
to achive that requirement. But wouldn't that go together with a waste
of unused transports resources (i.e., IB binds would create
and bind also TCP sockets..?).
Many thanks for clarification!


Thanks,
Bernard.

Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote on 07/06/2010 11:03:15 PM:

> I haven't thought through all the details, but in principal this should
> work.  But this isn't just and iWARP issue.  Currently all RDMA-CM users
> share the same port space.   I think we need to maintain this, so a
> transport-independent RDMA app can run over both IB and IW.  This goes
> for server side wrt listen/accept as well.
>
> Steve.
>
>
> Tung, Chien Tin wrote:
> > Steve,
> >
> > Do you see any issues with Bernard's proposal?  Is this something
> we can agree on?
> >
> > Chien
> >
> >
> >> -----Original Message-----
> >> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Tung,
> >> Chien Tin
> >> Sent: Friday, June 25, 2010 3:15 PM
> >> To: Bernard Metzler; Roland Dreier
> >> Cc: Jason Gunthorpe; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Waskiewicz Jr,
> >> Peter P; Steve Wise
> >> Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space
usage
> >>
> >>
> >>> To my understanding, our discussion touches two topics. One is
> >>> to solve the TCP port space issue, the other is more general, its
about
> >>> proper integration of offloaded TCP within Linux. So, the second
> >>> topic is a generalization of the first.
> >>>
> >>> Regarding the first topic, what I was about to propose is that the
> >>> iWARP kernel driver (software iWARP or RNIC) itself should take care
of
> >>> port space allocations. Port space maintenance functionality should
> >>> be minimized at iWARP CM level. It looks straightforward to me if
> >>> during the rdma_connect() call the driver picks a free port using
> >>> a socket/bind sequence for its local interface. The same would be
possible
> >>> for
> >>> the passive connection setup, which always involves an rdma_bind_addr
()
> >>> - we would have to pass the rdma_bind_addr() call down to the driver
> >>> and EADDRINUSE would be a reasonable return value.
> >>> Here things are getting a little more complicated, if it comes to
> >>> INADDR_ANY and port 0 bindings. In private email, Bob Sharp already
> >>> suggested it -  the iWARP CM would have to pick a port and
> >>> try it on all interfaces....maybe by starting with port 0 binding
> >>> on one interface and trying to extend with the returned port on
> >>> all remaining interfaces. That introduces an unbind() call if things
> >>> fail, too. In any case, the rdma_bind_addr() call would create
additional
> >>> state
> >>> at driver level.
> >>>
> >> I am okay with adding rdma_bind_addr and rdma_unbind_addr calls.  I
won't
> >> speak for Sean and the work that needs to go into the CM.  But
> this will allow
> >> all known iWARP implementations to work together.
> >>
> >>
> >>> For softiwarp, during bind() or connect(), a TCP socket would be
created
> >>> and bound, for an RNIC driver (currently) the same would happen.While
with
> >>> softiwarp this socket would be used for communication later, the RNIC
> >>> driver
> >>> would simply have to keep it around until the connection endpoint
gets
> >>> destroyed
> >>> or the port gets unbound.
> >>>
> >> We want to be careful and make sure there is only one iWARP
> provider per IP address.
> >> If softiWARP binds and surfaces another verbs interface on an existing
one,
> >> this scheme will not work.
> >>
> >> Chien
> >>
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
in
> >> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Bernard Metzler wrote:
> Steve,
>
> merging the port space of different transports is very particular
> and it would be good if we could avoid it w/o losing on transport
> independence. 

But the IB and IW port space is _already_ merged by the RDMA_CM.  That's 
my point.  So a solution that breaks this impacts transport-independent 
applications that use the rdma cm...

> Could we prove a strong need for that short-cut?
> If its really needed, one global RDMA_CM instance should be able
> to achive that requirement. But wouldn't that go together with a waste
> of unused transports resources (i.e., IB binds would create
> and bind also TCP sockets..?).
>   

My comment has nothing to do with TCP ports but rather RDMA ports as 
abstracted by the RDMA_CM.  The RDMA_PS_TCP port space is shared among 
all the RDMA transports.  So if you rdma_bind() to port space 
RDMA_PS_TCP, addr INADDR_ANY, port 9999, then it will bind the cm_id to 
all IB and IW providers for that address/port (note the providers don't 
really get called until rdma_listen()).  Thus if you propose moving the 
bind operation down into the device providers, then the IB providers 
will also have to add this logic.  Or some solution must be defined to 
keep the IB and IW port spaces common.  Otherwise the RDMA_CM cannot 
support transport independent applications...

I don't think there is anything wrong with doing this, but realize this 
change will impact all rdma providers as well as the core.


Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Tung, Chien Tin]

> But the IB and IW port space is _already_ merged by the RDMA_CM.  That's
> my point.  So a solution that breaks this impacts transport-independent
> applications that use the rdma cm...
> 
> > Could we prove a strong need for that short-cut?
> > If its really needed, one global RDMA_CM instance should be able
> > to achive that requirement. But wouldn't that go together with a waste
> > of unused transports resources (i.e., IB binds would create
> > and bind also TCP sockets..?).
> >
> 
> My comment has nothing to do with TCP ports but rather RDMA ports as
> abstracted by the RDMA_CM.  The RDMA_PS_TCP port space is shared among
> all the RDMA transports.  So if you rdma_bind() to port space
> RDMA_PS_TCP, addr INADDR_ANY, port 9999, then it will bind the cm_id to
> all IB and IW providers for that address/port (note the providers don't
> really get called until rdma_listen()).  Thus if you propose moving the
> bind operation down into the device providers, then the IB providers
> will also have to add this logic.  Or some solution must be defined to
> keep the IB and IW port spaces common.  Otherwise the RDMA_CM cannot
> support transport independent applications...

I didn't think about this before, a bit too focused on solving the iWARP side.
Bernard, what about letting RDMA CM allocate the socket and exposing it to 
softiWARP?

Chien


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

[Steve Wise]

Tung, Chien Tin wrote:
>> But the IB and IW port space is _already_ merged by the RDMA_CM.  That's
>> my point.  So a solution that breaks this impacts transport-independent
>> applications that use the rdma cm...
>>
>>     
>>> Could we prove a strong need for that short-cut?
>>> If its really needed, one global RDMA_CM instance should be able
>>> to achive that requirement. But wouldn't that go together with a waste
>>> of unused transports resources (i.e., IB binds would create
>>> and bind also TCP sockets..?).
>>>
>>>       
>> My comment has nothing to do with TCP ports but rather RDMA ports as
>> abstracted by the RDMA_CM.  The RDMA_PS_TCP port space is shared among
>> all the RDMA transports.  So if you rdma_bind() to port space
>> RDMA_PS_TCP, addr INADDR_ANY, port 9999, then it will bind the cm_id to
>> all IB and IW providers for that address/port (note the providers don't
>> really get called until rdma_listen()).  Thus if you propose moving the
>> bind operation down into the device providers, then the IB providers
>> will also have to add this logic.  Or some solution must be defined to
>> keep the IB and IW port spaces common.  Otherwise the RDMA_CM cannot
>> support transport independent applications...
>>     
>
> I didn't think about this before, a bit too focused on solving the iWARP side.
> Bernard, what about letting RDMA CM allocate the socket and exposing it to 
> softiWARP?
>
>   

We don't want the RDMA CM allocating sockets.  That won't fly upstream 
eh?  Unless we have a provider attribute or something that tells the 
RDMA CM that this provider uses a host socket.  And this might really 
need to go into the IWCM...





--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html