failure to get gid with rdma_bind_addr with >= 3.10 kernels

failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

[-- Attachment #1: Type: text/plain, Size: 2803 bytes --]

Hi Sean,

So somewhere between 3.10 and 3.11 a little bug was introduced in 
ucma/cma that when
you bind to IPv4 address the resulted device address returned from the 
kernel is all zeros.

You can use the attached program to reproduce, just run it and provide 
an IPv4 address of IPoIB NIC

--> with 3.10

$ ./rb 192.168.20.47
event_channel: 0x6093c0
cm_id: 0x602cb0
sgid: fe 80 00 00 00 00 00 00 00 02 c9 03 00 e9 c0 81

--> with 3.12

$ ./rb 192.168.20.17
event_channel: 0x6093c0
cm_id: 0x6034c0
sgid: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

There many commits for the 3.11 AF_IB merge, hope you can dig it out 
quickly..

Or.

linux-2.6]# git log --oneline v3.10.. drivers/infiniband/core/cma.c 
drivers/infiniband/core/ucma.c
7c049d0 Merge tag 'rdma-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
27703bb Merge tag 'PTR_RET-for-linus' of 
git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
24d44a3 RDMA/cma: Add IPv6 support for iWARP
5eb695c RDMA/cma: Only call cma_save_ib_info() for CM REQs
e511d1a RDMA/cma: Fix accessing invalid private data for UD
8fb488d RDMA/cma: Fix gcc warning
8c6ffba PTR_RET is now PTR_ERR_OR_ZERO(): Replace most.
ce117ff RDMA/cma: Export AF_IB statistics
5bc2b7b RDMA/ucma: Allow user space to specify AF_IB when joining multicast
209cf2a RDMA/ucma: Allow user space to pass AF_IB into resolve
eebe4c3 RDMA/ucma: Allow user space to bind to AF_IB
05ad945 RDMA/ucma: Name changes to indicate only IP addresses supported
edaa7a5 RDMA/ucma: Add ability to query GID addresses
cf53936 RDMA/cma: Export cma_get_service_id()
ac53b26 RDMA/ucma: Support querying when IB paths are not reversible
ee7aed4 RDMA/ucma: Support querying for AF_IB addresses
94d0c93 RDMA/cma: Only listen on IB devices when using AF_IB
5c43813 RDMA/cma: Set qkey for AF_IB
e8160e1 RDMA/cma: Expose private data when using AF_IB
fbaa1a6 RDMA/cma: Merge cma_get/save_net_info
01602f1 RDMA/cma: Remove unused SDP related code
496ce3c RDMA/cma: Add support for AF_IB to cma_get_service_id()
f68194c RDMA/cma: Add support for AF_IB to rdma_resolve_route()
f17df3b RDMA/cma: Add support for AF_IB to rdma_resolve_addr()
4ae7152 RDMA/cma: Verify that source and dest sa_family are the same
b0569e4 RDMA/cma: Restrict AF_IB loopback to binding to IB devices only
f475383 RDMA/cma: Add helper functions to return id address information
6a3e362 RDMA/cma: Do not modify sa_family when setting loopback address
680f920 RDMA/cma: Allow user to specify AF_IB when binding
58afdcb RDMA/cma: Update port reservation to support AF_IB
ef56086 IB/addr: Add AF_IB support to ip_addr_size
2e2d190 RDMA/cma: Include AF_IB in loopback and any address checks
c8dea2f9 RDMA/cma: Allow enabling reuseaddr in any state
351638e net: pass info struct via netdevice notifier






[-- Attachment #2: rb.c --]
[-- Type: text/plain, Size: 1348 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <errno.h>
#include <string.h>
#include <rdma/rdma_cma.h>

int main( int argc, char **argv ) {
  struct rdma_cm_id *         cm_id         = NULL;
  struct rdma_event_channel * event_channel = NULL;
  int i = 0;
  struct sockaddr_in sa;

  if( argc < 2 ) {
    fprintf( stderr, "Specify interface address\n" );
    exit(1);
  }
 
  if( !inet_aton( argv[1], &sa.sin_addr.s_addr ) ) {
    fprintf( stderr, "Address invalid: %s\n", argv[1] );
    exit(1);
  }

  sa.sin_port   = 0;
  sa.sin_family = AF_INET;

  event_channel = rdma_create_event_channel();
  if( !event_channel ) {
    fprintf( stderr, "rdma_create_event_channel failed" );
    exit(1);
  }
  printf( "event_channel: %p\n", event_channel );

  if( rdma_create_id( event_channel, &cm_id, 0, RDMA_PS_IPOIB ) ) {
    fprintf( stderr, "rdma_create_id failed (%s)", strerror(errno) );
    exit(1);
  }
  printf( "cm_id: %p\n", cm_id );

  if( rdma_bind_addr( cm_id, (void*)&sa ) ) {
    fprintf( stderr, "rdma_bind_addr failed (%s)", strerror(errno) );
    exit(1);
  }

  uint8_t * x  = (uint8_t*)&cm_id->route.addr.addr.ibaddr.sgid;
  size_t    sz = sizeof( cm_id->route.addr.addr.ibaddr.sgid );

  printf( "sgid: " );

  for( i = 0; i < sz; ++i ) {
    printf( "%2.2x ", x[i] );
  }

  printf( "\n" );
  return 0;
}

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

On 10/11/2013 13:10, Or Gerlitz wrote:
> So somewhere between 3.10 and 3.11
between 3.10 and 3.12
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

> On 10/11/2013 13:10, Or Gerlitz wrote:
> > So somewhere between 3.10 and 3.11
> between 3.10 and 3.12

I'll take a look.

What is the impact to the application?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Christoph Lameter]

On Mon, 11 Nov 2013, Hefty, Sean wrote:

> > On 10/11/2013 13:10, Or Gerlitz wrote:
> > > So somewhere between 3.10 and 3.11
> > between 3.10 and 3.12
>
> I'll take a look.
>
> What is the impact to the application?

Application will run with MOFED 2.0/OFED 1.5.3 but wont start with Linux
3.12.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

> $ ./rb 192.168.20.47
> event_channel: 0x6093c0
> cm_id: 0x602cb0
> sgid: fe 80 00 00 00 00 00 00 00 02 c9 03 00 e9 c0 81
> 
> --> with 3.12
> 
> $ ./rb 192.168.20.17
> event_channel: 0x6093c0
> cm_id: 0x6034c0
> sgid: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> There many commits for the 3.11 AF_IB merge, hope you can dig it out
> quickly..
> 
> Or.
> 
> linux-2.6]# git log --oneline v3.10.. drivers/infiniband/core/cma.c
..
> edaa7a5 RDMA/ucma: Add ability to query GID addresses

As part of the AF_IB changes, additional queries were introduced that separated out retrieving the different kernel data -- assigned address, GID mappings, and IB PR data.  (New queries were necessary, since sockaddr_ib is larger than sockaddr_in6, and we want to eventually handle non-reversible paths.)   The librdmacm 1.0.17 changed which query was called after rdma_bind_addr was invoked to only retrieve the assigned address, versus retrieving everything.  I think this is why you see an SGID of all 0's after calling rdma_bind_addr.

librdmacm 1.0.16 does not enable AF_IB for user space, and should return the SGID.  Assuming that this interpretation of the problem is correct, a fix for 1.0.17 would be to add a call to ucma_query_gid() from rdma_bind_addr2() - note '2' - if the specified address is not 'any address'.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Christoph Lameter]

On Mon, 11 Nov 2013, Hefty, Sean wrote:

> As part of the AF_IB changes, additional queries were introduced that
> separated out retrieving the different kernel data -- assigned address,
> GID mappings, and IB PR data.  (New queries were necessary, since
> sockaddr_ib is larger than sockaddr_in6, and we want to eventually
> handle non-reversible paths.)  The librdmacm 1.0.17 changed which query
> was called after rdma_bind_addr was invoked to only retrieve the
> assigned address, versus retrieving everything.  I think this is why you
> see an SGID of all 0's after calling rdma_bind_addr.

Is there another call that allows the retrieval of the SGID?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

> > As part of the AF_IB changes, additional queries were introduced that
> > separated out retrieving the different kernel data -- assigned address,
> > GID mappings, and IB PR data.  (New queries were necessary, since
> > sockaddr_ib is larger than sockaddr_in6, and we want to eventually
> > handle non-reversible paths.)  The librdmacm 1.0.17 changed which query
> > was called after rdma_bind_addr was invoked to only retrieve the
> > assigned address, versus retrieving everything.  I think this is why you
> > see an SGID of all 0's after calling rdma_bind_addr.
> 
> Is there another call that allows the retrieval of the SGID?

Yes - something like this patch should help, but I don't think this is the correct behavior when the IP address is 'any' addresss.

Retrieve SGID after calling rdma_bind_addr

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

A change was made to rdma_bind_addr when AF_IB is enabled
to only retrieve the resulting bound address.  Previously,
rdma_bind_addr would retrieve the corresponding SGID as
well.  This breaks some apps which were checking the
SGID after binding to an IP address.  Revert to the
previous behavior of also retrieving the SGID after
calling rdma_bind_addr.
---
 src/cma.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/src/cma.c b/src/cma.c
index 4f41879..0cf4203 100644
--- a/src/cma.c
+++ b/src/cma.c
@@ -753,7 +753,10 @@ static int rdma_bind_addr2(struct rdma_cm_id *id, struct sockaddr *addr,
 	if (ret != sizeof cmd)
 		return (ret >= 0) ? ERR(ENODATA) : -1;
 
-	return ucma_query_addr(id);
+	ret = ucma_query_addr(id);
+	if (!ret)
+		ret = ucma_query_gid(id);
+	return ret;
 }
 
 int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Christoph Lameter]

On Mon, 11 Nov 2013, Hefty, Sean wrote:

> Yes - something like this patch should help, but I don't think this is the correct behavior when the IP address is 'any' addresss.

Well it works fine. Application starts now.

Tested-by: Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

On 11/11/2013 20:46, Hefty, Sean wrote:
> Yes - something like this patch should help, but I don't think this is the correct behavior when the IP address is 'any' addresss.
>
> Retrieve SGID after calling rdma_bind_addr
>
> From: Sean Hefty<sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> A change was made to rdma_bind_addr when AF_IB is enabled
> to only retrieve the resulting bound address.  Previously,
> rdma_bind_addr would retrieve the corresponding SGID as
> well.  This breaks some apps which were checking the
> SGID after binding to an IP address.  Revert to the
> previous behavior of also retrieving the SGID after
> calling rdma_bind_addr.
> ---
>   src/cma.c |    5 ++++-
>   1 files changed, 4 insertions(+), 1 deletions(-)
>
> diff --git a/src/cma.c b/src/cma.c
> index 4f41879..0cf4203 100644
> --- a/src/cma.c
> +++ b/src/cma.c
> @@ -753,7 +753,10 @@ static int rdma_bind_addr2(struct rdma_cm_id *id, struct sockaddr *addr,
>   	if (ret != sizeof cmd)
>   		return (ret >= 0) ? ERR(ENODATA) : -1;
>   
> -	return ucma_query_addr(id);
> +	ret = ucma_query_addr(id);
> +	if (!ret)
> +		ret = ucma_query_gid(id);
> +	return ret;
>   }
>   
>   int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)

Sean, how do we continue here? the patch worked for Christoph, so are 
going to merge it into librdmacm or it needs more work?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

 
> Sean, how do we continue here? the patch worked for Christoph, so are
> going to merge it into librdmacm or it needs more work?

I will merge it, since it fixes the issue.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

On Wed, Nov 13, 2013 at 10:15 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>
>> Sean, how do we continue here? the patch worked for Christoph, so are
>> going to merge it into librdmacm or it needs more work?
>
> I will merge it, since it fixes the issue.

cool
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

Hi Sean,

Can you please prepare a fresh (gift to 2014) release of librdmacm, such 
that distributions && people using kernels >= 3.11 will be able to have 
this functionality restored once they upgrade librdmacm.

thanks --

Or
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Yann Droneaud]

Hi,

Le jeudi 02 janvier 2014 à 11:39 +0200, Or Gerlitz a écrit :

> Can you please prepare a fresh (gift to 2014) release of librdmacm, such 
> that distributions && people using kernels >= 3.11 will be able to have 
> this functionality restored once they upgrade librdmacm.
> 

Why is this issue addressed by a patch in librdmacm ?

After reading quickly the thread, I still believe that's sound like a
kernel regression which broke existing userspace applications.

So a patch must be applied on the kernel to fix that regression.

Introducing AF_IB must not have changed the behavor for existing
applications. Existing applications must not need a new librdmacm when
they don't use newer kernel extensions.

Regards.

PS: Happy new (gregorian calendar's) year 2014 and best wishes ;)

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

> After reading quickly the thread, I still believe that's sound like a
> kernel regression which broke existing userspace applications.
> 
> So a patch must be applied on the kernel to fix that regression.

When AF_IB was added to the kernel, corresponding changes were added to the librdmacm.  Those changes to the librdmacm are the cause of the issue.

The kernel rdma cm previously supported a single 'query' call.  With AF_IB, it supports the original call, plus 3 additional calls.  The problem is the librdmacm using the newer query calls.
 
> Introducing AF_IB must not have changed the behavor for existing
> applications. Existing applications must not need a new librdmacm when
> they don't use newer kernel extensions.

An older version of the librdmacm that does NOT support AF_IB should work fine.

- Sean

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Yann Droneaud]

Le jeudi 02 janvier 2014 à 16:33 +0000, Hefty, Sean a écrit :
> > After reading quickly the thread, I still believe that's sound like a
> > kernel regression which broke existing userspace applications.
> > 
> > So a patch must be applied on the kernel to fix that regression.
> 
> When AF_IB was added to the kernel, corresponding changes were added to the librdmacm.
> Those changes to the librdmacm are the cause of the issue.
> 

I was missing this part, the original message was misleading for me.

So it's the combination of librdmacm >= 1.0.17 and kernel >= 3.10+ which
is affected by the problem reported.

> The kernel rdma cm previously supported a single 'query' call. 
> With AF_IB, it supports the original call, plus 3 additional calls.
> The problem is the librdmacm using the newer query calls.
>  
> > Introducing AF_IB must not have changed the behavor for existing
> > applications. Existing applications must not need a new librdmacm when
> > they don't use newer kernel extensions.
> 
> An older version of the librdmacm that does NOT support AF_IB should work fine.
> 

Thanks for clarifying this for me.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Hefty, Sean]

> Can you please prepare a fresh (gift to 2014) release of librdmacm, such
> that distributions && people using kernels >= 3.11 will be able to have
> this functionality restored once they upgrade librdmacm.

There's a 1.0.17.1 release available with this fix.

I will release 1.0.18 within a couple of weeks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: failure to get gid with rdma_bind_addr with >= 3.10 kernels

[Or Gerlitz]

On 02/01/2014 19:04, Hefty, Sean wrote:
>> Can you please prepare a fresh (gift to 2014) release of librdmacm, such
>> that distributions && people using kernels >= 3.11 will be able to have
>> this functionality restored once they upgrade librdmacm.
> There's a 1.0.17.1 release available with this fix.

good to know... was it announced on the list?


> I will release 1.0.18 within a couple of weeks.

cool
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html