rdma: 3.1.0-rc9 breaks UD

rdma: 3.1.0-rc9 breaks UD

[Christoph Lameter]

Running ibv_ud_pingpong and ibc_uc_pingpong between two hosts. One with
OFED 1.5.3.1 (Ubuntu LTS 10.04) and another on linux 3.1.0-rc9 (Same
ubuntu version uderlying) with the upstream libraries.

ibv_uc_pingpong

OFED:

# ibv_uc_pingpong
  local address:  LID 0x000a, QPN 0x54004d, PSN 0x00637e, GID ::
  remote address: LID 0x000b, QPN 0x04004c, PSN 0x7fdc0a, GID ::
8192000 bytes in 0.01 seconds = 7526.82 Mbit/sec
1000 iters in 0.01 seconds = 8.71 usec/iter


3.1.0-rc9:

ibv_uc_pingpong 10.2.35.21
  local address:  LID 0x000b, QPN 0x04004c, PSN 0x7fdc0a, GID ::
  remote address: LID 0x000a, QPN 0x54004d, PSN 0x00637e, GID ::
8192000 bytes in 0.01 seconds = 7634.67 Mbit/sec
1000 iters in 0.01 seconds = 8.58 usec/iter


ibv_ud_pingpong

OFED:

# ibv_ud_pingpong
  local address:  LID 0x000a, QPN 0x50004d, PSN 0x8b572b: GID ::
  remote address: LID 0x000b, QPN 0x02004c, PSN 0x2117ef, GID ::

3.1.0-rc9:

# ibv_ud_pingpong 10.2.35.21
  local address:  LID 0x000b, QPN 0x02004c, PSN 0x2117ef: GID ::
  remote address: LID 0x000a, QPN 0x50004d, PSN 0x8b572b, GID ::


No traffic flows.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma: 3.1.0-rc9 breaks UD

[Hefty, Sean]

> Running ibv_ud_pingpong and ibc_uc_pingpong between two hosts. One with
> OFED 1.5.3.1 (Ubuntu LTS 10.04) and another on linux 3.1.0-rc9 (Same
> ubuntu version uderlying) with the upstream libraries.

FWIW, I was able to run 3.1-rc9 in loopback and between 3.0 and 3.1-rc9 systems.  I don't have 2 systems configured with 3.1-rc9.  I'm using upstream builds of all libraries.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma: 3.1.0-rc9 breaks UD

[Christoph Lameter]

On Fri, 14 Oct 2011, Hefty, Sean wrote:

> > Running ibv_ud_pingpong and ibc_uc_pingpong between two hosts. One with
> > OFED 1.5.3.1 (Ubuntu LTS 10.04) and another on linux 3.1.0-rc9 (Same
> > ubuntu version uderlying) with the upstream libraries.
>
> FWIW, I was able to run 3.1-rc9 in loopback and between 3.0 and 3.1-rc9 systems.  I don't have 2 systems configured with 3.1-rc9.  I'm using upstream builds of all libraries.

Our 3.1-rc9 included Rolands for-next branch.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Roland Dreier]

On Fri, Oct 14, 2011 at 9:31 AM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
> Our 3.1-rc9 included Rolands for-next branch.

There are a few RDMA-related changes there :)

A bisect from 3.1-rc9 to my for-next branch shouldn't take too long if
you get chance,
otherwise I'll try to take a look soon.

This is with mlx4 drivers?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma: 3.1.0-rc9 breaks UD

[Hefty, Sean]

> Our 3.1-rc9 included Rolands for-next branch.

Actually, mine did too.  I wonder if this OFED patch to libibverbs is causing the issue:

http://git.openfabrics.org/git?p=ofed_1_5/libibverbs.git;a=blob;f=fixes/rocee_examples.patch;h=eda5a401a3424a104e8100848b5b6bf4e5b63bee;hb=HEAD
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Roland Dreier]

On Fri, Oct 14, 2011 at 9:36 AM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> Actually, mine did too.  I wonder if this OFED patch to libibverbs is causing the issue:
>
> http://git.openfabrics.org/git?p=ofed_1_5/libibverbs.git;a=blob;f=fixes/rocee_examples.patch;h=eda5a401a3424a104e8100848b5b6bf4e5b63bee;hb=HEAD

Hmm, yeah, does that change the wire protocol on the TCP setup socket
between the pingpong processes?
But that doesn't explain why UC works and UD doesn't.

I actually pulled that into my libibverbs git tree (haven't release
the IBoE version yet)...

I wonder how important wire compatibilty of the ibv_xx_pingpong
examples is... should I worry about this?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma: 3.1.0-rc9 breaks UD

[Hefty, Sean]

> I wonder how important wire compatibilty of the ibv_xx_pingpong
> examples is... should I worry about this?

I know that for some of the larger clusters, they may not be able to upgrade the software on all systems during a scheduled downtime.  This is about the only reason I can think of why you might even care.  It doesn't look that important to me, and DAPL seems to break its connection wire protocol just about every release without any complaints yet.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Or Gerlitz]

On Fri, Oct 14, 2011 at 4:02 PM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:
> Running ibv_ud_pingpong and ibc_uc_pingpong between two hosts. One with
> OFED 1.5.3.1 (Ubuntu LTS 10.04) and another on linux 3.1.0-rc9 (Same
> ubuntu version uderlying) with the upstream libraries.
> ibv_ud_pingpong
> OFED:
> # ibv_ud_pingpong
>  local address:  LID 0x000a, QPN 0x50004d, PSN 0x8b572b: GID ::
>  remote address: LID 0x000b, QPN 0x02004c, PSN 0x2117ef, GID ::
> 3.1.0-rc9:
> # ibv_ud_pingpong 10.2.35.21
>  local address:  LID 0x000b, QPN 0x02004c, PSN 0x2117ef: GID ::
>  remote address: LID 0x000a, QPN 0x50004d, PSN 0x8b572b, GID ::
> No traffic flows.

Could you try that again with small msg size e.g use "-s 100", if that
doesn't help
can you send the output of "ibv_devinfo" from both nodes. Also please
specify the exact top commit of the two libraries (libibverbs and I
assume libmlx4 if you use Mellanox ConnectX) in case you built them
from the git trees, or the library version if you're using official
release, thanks,

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Christoph Lameter]

On Mon, 17 Oct 2011, Or Gerlitz wrote:

> Could you try that again with small msg size e.g use "-s 100", if that
> doesn't help
> can you send the output of "ibv_devinfo" from both nodes. Also please
> specify the exact top commit of the two libraries (libibverbs and I
> assume libmlx4 if you use Mellanox ConnectX) in case you built them
> from the git trees, or the library version if you're using official
> release, thanks,

-s 100 fixes the issue.

The one side has OFED 1.5.3.1 kernel modules and libraries from the same
tarball.

The kernel ib side has

Libibverbs_git from a mirror of kernel.org

commit d0ebae72223298d6dbd7a2e2c812c529bf4d4a1d
Author: Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org>
Date:   Sun Aug 7 18:01:48 2011 +0000

    Makefile.am: Fix an automake warning

    Fix the following automake warning message:

        Makefile.am:1: `INCLUDES' is the old name for `AM_CPPFLAGS' (or `*_CPPFLAGS')

    A quote from the automake manual:

        INCLUDES
            This does the same job as AM_CPPFLAGS (or any per-target _CPPFLAGS variable
            if it is used). It is an older name for the same functionality. This
            variable is deprecated; we suggest using AM_CPPFLAGS and er-target
            _CPPFLAGS instead.

    Signed-off-by: Bart Van Assche bvanassche-HInyCGIudOg@public.gmane.org

Mlx4_git from the same place.

commit 60dade7a5e97da3b2eedf5839169c044f67577b3
Author: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>
Date:   Thu Aug 11 09:35:20 2011 -0700

    Add "foreign" option to AM_INIT_AUTOMAKE

    Switch to the modern form of the AM_INIT_AUTOMAKE macro and tell
    automake that the libmlx4 package does not follow the GNU standards.
    This change makes it possible to use 'autoreconf' for libmlx4.

    Signed-off-by: Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org>


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Or Gerlitz]

> On Tue, Oct 18, 2011 at 5:14 PM, Christoph Lameter <cl-gkYfJU5Cukgdnm+yROfE0A@public.gmane.org> wrote:

> -s 100 fixes the issue.

Good to know! I'll take a look into whether/how we could improve this
little but annoying incompatibility...

 Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Roland Dreier]

On Tue, Oct 18, 2011 at 3:13 PM, Or Gerlitz <or.gerlitz-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> -s 100 fixes the issue.
>
> Good to know! I'll take a look into whether/how we could improve this
> little but annoying incompatibility...

Do you understand why -s 100 fixes things?  If so can you explain it to me? ;)

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Or Gerlitz]

On Wed, Oct 19, 2011 at 12:52 AM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:

> Do you understand why -s 100 fixes things?  If so can you explain it to me? ;)

Oh sorry if I was a bit cryptic here (BTW 99 and 101 and 256,512,2048 would
work as well...), I'm not next to the code and writing from backoffice my
head cache, I think before that patch for some reason the rc/uc/srq
pingpongs were defaulted to 2048 wheres the ud pingpong wad defaulted to
1024 and this patch changed also ud default to be 2048 so one side sends
2048 and the rx side does post receive of 1024 - will that hang the  test or
should generate completion with error or actually no since its UD - in few
hours I will be next to the code/spec/setup...

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma: 3.1.0-rc9 breaks UD

[Or Gerlitz]

On 10/19/2011 1:05 AM, Or Gerlitz wrote:
> in few hours I will be next to the code...

The theory failed, looking on commit 
577818b50990853bf112ba2a2c336020dd278622 "Update examples for IBoE"
it doesn't change the default message size for the ud pingpong, so we 
have to dig deeper, Christoph,
can you patch out that commit on your system and say if you still see 
the problem? thanks,

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html