mode connected infiniband

mode connected infiniband

[Nico]

Hello,

I'am using an infiniband kernel 2.6.32-trunk-686 from debian,
and it seems /sys/class/net/ib0/mode is missing :

vza:/sys/class/net/ib0# ls -l
total 0
-r--r--r-- 1 root root 4096 janv. 14 08:55 address
-r--r--r-- 1 root root 4096 janv. 14 08:55 addr_len
-r--r--r-- 1 root root 4096 janv. 14 08:55 broadcast
-r--r--r-- 1 root root 4096 janv. 14 08:55 carrier
--w--w--w- 1 root root 4096 janv. 14 08:55 create_child
--w--w--w- 1 root root 4096 janv. 14 08:55 delete_child
lrwxrwxrwx 1 root root    0 janv. 14 08:55 device -> ../../../0000:05:00.0
-r--r--r-- 1 root root 4096 janv. 14 08:55 dev_id
-r--r--r-- 1 root root 4096 janv. 14 08:55 dormant
-r--r--r-- 1 root root 4096 janv. 14 08:55 features
-rw-r--r-- 1 root root 4096 janv. 14 08:55 flags
-rw-r--r-- 1 root root 4096 janv. 14 08:55 ifalias
-r--r--r-- 1 root root 4096 janv. 14 08:55 ifindex
-r--r--r-- 1 root root 4096 janv. 14 08:55 iflink
-r--r--r-- 1 root root 4096 janv. 14 08:55 link_mode
-rw-r--r-- 1 root root 4096 janv. 14 08:55 mtu
-r--r--r-- 1 root root 4096 janv. 14 08:55 operstate
-r--r--r-- 1 root root 4096 janv. 14 08:55 pkey
drwxr-xr-x 2 root root    0 janv. 14 08:55 power
drwxr-xr-x 2 root root    0 janv. 14 08:55 statistics
lrwxrwxrwx 1 root root    0 janv. 14 08:55 subsystem ->
../../../../../../../../class/net
-rw-r--r-- 1 root root 4096 janv. 14 08:55 tx_queue_len
-r--r--r-- 1 root root 4096 janv. 14 08:55 type
-rw-r--r-- 1 root root 4096 janv. 14 08:55 uevent
-rw-r--r-- 1 root root 4096 janv. 14 08:55 umcast

as I am doing some benchs and wanted to size up MTU with connected mode,

* is there another way to use connected mode ?
* is it related to debian packaging ?

Regards.

Nicolas
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Frank Leers]

On Jan 14, 2010, at 12:11 AM, Nico wrote:

> Hello,
> 
> I'am using an infiniband kernel 2.6.32-trunk-686 from debian,
> and it seems /sys/class/net/ib0/mode is missing :
> 

It might be that when OFED was built, the "CONFIG_INFINIBAND_IPOIB_CM=y" was set to 'n' in your 'configure.mk.kernel' or on the command line during the ./configure step - 

--with-ipoib-cm    make CONFIG_INFINIBAND_IPOIB_CM=y [yes]
--without-ipoib-cm   [no]

-frank


> vza:/sys/class/net/ib0# ls -l
> total 0
> -r--r--r-- 1 root root 4096 janv. 14 08:55 address
> -r--r--r-- 1 root root 4096 janv. 14 08:55 addr_len
> -r--r--r-- 1 root root 4096 janv. 14 08:55 broadcast
> -r--r--r-- 1 root root 4096 janv. 14 08:55 carrier
> --w--w--w- 1 root root 4096 janv. 14 08:55 create_child
> --w--w--w- 1 root root 4096 janv. 14 08:55 delete_child
> lrwxrwxrwx 1 root root    0 janv. 14 08:55 device -> ../../../0000:05:00.0
> -r--r--r-- 1 root root 4096 janv. 14 08:55 dev_id
> -r--r--r-- 1 root root 4096 janv. 14 08:55 dormant
> -r--r--r-- 1 root root 4096 janv. 14 08:55 features
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 flags
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 ifalias
> -r--r--r-- 1 root root 4096 janv. 14 08:55 ifindex
> -r--r--r-- 1 root root 4096 janv. 14 08:55 iflink
> -r--r--r-- 1 root root 4096 janv. 14 08:55 link_mode
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 mtu
> -r--r--r-- 1 root root 4096 janv. 14 08:55 operstate
> -r--r--r-- 1 root root 4096 janv. 14 08:55 pkey
> drwxr-xr-x 2 root root    0 janv. 14 08:55 power
> drwxr-xr-x 2 root root    0 janv. 14 08:55 statistics
> lrwxrwxrwx 1 root root    0 janv. 14 08:55 subsystem ->
> ../../../../../../../../class/net
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 tx_queue_len
> -r--r--r-- 1 root root 4096 janv. 14 08:55 type
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 uevent
> -rw-r--r-- 1 root root 4096 janv. 14 08:55 umcast
> 
> as I am doing some benchs and wanted to size up MTU with connected mode,
> 
> * is there another way to use connected mode ?
> * is it related to debian packaging ?
> 
> Regards.
> 
> Nicolas
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Roland Dreier]

 > I'am using an infiniband kernel 2.6.32-trunk-686 from debian,

I guess the problem is that standard don't enable
CONFIG_INFINIBAND_IPOIB_CM.  It's probably worth filing a Debian bug for
that, and we should probably change the upstream kernel default to "yes"
as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Nico]

Hi,

I just created a debian bug :

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565494

Regards.

Nicolas



2010/1/14 Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>:
>
>  > I'am using an infiniband kernel 2.6.32-trunk-686 from debian,
>
> I guess the problem is that standard don't enable
> CONFIG_INFINIBAND_IPOIB_CM.  It's probably worth filing a Debian bug for
> that, and we should probably change the upstream kernel default to "yes"
> as well.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Tziporet Koren]

On 1/14/2010 7:22 PM, Roland Dreier wrote:
> and we should probably change the upstream kernel default to "yes"
> as well.
>
>    
I agree
IPoIB CM is very stable now and has a better performance

Tziporet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Jason Gunthorpe]

On Sun, Jan 17, 2010 at 10:03:34AM +0200, Tziporet Koren wrote:
> IPoIB CM is very stable now and has a better performance

The last time I tried to use it the kernel began reporting lots of
OOM events (2.6.30 stock). I thought this was well known because CM
mode uses high order allocations??

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Roland Dreier]

 > The last time I tried to use it the kernel began reporting lots of
 > OOM events (2.6.30 stock). I thought this was well known because CM
 > mode uses high order allocations??

That's not well-known to me.  What's the backtrace for those high-order
allocations?  I thought the CM code was careful to allocate receive
buffers using page-size fragments but maybe there's some other path (skb
rings, QP/CQ structures?) that does a higher-order alloc.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Roland Dreier]

 > I'll contribute my 2c here, as I have experienced some page allocation
 > errors in the past when using drbd and cm. I think drbd in particular
 > might trigger it since it's pushes a lot of data through the link, but
 > regardless the circumstances are somewhat difficult to replicate. I'm not
 > completely certain they are relevant, but here are some of the traces that
 > I've noticed then; these were observed on some version of OFED 1.4 if I
 > recall. Setting /proc/sys/vm/min_free_kbytes to some ridiculous number does
 > work around this.
 > 
 > http://www.mail-archive.com/drbd-user-cunTk1MwBs8qoQakbn7OcQ@public.gmane.org/msg00441.html
 > http://lists.linbit.com/pipermail/drbd-user/2009-April/011880.html

Thanks.  I notice that neither of these tracebacks actually have any
IPoIB functions (or indeed anything RDMA-related).  So it seems the
problem is probably somewhere more generic, triggered by the large MTU
allowed by IPoIB CM.

One traceback starts with

    drbd0_worker: page allocation failure. order:5, mode:0x10

so someone is trying to do an order 5 allocation on the send path.  And
the other one starts with

    drbd0_receiver: page allocation failure. order:4, mode:0x20

So similarly, the receive path is trying an order 4 allocation.

Are both of these tracebacks using IPoIB CM with a ~64K MTU?

 - R.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Jason Gunthorpe]

On Tue, Jan 19, 2010 at 05:09:49PM -0800, Roland Dreier wrote:
> 
>  > The last time I tried to use it the kernel began reporting lots of
>  > OOM events (2.6.30 stock). I thought this was well known because CM
>  > mode uses high order allocations??
> 
> That's not well-known to me.  What's the backtrace for those high-order
> allocations?  I thought the CM code was careful to allocate receive
> buffers using page-size fragments but maybe there's some other path (skb
> rings, QP/CQ structures?) that does a higher-order alloc.

Hmm, I'll keep an eye out then. I've seen it several times in very
different systems, usually just turn CM mode off and it goes
away. Unfortunately I have no systems like that now and I didn't keep
the traces... The traces did have ipoib function calls.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mode connected infiniband

[Roland Dreier]

 > > IPoIB CM is very stable now and has a better performance
 > 
 > The last time I tried to use it the kernel began reporting lots of
 > OOM events (2.6.30 stock). I thought this was well known because CM
 > mode uses high order allocations??

And in any case changing the kernel config option default just means
that the code gets built, but we start in datagram mode by default
anyway.  So it just means people can enable it at runtime without having
to rebuild the kernel.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html