* Re: [PATCH 06/10] iw_cxgb4: Add memory management functions.
From: Roland Dreier @ 2010-04-07 21:40 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192419.14117.40759.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
> + req->wr.wr_lo = (u64)&wr_wait;
wr_lo is __be64. The cast should be to __force __be64 here I think.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 05/10] iw_cxgb4: Add connection management functions.
From: Roland Dreier @ 2010-04-07 21:37 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192413.14117.56138.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
> + wr_waitp = (struct c4iw_wr_wait *)rpl->data[1];
Sparse complains about this case from __be64 to a pointer. I assume
this is OK but you probably want to stick a __force in there to annotate it.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 05/10] iw_cxgb4: Add connection management functions.
From: Roland Dreier @ 2010-04-07 21:36 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192413.14117.56138.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
> +void _c4iw_free_ep(struct kref *kref)
...
> + ep = container_of(container_of(kref, struct c4iw_ep_common, kref),
> + struct c4iw_ep, com);
sparse warns of some internal container_of variable shadowing itself
here. You can avoid that and write this more simply as:
ep = container_of(kref, struct c4iw_ep, com.kref);
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Roland Dreier @ 2010-04-07 21:33 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192357.14117.39150.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
You have:
> +struct fw_ri_send_wr {
...
> + __be16 wrid;
> +struct fw_ri_recv_wr {
...
> + __be16 wrid;
But also:
> +static inline void init_wr_hdr(union t4_wr *wqe, u16 wrid,
> + enum fw_wr_opcodes opcode, u8 flags, u8 len16)
...
> + wqe->send.wrid = wrid;
and similar for recv.wrid in qp.c. sparse correctly warns about this
endianness clash.
The intention is that the device just treats wrid as opaque I assume so
I think the correct fix is to go from __be16 to u16 in the structure
declarations.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 05/10] iw_cxgb4: Add connection management functions.
From: Roland Dreier @ 2010-04-07 21:32 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192413.14117.56138.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
> +int peer2peer = 0;
> +module_param(peer2peer, int, 0644);
> +MODULE_PARM_DESC(peer2peer, "Support peer2peer ULPs (default=0)");
If you build iw_cxgb3 and iw_cxgb4 into the kernel, the peer2peer symbol
names clash. (Same problem occurs if you try to load cxgb3 and cxgb4
modules at the same time, I think). We can fix it here in cxgb4 by just
making peer2peer static (and deleting the extern declaration).
However peer2peer is not that great of a name for a global symbol; might
be good to add a patch to cxgb3 to rename peer2peer to something like
iwch_peer2peer and using module_param_named()...
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-07 21:18 UTC (permalink / raw)
To: Dan Carpenter
Cc: Roland Dreier, Sean Hefty, Hal Rosenstock, Tejun Heo,
Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407093901.GG5157@bicker>
thanks, applied.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] RDMA/nes: correct cap.max_inline_data assignment in nes_query_qp
From: Roland Dreier @ 2010-04-07 21:16 UTC (permalink / raw)
To: Chien Tung; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100325133950.GA9752@ctung-MOBL>
thanks, applied
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] [for-2.6.34] rdma/cm: set num_paths when manually assigning path records
From: Roland Dreier @ 2010-04-07 21:14 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma
In-Reply-To: <3946EFA63D2D4C2A93F45CF7CD715B5C-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
thanks, applied
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 31/37] librdmacm: provide abstracted verb calls
From: Steve Wise @ 2010-04-07 20:38 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <411E3A9BECF04B598FFEFFF08505DE6F-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
Sean Hefty wrote:
>>> +static inline int
>>> +rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
>>> +{
>>> + struct ibv_cq *cq;
>>> + void *context;
>>> + int ret;
>>> +
>>> + ret = ibv_poll_cq(id->send_cq, 1, wc);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + ret = ibv_req_notify_cq(id->send_cq, 0);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + ret = ibv_poll_cq(id->send_cq, 1, wc);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + ret = ibv_get_cq_event(id->send_cq_channel, &cq, &context);
>>> + if (ret)
>>> + return ret;
>>>
>>>
>> This doesn't look correct. If the send isn't complete by the time the
>> 2nd ibv_poll_cq() completes, then this function will return without
>> having filled in the wc. Or am I missing something? Shouldn't the
>> ibv_get_cq_event() be the first thing this function does? The same
>> issue/question exists for rdma_get_recv_comp().
>>
>
> I think it's possible for the function to return without having filled in a wc.
>
So its busted? Or is this intended behavior?
> If the 2nd poll removes a completion, it can leave a cq event on the channel,
> which a subsequent call could retrieve, but then find the cq empty.
>
> The idea for this call is to abstract poll, notify_cq, and get_cq_event, but
> still provide decent performance. (Scalability is a separate matter. I
> couldn't find a decent way to abstract a CQ shared across QPs or between the
> receive and send queues.)
>
> To avoid returning from the call without a completion, I think the following
> structure works:
>
> poll()
> notify_cq()
> poll()
> while (no completion) {
> get_cq_event()
> poll()
> }
>
Is rdma_get_send_completion() supposed to return exactly one wc? If so
then the 2 polls can cause a wc to get silently discarded. I must
still not be understanding the intended use?
I would think this should just be:
get_cq_event()
notify_cq()
poll()
> The only drawback I see is that it's theoretically possible to build up a queue
> of cq events in the kernel. Not sure how to fix that. Any ideas?
>
>
That can always happen, yes?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 31/37] librdmacm: provide abstracted verb calls
From: Sean Hefty @ 2010-04-07 20:14 UTC (permalink / raw)
To: 'Steve Wise'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBCDF57.20303-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>> +static inline int
>> +rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
>> +{
>> + struct ibv_cq *cq;
>> + void *context;
>> + int ret;
>> +
>> + ret = ibv_poll_cq(id->send_cq, 1, wc);
>> + if (ret)
>> + return ret;
>> +
>> + ret = ibv_req_notify_cq(id->send_cq, 0);
>> + if (ret)
>> + return ret;
>> +
>> + ret = ibv_poll_cq(id->send_cq, 1, wc);
>> + if (ret)
>> + return ret;
>> +
>> + ret = ibv_get_cq_event(id->send_cq_channel, &cq, &context);
>> + if (ret)
>> + return ret;
>>
>
>This doesn't look correct. If the send isn't complete by the time the
>2nd ibv_poll_cq() completes, then this function will return without
>having filled in the wc. Or am I missing something? Shouldn't the
>ibv_get_cq_event() be the first thing this function does? The same
>issue/question exists for rdma_get_recv_comp().
I think it's possible for the function to return without having filled in a wc.
If the 2nd poll removes a completion, it can leave a cq event on the channel,
which a subsequent call could retrieve, but then find the cq empty.
The idea for this call is to abstract poll, notify_cq, and get_cq_event, but
still provide decent performance. (Scalability is a separate matter. I
couldn't find a decent way to abstract a CQ shared across QPs or between the
receive and send queues.)
To avoid returning from the call without a completion, I think the following
structure works:
poll()
notify_cq()
poll()
while (no completion) {
get_cq_event()
poll()
}
The only drawback I see is that it's theoretically possible to build up a queue
of cq events in the kernel. Not sure how to fix that. Any ideas?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 26/37] librdmacm: set src_addr in rdma_getaddrinfo
From: Jason Gunthorpe @ 2010-04-07 20:14 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BAFD82633A744729F9685D44B250100-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Wed, Apr 07, 2010 at 12:54:56PM -0700, Sean Hefty wrote:
> >I haven't looked through everything you posted to make a suggestion
> >here, but this bothers me..
> >
> >The resources should be allocated after the rdma_bind syscall, prior to
> >listen/accept or connect, IMHO.
> >
> >How does tha rai->ai_src_addr get used to allocate resources anyhow?
>
> Maybe the patch description is off.
>
> All this does (in a very non-sexy way) is set ai_src_addr. It does
> not allocate any hardware resources. A user can provide ai_src_addr
> as input into rdma_bind or rdma_resolve_addr.
>
> The motivation is twofold. First, the user can select the
> rdma_addrinfo for a connection by examining the src/dst address
> pair. This may be desired for failover or performance reasons.
> Second, route resolution may require knowing both the source and
> destination addresses. For example, IB ACM requires both addresses
> as input.
Huumm
I don't have a problem with ai_src_addr being set, when necessary, but
setting it unconditionally seems wrong to me. In most cases the kernel
should select the source during route resolution, not be forced to
something in userspace.
Certainly for AF_INET/6 I don't think this should be done..
Apps doing complex things for failover should supply a source address
in the hints and call rdma_getaddrinfo for each adaptor.
AF_IB has the scope ID in the destination to specify the adaptor for
link-local GIDs, so the source should not often be needed.
Not sure what you mean that ACM requires it? Doesn't ACM plug in at
the rdma_getaddrinfo stage? If so it can get the source on its own
like you did in this patch. I agree that ACM should always return
results with the source set, because it is providing path records
relative to a specific adaptor.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Ummunotify: progress at last!
From: Jason Gunthorpe @ 2010-04-07 20:05 UTC (permalink / raw)
To: Roland Dreier; +Cc: Jeff Squyres, Linux RDMA List, Brad Benton
In-Reply-To: <ada39z72gww.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Wed, Apr 07, 2010 at 12:37:03PM -0700, Roland Dreier wrote:
> > No, there is no mmap. Like this:
> >
> > u64 my_counter = 0;
> >
> > ibv_set_mmu_counter(verbs, &my_counter);
> > [..]
> > while (my_counter != last_my_counter) {
> > last_my_counter = my_counter;
> > ibv_get_mmu_notifications(verbs, ...); // <- I am a memory barrier as well
> > }
> >
> > The kernel 'syscall' ibv_set_mmu_counter would bind the given verbs to
> > the 8 byte counter you specified without having to the mmap thing. As
> > I understand it this is what perfevents does.
>
> I was trying to look at how perf events handles this, and AFAICT it
> looks like kernel/perf_event.c just supports mmap(). Can you expand on
> what you meant here?
>
> (I was trying to figure out how one would handle the case where
> userspace gives us a counter in highmem -- doing kmap_atomic() seems to
> be to only option but then I'm not sure if I want to deal with that...)
I think I was mistaken here, disregard..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 26/37] librdmacm: set src_addr in rdma_getaddrinfo
From: Sean Hefty @ 2010-04-07 19:54 UTC (permalink / raw)
To: 'Jason Gunthorpe'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407192810.GG15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>I haven't looked through everything you posted to make a suggestion
>here, but this bothers me..
>
>The resources should be allocated after the rdma_bind syscall, prior to
>listen/accept or connect, IMHO.
>
>How does tha rai->ai_src_addr get used to allocate resources anyhow?
Maybe the patch description is off.
All this does (in a very non-sexy way) is set ai_src_addr. It does not allocate
any hardware resources. A user can provide ai_src_addr as input into rdma_bind
or rdma_resolve_addr.
The motivation is twofold. First, the user can select the rdma_addrinfo for a
connection by examining the src/dst address pair. This may be desired for
failover or performance reasons. Second, route resolution may require knowing
both the source and destination addresses. For example, IB ACM requires both
addresses as input.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 22/37] librdmacm: add new call to create id
From: Hefty, Sean @ 2010-04-07 19:42 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100407193402.GH15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>> + * The rdma_cm_id will be set to use synchronous operations (connect,
>> + * listen, and get_request). To convert to synchronous operation, the
> ^^^^^^^^^
>asynchronous?
yes - thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 31/37] librdmacm: provide abstracted verb calls
From: Steve Wise @ 2010-04-07 19:39 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <27A4CFF94530491F88C8E528428FF93E-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
Sean Hefty wrote:
> Provide abstractions to the verb calls to simplify the user
> interface for more casual verbs consumers. Users still have
> access to the full range of verbs functionality by calling
> verbs directly.
>
> Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>
> Makefile.am | 5 -
> include/rdma/rdma_verbs.h | 287 +++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 290 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile.am b/Makefile.am
> index 8d86045..8aef24a 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -31,7 +31,8 @@ librdmacmincludedir = $(includedir)/rdma $(includedir)/infiniband
>
> librdmacminclude_HEADERS = include/rdma/rdma_cma_abi.h \
> include/rdma/rdma_cma.h \
> - include/infiniband/ib.h
> + include/infiniband/ib.h \
> + include/rdma/rdma_verbs.h
>
> man_MANS = \
> man/rdma_accept.3 \
> @@ -69,7 +70,7 @@ man_MANS = \
> man/rdma_cm.7
>
> EXTRA_DIST = include/rdma/rdma_cma_abi.h include/rdma/rdma_cma.h \
> - include/infiniband/ib.h \
> + include/infiniband/ib.h include/rdma/rdma_verbs.h \
> src/cma.h src/librdmacm.map librdmacm.spec.in $(man_MANS)
>
> dist-hook: librdmacm.spec
> diff --git a/include/rdma/rdma_verbs.h b/include/rdma/rdma_verbs.h
> new file mode 100644
> index 0000000..05964c1
> --- /dev/null
> +++ b/include/rdma/rdma_verbs.h
> @@ -0,0 +1,287 @@
> +/*
> + * Copyright (c) 2010 Intel Corporation. All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses. You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * - Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer.
> + *
> + * - Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + */
> +
> +#if !defined(RDMA_VERBS_H)
> +#define RDMA_VERBS_H
> +
> +#include <assert.h>
> +#include <infiniband/verbs.h>
> +#include <rdma/rdma_cma.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/*
> + * Memory registration helpers.
> + */
> +static inline struct ibv_mr *
> +rdma_reg_msgs(struct rdma_cm_id *id, void *addr, size_t length)
> +{
> + return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE);
> +}
> +
> +static inline struct ibv_mr *
> +rdma_reg_read(struct rdma_cm_id *id, void *addr, size_t length)
> +{
> + return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE|
> + IBV_ACCESS_REMOTE_READ);
> +}
> +
> +static inline struct ibv_mr *
> +rdma_reg_write(struct rdma_cm_id *id, void *addr, size_t length)
> +{
> + return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE |
> + IBV_ACCESS_REMOTE_WRITE);
> +}
> +
> +static inline int
> +rdma_dereg_mr(struct ibv_mr *mr)
> +{
> + return ibv_dereg_mr(mr);
> +}
> +
> +
> +/*
> + * Vectored send, receive, and RDMA operations.
> + * Support multiple scatter-gather entries.
> + */
> +static inline int
> +rdma_post_recvv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
> + int nsge)
> +{
> + struct ibv_recv_wr wr, *bad;
> +
> + wr.wr_id = (uintptr_t) context;
> + wr.next = NULL;
> + wr.sg_list = sgl;
> + wr.num_sge = nsge;
> +
> + return ibv_post_recv(id->qp, &wr, &bad);
> +}
> +
> +static inline int
> +rdma_post_sendv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
> + int nsge, int flags)
> +{
> + struct ibv_send_wr wr, *bad;
> +
> + wr.wr_id = (uintptr_t) context;
> + wr.next = NULL;
> + wr.sg_list = sgl;
> + wr.num_sge = nsge;
> + wr.opcode = IBV_WR_SEND;
> + wr.send_flags = flags;
> +
> + return ibv_post_send(id->qp, &wr, &bad);
> +}
> +
> +static inline int
> +rdma_post_readv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
> + int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
> +{
> + struct ibv_send_wr wr, *bad;
> +
> + wr.wr_id = (uintptr_t) context;
> + wr.next = NULL;
> + wr.sg_list = sgl;
> + wr.num_sge = nsge;
> + wr.opcode = IBV_WR_RDMA_READ;
> + wr.send_flags = flags;
> + wr.wr.rdma.remote_addr = remote_addr;
> + wr.wr.rdma.rkey = rkey;
> +
> + return ibv_post_send(id->qp, &wr, &bad);
> +}
> +
> +static inline int
> +rdma_post_writev(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
> + int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
> +{
> + struct ibv_send_wr wr, *bad;
> +
> + wr.wr_id = (uintptr_t) context;
> + wr.next = NULL;
> + wr.sg_list = sgl;
> + wr.num_sge = nsge;
> + wr.opcode = IBV_WR_RDMA_WRITE;
> + wr.send_flags = flags;
> + wr.wr.rdma.remote_addr = remote_addr;
> + wr.wr.rdma.rkey = rkey;
> +
> + return ibv_post_send(id->qp, &wr, &bad);
> +}
> +
> +/*
> + * Simple send, receive, and RDMA calls.
> + */
> +static inline int
> +rdma_post_recv(struct rdma_cm_id *id, void *context, void *addr,
> + size_t length, struct ibv_mr *mr)
> +{
> + struct ibv_sge sge;
> +
> + assert((addr >= mr->addr) && ((addr + length) <= (mr->addr + mr->length)));
> + sge.addr = (uint64_t) addr;
> + sge.length = (uint32_t) length;
> + sge.lkey = mr->lkey;
> +
> + return rdma_post_recvv(id, context, &sge, 1);
> +}
> +
> +static inline int
> +rdma_post_send(struct rdma_cm_id *id, void *context, void *addr,
> + size_t length, struct ibv_mr *mr, int flags)
> +{
> + struct ibv_sge sge;
> +
> + sge.addr = (uint64_t) addr;
> + sge.length = (uint32_t) length;
> + sge.lkey = mr ? mr->lkey : 0;
> +
> + return rdma_post_sendv(id, context, &sge, 1, flags);
> +}
> +
> +static inline int
> +rdma_post_read(struct rdma_cm_id *id, void *context, void *addr,
> + size_t length, struct ibv_mr *mr, int flags,
> + uint64_t remote_addr, uint32_t rkey)
> +{
> + struct ibv_sge sge;
> +
> + sge.addr = (uint64_t) addr;
> + sge.length = (uint32_t) length;
> + sge.lkey = mr->lkey;
> +
> + return rdma_post_readv(id, context, &sge, 1, flags, remote_addr, rkey);
> +}
> +
> +static inline int
> +rdma_post_write(struct rdma_cm_id *id, void *context, void *addr,
> + size_t length, struct ibv_mr *mr, int flags,
> + uint64_t remote_addr, uint32_t rkey)
> +{
> + struct ibv_sge sge;
> +
> + sge.addr = (uint64_t) addr;
> + sge.length = (uint32_t) length;
> + sge.lkey = mr ? mr->lkey : 0;
> +
> + return rdma_post_writev(id, context, &sge, 1, flags, remote_addr, rkey);
> +}
> +
> +static inline int
> +rdma_post_ud_send(struct rdma_cm_id *id, void *context, void *addr,
> + size_t length, struct ibv_mr *mr, int flags,
> + struct ibv_ah *ah, uint32_t remote_qpn)
> +{
> + struct ibv_send_wr wr, *bad;
> + struct ibv_sge sge;
> +
> + sge.addr = (uint64_t) addr;
> + sge.length = (uint32_t) length;
> + sge.lkey = mr ? mr->lkey : 0;
> +
> + wr.wr_id = (uintptr_t) context;
> + wr.next = NULL;
> + wr.sg_list = &sge;
> + wr.num_sge = 1;
> + wr.opcode = IBV_WR_SEND;
> + wr.send_flags = flags;
> + wr.wr.ud.ah = ah;
> + wr.wr.ud.remote_qpn = remote_qpn;
> + wr.wr.ud.remote_qkey = RDMA_UDP_QKEY;
> +
> + return ibv_post_send(id->qp, &wr, &bad);
> +}
> +
> +static inline int
> +rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
> +{
> + struct ibv_cq *cq;
> + void *context;
> + int ret;
> +
> + ret = ibv_poll_cq(id->send_cq, 1, wc);
> + if (ret)
> + return ret;
> +
> + ret = ibv_req_notify_cq(id->send_cq, 0);
> + if (ret)
> + return ret;
> +
> + ret = ibv_poll_cq(id->send_cq, 1, wc);
> + if (ret)
> + return ret;
> +
> + ret = ibv_get_cq_event(id->send_cq_channel, &cq, &context);
> + if (ret)
> + return ret;
>
This doesn't look correct. If the send isn't complete by the time the
2nd ibv_poll_cq() completes, then this function will return without
having filled in the wc. Or am I missing something? Shouldn't the
ibv_get_cq_event() be the first thing this function does? The same
issue/question exists for rdma_get_recv_comp().
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Ummunotify: progress at last!
From: Roland Dreier @ 2010-04-07 19:37 UTC (permalink / raw)
To: Jason Gunthorpe; +Cc: Jeff Squyres, Linux RDMA List, Brad Benton
In-Reply-To: <20100323201124.GK29129-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
> No, there is no mmap. Like this:
>
> u64 my_counter = 0;
>
> ibv_set_mmu_counter(verbs, &my_counter);
> [..]
> while (my_counter != last_my_counter) {
> last_my_counter = my_counter;
> ibv_get_mmu_notifications(verbs, ...); // <- I am a memory barrier as well
> }
>
> The kernel 'syscall' ibv_set_mmu_counter would bind the given verbs to
> the 8 byte counter you specified without having to the mmap thing. As
> I understand it this is what perfevents does.
I was trying to look at how perf events handles this, and AFAICT it
looks like kernel/perf_event.c just supports mmap(). Can you expand on
what you meant here?
(I was trying to figure out how one would handle the case where
userspace gives us a counter in highmem -- doing kmap_atomic() seems to
be to only option but then I'm not sure if I want to deal with that...)
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 22/37] librdmacm: add new call to create id
From: Jason Gunthorpe @ 2010-04-07 19:34 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <F818D641D3E64DA197E789362F5F0222-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Wed, Apr 07, 2010 at 10:12:44AM -0700, Sean Hefty wrote:
> + * The rdma_cm_id will be set to use synchronous operations (connect,
> + * listen, and get_request). To convert to synchronous operation, the
^^^^^^^^^
asynchronous?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 13/51] IB/qib: Add qib_driver.c
From: Roland Dreier @ 2010-04-07 19:33 UTC (permalink / raw)
To: Steve Wise; +Cc: Ralph Campbell, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBCDBD6.8010809-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Where can I find information on trace events? Something in Documentation/*?
Yep, Documentation/trace/events.txt.
Also look at eg include/trace/events/ext4.h and how the trace_ext4_{xxx}
functions are used in fs/ext4/*.
It's much better than anything you're going to hack up for a single
driver to use.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 26/37] librdmacm: set src_addr in rdma_getaddrinfo
From: Jason Gunthorpe @ 2010-04-07 19:28 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <B7B5C575DF7D4C51AB84AFFAC7A5D5C5-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Wed, Apr 07, 2010 at 10:12:43AM -0700, Sean Hefty wrote:
> RDMA requires the user to allocate hardware resources before
> establishing a connection. To support this, the user must know
> the source address that the connection will use to reach the
> remote endpoint. Modify rdma_getaddrinfo to determine an
> appropriate source address based on the specified destination,
> when a source address is not given.
I haven't looked through everything you posted to make a suggestion
here, but this bothers me..
The resources should be allocated after the rdma_bind syscall, prior to
listen/accept or connect, IMHO.
How does tha rai->ai_src_addr get used to allocate resources anyhow?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] Dimension port order file support
From: Dale Purdy @ 2010-04-07 19:27 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100324133221.GK4808@me>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3544 bytes --]
On Wed, 24 Mar 2010, Sasha Khapyorsky wrote:
> Hi Dale,
>
> On 18:06 Wed 03 Mar , Dale Purdy wrote:
>>
>> Provide a means to specify on a per switch basis the mapping (order)
>> between switch ports and dimensions for Dimension Order Routing. This
>> allows the DOR routing engine to be used when the cabling is not
>> properly aligned for DOR, either initially, or for an upgrade.
>
> Nice stuff.
>
> Is this something useful with ! '-R dor'?
>
I'm not using the dimn_ports array in anything but DOR, but I do think
it could be useful for some of the other routing engines.
>>
>> Signed-off-by: Dale Purdy <purdy-sJ/iWh9BUns@public.gmane.org>
>
> The patch itself is broken somehow - it has double space at start of
> non-changed line (it is fixable with sed -e 's/^ / /', so don't resend
> patch only for this).
>
Yes I see - odd. The original patch file didn't have this - must have
happened when loading it into mail. Hopefully my updated patch will
be ok.
> Some more minor comments are below.
...
>
>> +static int set_dimn_ports(void *ctx, uint64_t guid, char *p)
>> +{
>> + osm_ucast_mgr_t *m = ctx;
>> + osm_node_t *node = osm_get_node_by_guid(m->p_subn, cl_hton64(guid));
>> + osm_switch_t *sw;
>> + uint8_t *dimn_ports = NULL;
>> + uint8_t port;
>> + uint *ports = NULL;
>
> 'uint' is not something standard (we had some build compatibility issues
> with 'uint' in infiniband-diags in the past), so what about 'unsigned
> int'?
>
ok, fixed.
>> + const int bpw = sizeof(*ports)*8;
>> + int words;
>> + int i = 1; /* port 0 maps to port 0 */
>> +
>> + if (!node || !(sw = node->sw)) {
>> + OSM_LOG(m->p_log, OSM_LOG_DEBUG,
>> + "switch with guid 0x%016" PRIx64 " is not found\n",
>> + guid);
>> + return 0;
>> + }
>> +
>> + if (sw->dimn_ports) {
>> + OSM_LOG(m->p_log, OSM_LOG_DEBUG,
>> + "switch with guid 0x%016" PRIx64 " already listed\n",
>> + guid);
>
> It is GIUD double listed case, right? Wouldn't OSM_LOG_VERBOSE be more
> appropriate?
>
fixed.
>> + while ((*p != '\0') && (*p != '#')) {
>> + char *e;
>> +
>> + port = strtoul(p, &e, 0);
>> + if ((p == e) || (port == 0) || (port >= sw->num_ports) ||
>> + !osm_node_get_physp_ptr(node, port)) {
>> + OSM_LOG(m->p_log, OSM_LOG_DEBUG,
>> + "bad port %d specified for guid 0x%016" PRIx64 "\n",
>> + port, guid);
>> + free(dimn_ports);
>> + free(ports);
>
> Ditto.
>
fixed.
>> + return 0;
>> + }
>> +
>> + if (ports[port/bpw] & (1u << (port%bpw))) {
>> + OSM_LOG(m->p_log, OSM_LOG_DEBUG,
>> + "port %d already specified for guid 0x%016" PRIx64 "\n",
>> + port, guid);
>
> Ditto.
>
fixed.
>> + cl_qmap_apply_func(p_sw_guid_tbl, free_dimn_ports, NULL);
>> + if (p_mgr->p_subn->opt.dimn_ports_file) {
>> + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG,
>> + "Fetching dimension ports file \'%s\'\n",
>> + p_mgr->p_subn->opt.dimn_ports_file);
>> + if (parse_node_map(p_mgr->p_subn->opt.dimn_ports_file,
>> + set_dimn_ports, p_mgr)) {
>> + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A05: "
>> + "cannot parse dimn_ports_file \'%s\'\n",
>> + p_mgr->p_subn->opt.dimn_ports_file);
>> + }
>> + }
>> +
>
> Hmm, if it is DOR only it can be done under 'if (is_dor)' (to save
> cycles of other REs). Otherwise (generic usability)
> ucast_mgr_setup_all_switches() seems as more appropriate place to have
> such setup, no?
>
moved to ucast_mgr_setup_all_switches() as you suggested.
> And what about adding:
>
> if (sw->dimn_ports)
> free(dimn_ports);
>
> in osm_switch_delete()?
>
fixed.
New patch attached.
Dale
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: TEXT/x-diff; name=dimn_ports.patch, Size: 14624 bytes --]
Dimension port order file support (V2)
Provide a means to specify on a per switch basis the mapping (order)
between switch ports and dimensions for Dimension Order Routing. This
allows the DOR routing engine to be used when the cabling is not
properly aligned for DOR, either initially, or for an upgrade.
Signed-off-by: Dale Purdy <purdy@sgi.com>
---
opensm/include/opensm/osm_subnet.h | 1 +
opensm/include/opensm/osm_switch.h | 30 +++++++++
opensm/man/opensm.8.in | 31 ++++++++--
opensm/opensm/main.c | 13 ++++-
opensm/opensm/osm_subnet.c | 7 ++
opensm/opensm/osm_switch.c | 4 +-
opensm/opensm/osm_ucast_mgr.c | 116 +++++++++++++++++++++++++++++++++++-
7 files changed, 192 insertions(+), 10 deletions(-)
diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h
index 3970e98..e4e298e 100644
--- a/opensm/include/opensm/osm_subnet.h
+++ b/opensm/include/opensm/osm_subnet.h
@@ -186,6 +186,7 @@ typedef struct osm_subn_opt {
uint16_t console_port;
char *port_prof_ignore_file;
char *hop_weights_file;
+ char *dimn_ports_file;
boolean_t port_profile_switch_nodes;
boolean_t sweep_on_trap;
char *routing_engine_names;
diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h
index cb6e5ac..1c6807e 100644
--- a/opensm/include/opensm/osm_switch.h
+++ b/opensm/include/opensm/osm_switch.h
@@ -100,6 +100,7 @@ typedef struct osm_switch {
uint16_t num_hops;
uint8_t **hops;
osm_port_profile_t *p_prof;
+ uint8_t *dimn_ports;
uint8_t *lft;
uint8_t *new_lft;
uint16_t lft_size;
@@ -871,6 +872,35 @@ static inline uint8_t osm_switch_get_mft_max_position(IN osm_switch_t * p_sw)
* RETURN VALUE
*/
+/****f* OpenSM: Switch/osm_switch_get_dimn_port
+* NAME
+* osm_switch_get_dimn_port
+*
+* DESCRIPTION
+* Get the routing ordered port
+*
+* SYNOPSIS
+*/
+static inline uint8_t osm_switch_get_dimn_port(IN const osm_switch_t * p_sw,
+ IN uint8_t port_num)
+{
+ CL_ASSERT(p_sw);
+ if (p_sw->dimn_ports == NULL)
+ return port_num;
+ return p_sw->dimn_ports[port_num];
+}
+/*
+* PARAMETERS
+* p_sw
+* [in] Pointer to the switch object.
+*
+* port_num
+* [in] Port number in the switch
+*
+* RETURN VALUES
+* Returns the port number ordered for routing purposes.
+*/
+
/****f* OpenSM: Switch/osm_switch_recommend_path
* NAME
* osm_switch_recommend_path
diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in
index 7aca8f9..9053611 100644
--- a/opensm/man/opensm.8.in
+++ b/opensm/man/opensm.8.in
@@ -37,6 +37,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA)
[\-console-port <port>]
[\-i(gnore-guids) <equalize-ignore-guids-file>]
[\-w | \-\-hop_weights_file <path to file>]
+[\-O | \-\-dimn_ports_file <path to file>]
[\-f <log file path> | \-\-log_file <log file path> ]
[\-L | \-\-log_limit <size in MB>] [\-e(rase_log_file)]
[\-P(config) <partition config file> ]
@@ -273,6 +274,16 @@ factor of 1. Lines starting with # are comments. Weights affect only the
output route from the port, so many useful configurations will require weights
to be specified in pairs.
.TP
+\fB\-O\fR, \fB\-\-dimn_ports_file\fR <path to file>
+This option provides a mapping between hypercube dimensions and ports
+on a per switch basis for the DOR routing engine. The file consists
+of lines containing a switch node GUID (specified as a 64 bit hex
+number, with leading 0x) followed by a list of non-zero port numbers,
+separated by spaces, one switch per line. The order for the port
+numbers is in one to one correspondence to the dimensions. Ports not
+listed on a line are assigned to the remaining dimensions, in port
+order. Anything after a # is a comment.
+.TP
\fB\-x\fR, \fB\-\-honor_guid2lid\fR
This option forces OpenSM to honor the guid2lid file,
when it comes out of Standby state, if such file exists
@@ -969,17 +980,20 @@ algorithm and so uses shortest paths. Instead of spreading traffic
out across different paths with the same shortest distance, it chooses
among the available shortest paths based on an ordering of dimensions.
Each port must be consistently cabled to represent a hypercube
-dimension or a mesh dimension. Paths are grown from a destination
-back to a source using the lowest dimension (port) of available paths
-at each step. This provides the ordering necessary to avoid deadlock.
+dimension or a mesh dimension. Alternatively, the -O option can be
+used to assign a custom mapping between the ports on a given switch,
+and the associated dimension. Paths are grown from a destination back
+to a source using the lowest dimension (port) of available paths at
+each step. This provides the ordering necessary to avoid deadlock.
When there are multiple links between any two switches, they still
represent only one dimension and traffic is balanced across them
unless port equalization is turned off. In the case of hypercubes,
the same port must be used throughout the fabric to represent the
-hypercube dimension and match on both ends of the cable. In the case
-of meshes, the dimension should consistently use the same pair of
-ports, one port on one end of the cable, and the other port on the
-other end, continuing along the mesh dimension.
+hypercube dimension and match on both ends of the cable, or the -O
+option used to accomplish the alignment. In the case of meshes, the
+dimension should consistently use the same pair of ports, one port on
+one end of the cable, and the other port on the other end, continuing
+along the mesh dimension, or the -O option used as an override.
Use '-R dor' option to activate the DOR algorithm.
@@ -1111,3 +1125,6 @@ Thomas Sodring
.TP
Ira Weiny
.RI < weiny2@llnl.gov >
+.TP
+Dale Purdy
+.RI < purdy@sgi.com >
diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c
index f9a33af..0093aa7 100644
--- a/opensm/opensm/main.c
+++ b/opensm/opensm/main.c
@@ -275,6 +275,10 @@ static void show_usage(void)
" This option provides the means to define a weighting\n"
" factor per port for customizing the least weight\n"
" hops for the routing.\n\n");
+ printf("--dimn_ports_file, -O <path to file>\n"
+ " This option provides the means to define a mapping\n"
+ " between ports and dimension (Order) for controlling\n"
+ " Dimension Order Routing (DOR).\n\n");
printf("--honor_guid2lid, -x\n"
" This option forces OpenSM to honor the guid2lid file,\n"
" when it comes out of Standby state, if such file exists\n"
@@ -543,7 +547,7 @@ int main(int argc, char *argv[])
char *conf_template = NULL, *config_file = NULL;
uint32_t val;
const char *const short_option =
- "F:c:i:w:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANBIQvVhoryxp:n:q:k:C:G:H:";
+ "F:c:i:w:O:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANBIQvVhoryxp:n:q:k:C:G:H:";
/*
In the array below, the 2nd parameter specifies the number
@@ -560,6 +564,7 @@ int main(int argc, char *argv[])
{"guid", 1, NULL, 'g'},
{"ignore_guids", 1, NULL, 'i'},
{"hop_weights_file", 1, NULL, 'w'},
+ {"dimn_ports_file", 1, NULL, 'O'},
{"lmc", 1, NULL, 'l'},
{"sweep", 1, NULL, 's'},
{"timeout", 1, NULL, 't'},
@@ -696,6 +701,12 @@ int main(int argc, char *argv[])
opt.hop_weights_file);
break;
+ case 'O':
+ opt.dimn_ports_file = optarg;
+ printf(" Dimension Ports File = %s\n",
+ opt.dimn_ports_file);
+ break;
+
case 'g':
/*
Specifies port guid with which to bind.
diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c
index e4126bc..e3f118b 100644
--- a/opensm/opensm/osm_subnet.c
+++ b/opensm/opensm/osm_subnet.c
@@ -324,6 +324,7 @@ static const opt_rec_t opt_tbl[] = {
{ "force_heavy_sweep", OPT_OFFSET(force_heavy_sweep), opts_parse_boolean, NULL, 1 },
{ "port_prof_ignore_file", OPT_OFFSET(port_prof_ignore_file), opts_parse_charp, NULL, 0 },
{ "hop_weights_file", OPT_OFFSET(hop_weights_file), opts_parse_charp, NULL, 0 },
+ { "dimn_ports_file", OPT_OFFSET(dimn_ports_file), opts_parse_charp, NULL, 0 },
{ "port_profile_switch_nodes", OPT_OFFSET(port_profile_switch_nodes), opts_parse_boolean, NULL, 1 },
{ "sweep_on_trap", OPT_OFFSET(sweep_on_trap), opts_parse_boolean, NULL, 1 },
{ "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, NULL, 0 },
@@ -742,6 +743,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt)
p_opt->accum_log_file = TRUE;
p_opt->port_prof_ignore_file = NULL;
p_opt->hop_weights_file = NULL;
+ p_opt->dimn_ports_file = NULL;
p_opt->port_profile_switch_nodes = FALSE;
p_opt->sweep_on_trap = TRUE;
p_opt->use_ucast_cache = FALSE;
@@ -1385,6 +1387,11 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)
p_opts->hop_weights_file ? p_opts->hop_weights_file : null_str);
fprintf(out,
+ "# The file holding non-default port order per switch for DOR routing \n"
+ "dimn_ports_file %s\n\n",
+ p_opts->dimn_ports_file ? p_opts->dimn_ports_file : null_str);
+
+ fprintf(out,
"# Routing engine\n"
"# Multiple routing engines can be specified separated by\n"
"# commas so that specific ordering of routing algorithms will\n"
diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c
index 1cd8bfc..20fa098 100644
--- a/opensm/opensm/osm_switch.c
+++ b/opensm/opensm/osm_switch.c
@@ -78,6 +78,8 @@ void osm_switch_delete(IN OUT osm_switch_t ** pp_sw)
osm_mcast_tbl_destroy(&p_sw->mcast_tbl);
if (p_sw->p_prof)
free(p_sw->p_prof);
+ if (p_sw->dimn_ports)
+ free(p_sw->dimn_ports);
if (p_sw->lft)
free(p_sw->lft);
if (p_sw->new_lft)
@@ -341,7 +343,7 @@ uint8_t osm_switch_recommend_path(IN const osm_switch_t * p_sw,
/* port number starts with one and num_ports is 1 + num phys ports */
for (i = start_from; i < start_from + num_ports; i++) {
- port_num = i%num_ports;
+ port_num = osm_switch_get_dimn_port(p_sw, i%num_ports);
if (!port_num ||
osm_switch_get_hop_count(p_sw, base_lid, port_num) !=
least_hops)
diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c
index 7ec58b5..e94370f 100644
--- a/opensm/opensm/osm_ucast_mgr.c
+++ b/opensm/opensm/osm_ucast_mgr.c
@@ -46,6 +46,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#include <ctype.h>
#include <iba/ib_types.h>
#include <complib/cl_qmap.h>
#include <complib/cl_debug.h>
@@ -488,6 +489,102 @@ static void set_default_hop_wf(cl_map_item_t * p_map_item, void *ctx)
}
}
+static int set_dimn_ports(void *ctx, uint64_t guid, char *p)
+{
+ osm_subn_t *p_subn = ctx;
+ osm_node_t *node = osm_get_node_by_guid(p_subn, cl_hton64(guid));
+ osm_switch_t *sw;
+ uint8_t *dimn_ports = NULL;
+ uint8_t port;
+ unsigned int *ports = NULL;
+ const int bpw = sizeof(*ports)*8;
+ int words;
+ int i = 1; /* port 0 maps to port 0 */
+
+ if (!node || !(sw = node->sw)) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_VERBOSE,
+ "switch with guid 0x%016" PRIx64 " is not found\n",
+ guid);
+ return 0;
+ }
+
+ if (sw->dimn_ports) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_VERBOSE,
+ "switch with guid 0x%016" PRIx64 " already listed\n",
+ guid);
+ return 0;
+ }
+
+ dimn_ports = malloc(sizeof(*dimn_ports)*sw->num_ports);
+ if (!dimn_ports) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_ERROR,
+ "ERR 3A07: cannot allocate memory for dimn_ports\n");
+ return -1;
+ }
+ memset(dimn_ports, 0, sizeof(*dimn_ports)*sw->num_ports);
+
+ /* the ports array is for record keeping of which ports have
+ * been seen */
+ words = (sw->num_ports + bpw - 1)/bpw;
+ ports = malloc(words*sizeof(*ports));
+ if (!ports) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_ERROR,
+ "ERR 3A08: cannot allocate memory for ports\n");
+ return -1;
+ }
+ memset(ports, 0, words*sizeof(*ports));
+
+ while ((*p != '\0') && (*p != '#')) {
+ char *e;
+
+ port = strtoul(p, &e, 0);
+ if ((p == e) || (port == 0) || (port >= sw->num_ports) ||
+ !osm_node_get_physp_ptr(node, port)) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_VERBOSE,
+ "bad port %d specified for guid 0x%016" PRIx64 "\n",
+ port, guid);
+ free(dimn_ports);
+ free(ports);
+ return 0;
+ }
+
+ if (ports[port/bpw] & (1u << (port%bpw))) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_VERBOSE,
+ "port %d already specified for guid 0x%016" PRIx64 "\n",
+ port, guid);
+ free(dimn_ports);
+ free(ports);
+ return 0;
+ }
+
+ ports[port/bpw] |= (1u << (port%bpw));
+ dimn_ports[i++] = port;
+
+ p = e;
+ while (isspace(*p)) {
+ p++;
+ }
+ }
+
+ if (i > 1) {
+ for (port = 1; port < sw->num_ports; port++) {
+ /* fill out the rest of the dimn_ports array
+ * in sequence using the remaining unspecified
+ * ports.
+ */
+ if (!(ports[port/bpw] & (1u << (port%bpw)))) {
+ dimn_ports[i++] = port;
+ }
+ }
+ sw->dimn_ports = dimn_ports;
+ } else {
+ free(dimn_ports);
+ }
+
+ free(ports);
+ return 0;
+}
+
int osm_ucast_mgr_build_lid_matrices(IN osm_ucast_mgr_t * p_mgr)
{
uint32_t i;
@@ -579,7 +676,7 @@ static int ucast_mgr_setup_all_switches(osm_subn_t * p_subn)
for (p_sw = (osm_switch_t *) cl_qmap_head(&p_subn->sw_guid_tbl);
p_sw != (osm_switch_t *) cl_qmap_end(&p_subn->sw_guid_tbl);
- p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item))
+ p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item)) {
if (osm_switch_prepare_path_rebuild(p_sw, lids)) {
OSM_LOG(&p_subn->p_osm->log, OSM_LOG_ERROR, "ERR 3A0B: "
"cannot setup switch 0x%016" PRIx64 "\n",
@@ -587,6 +684,23 @@ static int ucast_mgr_setup_all_switches(osm_subn_t * p_subn)
(p_sw->p_node)));
return -1;
}
+ if (p_sw->dimn_ports) {
+ free(p_sw->dimn_ports);
+ p_sw->dimn_ports = NULL;
+ }
+ }
+
+ if (p_subn->opt.dimn_ports_file) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_DEBUG,
+ "Fetching dimension ports file \'%s\'\n",
+ p_subn->opt.dimn_ports_file);
+ if (parse_node_map(p_subn->opt.dimn_ports_file,
+ set_dimn_ports, p_subn)) {
+ OSM_LOG(&p_subn->p_osm->log, OSM_LOG_ERROR, "ERR 3A05: "
+ "cannot parse dimn_ports_file \'%s\'\n",
+ p_subn->opt.dimn_ports_file);
+ }
+ }
return 0;
}
--
1.7.0
^ permalink raw reply related
* Re: [PATCH v2 13/51] IB/qib: Add qib_driver.c
From: Steve Wise @ 2010-04-07 19:24 UTC (permalink / raw)
To: Roland Dreier; +Cc: Ralph Campbell, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <adafx3743n6.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier wrote:
> > +unsigned qib_debug;
> > +module_param_named(debug, qib_debug, uint, S_IWUSR | S_IRUGO);
> > +MODULE_PARM_DESC(debug, "mask for debug prints");
>
> Did you look at using trace events for this stuff? That gives you
> extremely low overhead when tracing is turned off (dynamic patching to
> NOP out the tracing when it's disabled) and also very fine-grained (per
> trace site) control over what gets printed; plus you get dumping of the
> trace buffer on crash, etc.
>
> - R.
>
Where can I find information on trace events? Something in Documentation/*?
Thanks,
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 11/37] librdmacm: add zalloc call
From: Jason Gunthorpe @ 2010-04-07 19:19 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <53A8AB949BCE4311B58A4881EC549E90-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Wed, Apr 07, 2010 at 10:12:44AM -0700, Sean Hefty wrote:
> Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>
> src/cma.c | 6 ++----
> src/cma.h | 10 ++++++++++
> 2 files changed, 12 insertions(+), 4 deletions(-)
>
> diff --git a/src/cma.c b/src/cma.c
> index a85448b..4025aeb 100644
> +++ b/src/cma.c
> @@ -342,11 +342,10 @@ static struct cma_id_private *ucma_alloc_id(struct rdma_event_channel *channel,
> {
> struct cma_id_private *id_priv;
>
> - id_priv = malloc(sizeof *id_priv);
> + id_priv = zalloc(sizeof *id_priv);
> if (!id_priv)
> return NULL;
Maybe use calloc(,1 sizeof(x)); instead?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 9/37] librdmacm: add support for PF_IB multicast addresses
From: Sean Hefty @ 2010-04-07 18:13 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Allow the user to specify PF_IB addresses when joining
multicast groups. PF_IB addresses contain MGIDs directly.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
include/rdma/rdma_cma_abi.h | 12 +++++++++++-
src/cma.c | 44 +++++++++++++++++++++++++++++++------------
2 files changed, 43 insertions(+), 13 deletions(-)
diff --git a/include/rdma/rdma_cma_abi.h b/include/rdma/rdma_cma_abi.h
index 4a7a55d..c3981e6 100644
--- a/include/rdma/rdma_cma_abi.h
+++ b/include/rdma/rdma_cma_abi.h
@@ -68,7 +68,8 @@ enum {
UCMA_CMD_MIGRATE_ID,
UCMA_CMD_QUERY,
UCMA_CMD_BIND,
- UCMA_CMD_RESOLVE_ADDR
+ UCMA_CMD_RESOLVE_ADDR,
+ UCMA_CMD_JOIN_MCAST
};
struct ucma_abi_cmd_hdr {
@@ -243,6 +244,15 @@ struct ucma_abi_join_ip_mcast {
__u32 id;
};
+struct ucma_abi_join_mcast {
+ __u64 response; /* rdma_ucma_create_id_resp */
+ __u64 uid;
+ __u32 id;
+ __u16 addr_size;
+ __u16 reserved;
+ struct sockaddr_storage addr;
+};
+
struct ucma_abi_get_event {
__u64 response;
};
diff --git a/src/cma.c b/src/cma.c
index e22e1b4..c83d9d2 100644
--- a/src/cma.c
+++ b/src/cma.c
@@ -1244,21 +1244,16 @@ int rdma_disconnect(struct rdma_cm_id *id)
return 0;
}
-int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr,
- void *context)
+static int rdma_join_multicast2(struct rdma_cm_id *id, struct sockaddr *addr,
+ socklen_t addrlen, void *context)
{
- struct ucma_abi_join_ip_mcast *cmd;
struct ucma_abi_create_id_resp *resp;
struct cma_id_private *id_priv;
struct cma_multicast *mc, **pos;
void *msg;
- int ret, size, addrlen;
+ int ret, size;
id_priv = container_of(id, struct cma_id_private, id);
- addrlen = ucma_addrlen(addr);
- if (!addrlen)
- return ERR(EINVAL);
-
mc = malloc(sizeof *mc);
if (!mc)
return ERR(ENOMEM);
@@ -1277,10 +1272,23 @@ int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr,
id_priv->mc_list = mc;
pthread_mutex_unlock(&id_priv->mut);
- CMA_CREATE_MSG_CMD_RESP(msg, cmd, resp, UCMA_CMD_JOIN_IP_MCAST, size);
- cmd->id = id_priv->handle;
- memcpy(&cmd->addr, addr, addrlen);
- cmd->uid = (uintptr_t) mc;
+ if (af_ib_support) {
+ struct ucma_abi_join_mcast *cmd;
+
+ CMA_CREATE_MSG_CMD_RESP(msg, cmd, resp, UCMA_CMD_JOIN_MCAST, size);
+ cmd->id = id_priv->handle;
+ memcpy(&cmd->addr, addr, addrlen);
+ cmd->addr_size = addrlen;
+ cmd->uid = (uintptr_t) mc;
+ cmd->reserved = 0;
+ } else {
+ struct ucma_abi_join_ip_mcast *cmd;
+
+ CMA_CREATE_MSG_CMD_RESP(msg, cmd, resp, UCMA_CMD_JOIN_IP_MCAST, size);
+ cmd->id = id_priv->handle;
+ memcpy(&cmd->addr, addr, addrlen);
+ cmd->uid = (uintptr_t) mc;
+ }
ret = write(id->channel->fd, msg, size);
if (ret != size) {
@@ -1303,6 +1311,18 @@ err1:
return ret;
}
+int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr,
+ void *context)
+{
+ int addrlen;
+
+ addrlen = ucma_addrlen(addr);
+ if (!addrlen)
+ return ERR(EINVAL);
+
+ return rdma_join_multicast2(id, addr, addrlen, context);
+}
+
int rdma_leave_multicast(struct rdma_cm_id *id, struct sockaddr *addr)
{
struct ucma_abi_destroy_id *cmd;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/37] librdmacm: add support for AF_IB
From: Sean Hefty @ 2010-04-07 18:13 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
The following patch series adds several enhancements to the librdmacm
intended to simplify using RDMA devices and address scalability issues.
Major changes include:
* Adding support for AF_IB.
* The addition of a new API: rdma_getaddrinfo. This call provides
functionality similar to getaddrinfo for RDMA devices. In addition
to resolving names to addresses, it can also resolve route and
connection data. rdma_getaddrinfo can return addresses using
AF_INET, AF_INET6, and AF_IB.
* Add support for IB ACM. IB ACM defines a socket based protocol to
an IB address and route resolution service. One implementation of that
service is provided separately, but anyone can implement the service
provided that they adhere to the IB ACM communication protocol.
Use of IB ACM is not required.
* Support synchronous operation for library calls. Users can control
whether an rdma_cm_id operates asynchronously or synchronously based on
the rdma_event_channel parameter. Use of synchronous operations
reduces the amount of application code required to use the librdmacm.
* Allow the library to abstract RDMA resource creation for simpler RDMA
applications. The library can now allocate PDs, CQs, and QPs for the
user, if not provided.
* Provide a set of helper verbs calls for posting work requests and
checking for completions. These are simple wrappers around libibverbs
calls.
This patch series is also available through my git tree at:
git://git.openfabrics.org/~shefty/librdmacm.git af_ib
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] [RFC] ummunotify: Userspace support for MMU notifications
From: Randy Dunlap @ 2010-04-07 17:38 UTC (permalink / raw)
To: Eric B Munson
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI
In-Reply-To: <1270643429-20688-1-git-send-email-ebmunson-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
On Wed, 7 Apr 2010 13:30:29 +0100 Eric B Munson wrote:
> Signed-off-by: Roland Dreier <rolandd <at> cisco.com>
Use unobfuscated @.
> Signed-off-by: Eric B Munson <ebmunson-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
>
> ---
>
> Changes since v3:
> - Fixed replaced [get|put] user with copy_[from|to]_user to fix x86
> builds
> ---
> Documentation/Makefile | 3 +-
> drivers/char/Kconfig | 12 +
> drivers/char/Makefile | 1 +
> drivers/char/ummunotify.c | 567 +++++++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 582 insertions(+), 1 deletions(-)
> create mode 100644 drivers/char/ummunotify.c
>
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 6fc7ea1..27ba76a 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -1,3 +1,4 @@
> obj-m := DocBook/ accounting/ auxdisplay/ connector/ \
> filesystems/ filesystems/configfs/ ia64/ laptops/ networking/ \
> - pcmcia/ spi/ timers/ video4linux/ vm/ watchdog/src/
> + pcmcia/ spi/ timers/ video4linux/ vm/ ummunotify/ \
> + watchdog/src/
What is this change to Documentation/Makefile for?
Is there some file that should be added in Documentation/ummunotify/ ?
> diff --git a/drivers/char/Kconfig b/drivers/char/Kconfig
> index 3141dd3..cf26019 100644
> --- a/drivers/char/Kconfig
> +++ b/drivers/char/Kconfig
> @@ -1111,6 +1111,18 @@ config DEVPORT
> depends on ISA || PCI
> default y
>
> +config UMMUNOTIFY
> + tristate "Userspace MMU notifications"
> + select MMU_NOTIFIER
> + help
> + The ummunotify (userspace MMU notification) driver creates a
> + character device that can be used by userspace libraries to
> + get notifications when an application's memory mapping
> + changed. This is used, for example, by RDMA libraries to
> + improve the reliability of memory registration caching, since
> + the kernel's MMU notifications can be used to know precisely
> + when to shoot down a cached registration.
> +
> source "drivers/s390/char/Kconfig"
>
> endmenu
> diff --git a/drivers/char/Makefile b/drivers/char/Makefile
> index f957edf..521e5de 100644
> --- a/drivers/char/Makefile
> +++ b/drivers/char/Makefile
> @@ -97,6 +97,7 @@ obj-$(CONFIG_NSC_GPIO) += nsc_gpio.o
> obj-$(CONFIG_CS5535_GPIO) += cs5535_gpio.o
> obj-$(CONFIG_GPIO_TB0219) += tb0219.o
> obj-$(CONFIG_TELCLOCK) += tlclk.o
> +obj-$(CONFIG_UMMUNOTIFY) += ummunotify.o
>
> obj-$(CONFIG_MWAVE) += mwave/
> obj-$(CONFIG_AGP) += agp/
---
~Randy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox