* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Ira Weiny @ 2010-04-13 20:44 UTC (permalink / raw)
To: Ira Weiny
Cc: Sasha Khapyorsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413133826.00a8afc5.weiny2-i2BcT+NCU+M@public.gmane.org>
On Tue, 13 Apr 2010 13:38:26 -0700
Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> This changes the logic. "num_smps_outstanding" is NOT the number on the wire, but it appears you have made it so. This is the number which will cause process_smp_queue to continue being called.
>
> If you are going to do this I think you need to change process_mads as well as process_one_recv. We discussed process_one_recv in the error case.
>
> What were you trying to fix?
Ok, I think I see. We should move cl_qmap_insert to after a successful umad_send and putting total_smps here is ok. But num_smps_outstanding should be put back I think.
Ira
>
> Ira
>
> On Tue, 13 Apr 2010 19:38:36 +0300
> Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
>
> >
> > send_smp() failure blocks whole discovery execution (it never returns).
> > Fix this by updating all outstanding SMPs related counters only after
> > successful MAD sending.
> >
> > Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
> > ---
> > infiniband-diags/libibnetdisc/src/query_smp.c | 8 ++++----
> > 1 files changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
> > index b4322bc..08e3ef7 100644
> > --- a/infiniband-diags/libibnetdisc/src/query_smp.c
> > +++ b/infiniband-diags/libibnetdisc/src/query_smp.c
> > @@ -96,10 +96,12 @@ static int process_smp_queue(smp_engine_t * engine)
> > if (!smp)
> > return 0;
> >
> > - cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> > - (cl_map_item_t *) smp);
> > if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
> > return rc;
> > + engine->num_smps_outstanding++;
> > + cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> > + (cl_map_item_t *) smp);
> > + engine->total_smps++;
> > }
> > return 0;
> > }
> > @@ -133,8 +135,6 @@ int issue_smp(smp_engine_t * engine, ib_portid_t * portid,
> > portid->sl = 0;
> > portid->qp = 0;
> >
> > - engine->total_smps++;
> > - engine->num_smps_outstanding++;
> > queue_smp(engine, smp);
> > return process_smp_queue(engine);
> > }
> > --
> > 1.7.0.4
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://*vger.kernel.org/majordomo-info.html
> >
>
>
> --
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Ira Weiny @ 2010-04-13 20:38 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413163836.GM10830@me>
This changes the logic. "num_smps_outstanding" is NOT the number on the wire, but it appears you have made it so. This is the number which will cause process_smp_queue to continue being called.
If you are going to do this I think you need to change process_mads as well as process_one_recv. We discussed process_one_recv in the error case.
What were you trying to fix?
Ira
On Tue, 13 Apr 2010 19:38:36 +0300
Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
>
> send_smp() failure blocks whole discovery execution (it never returns).
> Fix this by updating all outstanding SMPs related counters only after
> successful MAD sending.
>
> Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
> ---
> infiniband-diags/libibnetdisc/src/query_smp.c | 8 ++++----
> 1 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
> index b4322bc..08e3ef7 100644
> --- a/infiniband-diags/libibnetdisc/src/query_smp.c
> +++ b/infiniband-diags/libibnetdisc/src/query_smp.c
> @@ -96,10 +96,12 @@ static int process_smp_queue(smp_engine_t * engine)
> if (!smp)
> return 0;
>
> - cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> - (cl_map_item_t *) smp);
> if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
> return rc;
> + engine->num_smps_outstanding++;
> + cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> + (cl_map_item_t *) smp);
> + engine->total_smps++;
> }
> return 0;
> }
> @@ -133,8 +135,6 @@ int issue_smp(smp_engine_t * engine, ib_portid_t * portid,
> portid->sl = 0;
> portid->qp = 0;
>
> - engine->total_smps++;
> - engine->num_smps_outstanding++;
> queue_smp(engine, smp);
> return process_smp_queue(engine);
> }
> --
> 1.7.0.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://*vger.kernel.org/majordomo-info.html
>
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Acupuncturists - 23,988 records 1,826 emails
From: zealot Minor @ 2010-04-13 20:35 UTC (permalink / raw)
To: whelanmphrd-XauvlLoUTru+XT7JhA+gdA,
garewade-rphTv4pjVZMJGwgDXS7ZQA,
vhoe-YD6nxZs2PyH5UeND2KvAfvWi4VT69sRu,
illmail-FXJ3zaHstaO0ok1jKN7RBg,
mdownsborough-IPPw5sWYmMS35wUkaLeaMCp2UmYkHbXO
Oscar,
Contact me at the email address below for a catalog of all our USA lists.
Also, this week we are offering reduced costs for more than one list.
Regards,
Abby Lin
Email Address: Albert.Shook @ expertdata2010.co.cc
Forward email to rembox @ expertdata2010.co.cc to purge you from our records
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Ira Weiny @ 2010-04-13 20:30 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413132531.GI10830@me>
On Tue, 13 Apr 2010 16:25:31 +0300
Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> On 13:46 Tue 13 Apr , Sasha Khapyorsky wrote:
> >
> > However see some comments and questions below.
>
> Another thought. What about API like:
>
> ibnd_discover_fabric(cosnt char *ca_name, unsigned port_num,
> struct ibnd_config *cfg);
>
> So libibnetdisc will be responsible for opening and closing its own port
> and in this way will be fully reenterable?
If we are going to do something like this why not more like a context?
Something like this?
ibqueryerrors.c
query_errors_ibmad_port = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 4);
{
ibnd_context_t ctx;
ctx.ca_name = ibd_ca;
ctx.ca_port = ibd_ca_port;
ctx.portid = portid;
ctx.hops = hops;
fabric = ibnd_discover_fabric(&ctx); /* opens it's own ibmad_port */
}
...
Ira
>
> Sasha
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-13 20:22 UTC (permalink / raw)
To: Steve Wise
Cc: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC4D22A.6020700-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5 code
>>> and it appears to do nothing in the map_phys_fmr functions.
>>>
>>
>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>
>No, but SDP uses FMRs. I don't think it will run without FMR support.
>
Good to know. Thanks.
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-13 20:20 UTC (permalink / raw)
To: Tung, Chien Tin
Cc: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <2EFBCAEF10980645BBCFB605689E08E9047C7405A0-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
Tung, Chien Tin wrote:
>>> NE020 supports 4 SGEs. I don't know enough about SDP to know why it is
>>> using this calculation for # of send_sge:
>>>
>>> #define SDP_MAX_RECV_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE)
>>>
>>>
>>>
>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5 code
>> and it appears to do nothing in the map_phys_fmr functions.
>>
>
> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>
No, but SDP uses FMRs. I don't think it will run without FMR support.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-13 20:02 UTC (permalink / raw)
To: Steve Wise
Cc: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC49E2B.3000804-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>> NE020 supports 4 SGEs. I don't know enough about SDP to know why it is
>> using this calculation for # of send_sge:
>>
>> #define SDP_MAX_RECV_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE)
>>
>>
>
>Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5 code
>and it appears to do nothing in the map_phys_fmr functions.
We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Ira Weiny @ 2010-04-13 18:24 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413171824.GP10830@me>
On Tue, 13 Apr 2010 20:18:24 +0300
Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> On 13:46 Tue 13 Apr , Sasha Khapyorsky wrote:
> >
> > > +static int recv_node_info(smp_engine_t *engine, ibnd_smp_t * smp,
> > > + uint8_t *mad, void *cb_data)
> > > +{
> > > + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> > > + int i = 0;
> > > + uint8_t *node_info = mad + IB_SMP_DATA_OFFS;
> > > + ibnd_node_t * rem_node = (ibnd_node_t *)cb_data;
> > > ibnd_node_t *node;
> > > + int node_is_new = 0;
> > > + uint64_t node_guid = mad_get_field64(node_info, 0, IB_NODE_GUID_F);
> > > + uint64_t port_guid = mad_get_field64(node_info, 0, IB_NODE_PORT_GUID_F);
> > > + int port_num = mad_get_field(node_info, 0, IB_NODE_LOCAL_PORT_F);
> > > + ibnd_port_t *port = NULL;
> > >
> > > - for (node = fabric->nodestbl[hash]; node; node = node->htnext)
> > > - if (node->guid == new->guid)
> > > - return node;
> > > + node = ibnd_find_node_guid(fabric, node_guid);
> > > + if (!node) {
> > > + node = create_node(engine, &smp->path, node_info);
> > > + if (!node)
> > > + return -1;
> > > + node_is_new = 1;
> > > + }
> > > + IBND_DEBUG("Found %s node GUID %lx (%s)\n",
> > > + (node_is_new) ? "new": "old", node->guid,
> > > + portid2str(&smp->path));
> > >
> > > - return NULL;
> > > + port = node->ports[port_num];
> > > + if (!port) {
> > > + /* If we have not see this port before create a shell for it */
> > > + port = node->ports[port_num] = calloc(1, sizeof(*port));
> > > + port->node = node;
> > > + port->portnum = port_num;
> > > + }
> > > + port->guid = port_guid;
> > > +
> > > + if (rem_node == NULL) /* this is the start node */
> > > + fabric->from_node = node;
> > > + else {
> > > + /* link ports... */
> > > + int rem_port_num = get_last_port(&smp->path);
> > > +
> > > + if (!rem_node->ports[rem_port_num]) {
> > > + IBND_ERROR("Internal Error; "
> > > + "Node(%p) %lx Port %d no port created!?!?!?\n\n",
> > > + rem_node, rem_node->guid, rem_port_num);
> > > + return (-1);
> > > + }
> > > +
> > > + link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
> > > + }
> > > +
> > > + if (!node_is_new)
> > > + return 0;
> > > +
> > > + query_node_desc(engine, &smp->path, node);
> > > +
> > > + if (node->type == IB_NODE_SWITCH)
> > > + query_switch_info(engine, &smp->path, node);
> > > +
> > > + /* process all the ports on this node */
> > > + for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
> > > + i <= node->numports; i++) {
> > > + query_port_info(engine, &smp->path, node, i);
> > > + }
>
> Actually it seems possible to save some MADs by not querying CA/Router
> ports which is not connected to our fabric. Something like this:
>
> diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
> index 2ce15b7..1c7d6f2 100644
> --- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
> +++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
> @@ -339,19 +339,16 @@ static int recv_node_info(smp_engine_t * engine, ibnd_smp_t * smp,
> link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
> }
>
> - if (!node_is_new)
> - return 0;
> -
> - query_node_desc(engine, &smp->path, node);
> -
> - if (node->type == IB_NODE_SWITCH)
> - query_switch_info(engine, &smp->path, node);
> + if (node_is_new) {
> + query_node_desc(engine, &smp->path, node);
>
> - /* process all the ports on this node */
> - for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
> - i <= node->numports; i++) {
> - query_port_info(engine, &smp->path, node, i);
> - }
> + if (node->type == IB_NODE_SWITCH) {
> + query_switch_info(engine, &smp->path, node);
> + for (i = 0; i <= node->numports; i++)
> + query_port_info(engine, &smp->path, node, i);
> + }
> + } else if (node->type != IB_NODE_SWITCH)
> + query_port_info(engine, &smp->path, node, port_num);
>
> return 0;
> }
>
>
> What do you think?
I like this better.
Ira
>
> Sasha
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Peter Zijlstra @ 2010-04-13 18:02 UTC (permalink / raw)
To: Roland Dreier
Cc: Håkon Bugge, Jason Gunthorpe, Andrew Morton, Eric B Munson,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
pavel-+ZI9xUNit7I, mingo-X9Un+BFzKDI,
jsquyres-FYB4Gu1CFyUAvxtiuMwx3w
In-Reply-To: <adahbnfme0j.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Tue, 2010-04-13 at 10:57 -0700, Roland Dreier wrote:
> Are those system calls the only possible way that virtual to physical
> mappings can change? Can't page migration or something like that
> potentially affect things? And even if you did have hooks into every
> system call that mattered (keep in mind that relying on glibc is not
> enough, since an MPI application may not use glibc) would decoding them
> and figuring out what happened really be preferable to a single event
> type that tells you exactly what address range was affected?
Yeah, virtual<->physical maps can change through swapping, page
migration, memory compaction, huge-page aggregation (the latter two not
yet being upstream).
Even mlock() doesn't pin virtual<->physical maps.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Roland Dreier @ 2010-04-13 17:57 UTC (permalink / raw)
To: Håkon Bugge
Cc: Jason Gunthorpe, Andrew Morton, Eric B Munson,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI, jsquyres-FYB4Gu1CFyUAvxtiuMwx3w
In-Reply-To: <3251DDDA-D705-4B1E-9595-9C24709EF146-U0mLk4xYmo8@public.gmane.org>
> I am not sure I agree with the premises here. ptMalloc and malloc
> hooks are not related to the issue in my opinion. User space library
> calls do not change virtual to physical mapping, system calls do. The
> following sys calls might change virtual to physical mapping:
> munmap(), mremap(), sbrk(), madvice(). What we need is glibc to
> provide hooks for these 4 sys calls and the general syscall() when
> its argument is one of the four mentioned syscalls. To me, that is
> what is needed, and the ummunotify direction seems way too
> complicated to me.
Are those system calls the only possible way that virtual to physical
mappings can change? Can't page migration or something like that
potentially affect things? And even if you did have hooks into every
system call that mattered (keep in mind that relying on glibc is not
enough, since an MPI application may not use glibc) would decoding them
and figuring out what happened really be preferable to a single event
type that tells you exactly what address range was affected?
> It is further claimed that "… other tricks are not robust". I wrote
> the code used in Scali/Platform MPI handling the issue. I do not
> think its fair to claim that this MPI is not robust in this matter
> nor that is performance is bad.
The Open MPI developers have spent a lot of effort trying to handle this
purely in userspace and still do not believe that a truly robust
solution is possible without kernel help. Perhaps they can expand on
what the obstacles are.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH V3 0/2] Add support for enhanced atomic operations
From: Roland Dreier @ 2010-04-13 17:43 UTC (permalink / raw)
To: Vladimir Sokolovsky; +Cc: Håkon Bugge, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BC2EB7F.20804-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
> Do you propose to use IB_ATOMIC_GLOB instead of IB_ATOMIC_HCA while setting
> atomic capability in the code below?
>
> props->atomic_cap = dev->dev->caps.flags &
> MLX4_DEV_CAP_FLAG_ATOMIC ?
> IB_ATOMIC_HCA : IB_ATOMIC_NONE;
>
> Or add IB_MASKED_ATOMIC to ib_atomic_cap enum and use this one
> instead of IB_ATOMIC_HCA?
>
> All this, of course, comes to replace setting
> IB_DEVICE_MASKED_ATOMIC for device capability.
Sorry, missed this reply first time around I guess.
Anyway, no I don't think we should use ATOMIC_GLOB unless an HCA truly
provides global atomic ops; not possible without PCIe 3.0 atomic support
I think. But as I said, I think the original suggestion:
> > Hence, I think it would be cleaner if a new capability,
> > masked_atomic_cap, were introduced, using the original definitions
> > (NONE, HCA, GLOB).
makes sense, ie create a new props->masked_atomic_cap and set it to
NONE, HCA or GLOB as appropriate (ie NONE for everyone except mlx4, and
HCA for mlx4).
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Sasha Khapyorsky @ 2010-04-13 17:18 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413104658.GA10830@me>
On 13:46 Tue 13 Apr , Sasha Khapyorsky wrote:
>
> > +static int recv_node_info(smp_engine_t *engine, ibnd_smp_t * smp,
> > + uint8_t *mad, void *cb_data)
> > +{
> > + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> > + int i = 0;
> > + uint8_t *node_info = mad + IB_SMP_DATA_OFFS;
> > + ibnd_node_t * rem_node = (ibnd_node_t *)cb_data;
> > ibnd_node_t *node;
> > + int node_is_new = 0;
> > + uint64_t node_guid = mad_get_field64(node_info, 0, IB_NODE_GUID_F);
> > + uint64_t port_guid = mad_get_field64(node_info, 0, IB_NODE_PORT_GUID_F);
> > + int port_num = mad_get_field(node_info, 0, IB_NODE_LOCAL_PORT_F);
> > + ibnd_port_t *port = NULL;
> >
> > - for (node = fabric->nodestbl[hash]; node; node = node->htnext)
> > - if (node->guid == new->guid)
> > - return node;
> > + node = ibnd_find_node_guid(fabric, node_guid);
> > + if (!node) {
> > + node = create_node(engine, &smp->path, node_info);
> > + if (!node)
> > + return -1;
> > + node_is_new = 1;
> > + }
> > + IBND_DEBUG("Found %s node GUID %lx (%s)\n",
> > + (node_is_new) ? "new": "old", node->guid,
> > + portid2str(&smp->path));
> >
> > - return NULL;
> > + port = node->ports[port_num];
> > + if (!port) {
> > + /* If we have not see this port before create a shell for it */
> > + port = node->ports[port_num] = calloc(1, sizeof(*port));
> > + port->node = node;
> > + port->portnum = port_num;
> > + }
> > + port->guid = port_guid;
> > +
> > + if (rem_node == NULL) /* this is the start node */
> > + fabric->from_node = node;
> > + else {
> > + /* link ports... */
> > + int rem_port_num = get_last_port(&smp->path);
> > +
> > + if (!rem_node->ports[rem_port_num]) {
> > + IBND_ERROR("Internal Error; "
> > + "Node(%p) %lx Port %d no port created!?!?!?\n\n",
> > + rem_node, rem_node->guid, rem_port_num);
> > + return (-1);
> > + }
> > +
> > + link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
> > + }
> > +
> > + if (!node_is_new)
> > + return 0;
> > +
> > + query_node_desc(engine, &smp->path, node);
> > +
> > + if (node->type == IB_NODE_SWITCH)
> > + query_switch_info(engine, &smp->path, node);
> > +
> > + /* process all the ports on this node */
> > + for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
> > + i <= node->numports; i++) {
> > + query_port_info(engine, &smp->path, node, i);
> > + }
Actually it seems possible to save some MADs by not querying CA/Router
ports which is not connected to our fabric. Something like this:
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
index 2ce15b7..1c7d6f2 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
@@ -339,19 +339,16 @@ static int recv_node_info(smp_engine_t * engine, ibnd_smp_t * smp,
link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
}
- if (!node_is_new)
- return 0;
-
- query_node_desc(engine, &smp->path, node);
-
- if (node->type == IB_NODE_SWITCH)
- query_switch_info(engine, &smp->path, node);
+ if (node_is_new) {
+ query_node_desc(engine, &smp->path, node);
- /* process all the ports on this node */
- for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
- i <= node->numports; i++) {
- query_port_info(engine, &smp->path, node, i);
- }
+ if (node->type == IB_NODE_SWITCH) {
+ query_switch_info(engine, &smp->path, node);
+ for (i = 0; i <= node->numports; i++)
+ query_port_info(engine, &smp->path, node, i);
+ }
+ } else if (node->type != IB_NODE_SWITCH)
+ query_port_info(engine, &smp->path, node, port_num);
return 0;
}
What do you think?
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [infiniband-diags] [0/3] support --diff and --diffcheck in ibnetdiscover
From: Al Chu @ 2010-04-13 17:17 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100413131033.GH10830@me>
Hey Sasha,
On Tue, 2010-04-13 at 06:10 -0700, Sasha Khapyorsky wrote:
> On 16:08 Tue 13 Apr , Sasha Khapyorsky wrote:
> >
> > Would it be better to keep cache in a regular human readable
> > ibnetdiscover output format, so '--diff' will be usable not just against
> > cache, but also against a regular ibnetdiscover output files?
I had considered this at one point. There were several reasons I
decided to go w/ the cache idea. Perhaps the major reason is that the
current cache system has "all" the data (nodeinfo, portinfo, etc.)
saved, whereas the normal ibnetdiscover output does not. So we would be
limited in our diff output for what ibnetdiscover outputs (or perhaps
limited in future extensions). Adding a --diff into iblinkinfo is also
on my list to do, but that wouldn't be possible w/ a cached text output
(the cached text file doesn't store portinfo data).
This isn't to say there are downsides. The major downside is that it's
difficult to edit the cache when minor changes happen on the network
(e.g. for our system administrators, when an HCA is replaced b/c a node
dies). I have a tool for that (to be submitted soon too I hope :-).
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Ira Weiny @ 2010-04-13 17:07 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413104658.GA10830@me>
On Tue, 13 Apr 2010 13:46:58 +0300
Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> On 12:49 Thu 18 Feb , Ira Weiny wrote:
> >
> > From: Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org>
> > Date: Fri, 22 Jan 2010 17:33:30 -0800
> > Subject: [PATCH] libibnetdisc: Convert to a multi-smp algorithm
> >
> > v3: change DEFAULT_MAX_SMP_ON_WIRE to 2
> >
> > Allow for multiple SMP's to be on the wire at a single time. This
> > algorithm splits the processing of SMP's to a small smp engine which
> > may be useful to split out in the future.
> >
> > Signed-off-by: Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org>
>
> Applied. Thanks.
>
> However see some comments and questions below.
>
> > +static int recv_port_info(smp_engine_t *engine, ibnd_smp_t * smp,
> > + uint8_t *mad, void *cb_data)
> > +{
> > + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> > + ibnd_node_t *node = (ibnd_node_t *)cb_data;
> > + ibnd_port_t *port;
> > + uint8_t *port_info = mad + IB_SMP_DATA_OFFS;
> > + uint8_t port_num, local_port;
> > +
> > + port_num = mad_get_field(mad, 0, IB_MAD_ATTRMOD_F);
> > + local_port = mad_get_field(port_info, 0, IB_PORT_LOCAL_PORT_F);
> > +
> > + /* this may have been created before */
> > + port = node->ports[port_num];
> > + if (!port) {
> > + port = node->ports[port_num] = calloc(1, sizeof(*port));
> > + if (!port) {
> > + IBND_ERROR("Failed to allocate port\n");
> > + return -1;
> > + }
> > + }
> > +
> > + memcpy(port->info, port_info, sizeof(port->info));
> > + port->node = node;
> > + port->portnum = port_num;
> > + port->ext_portnum = 0;
> > + port->base_lid = (uint16_t) mad_get_field(port->info, 0, IB_PORT_LID_F);
> > + port->lmc = (uint8_t) mad_get_field(port->info, 0, IB_PORT_LMC_F);
> > +
> > + if (port_num == 0) {
> > + node->smalid = port->base_lid;
> > + node->smalmc = port->lmc;
> > + } else if (node->type == IB_NODE_SWITCH) {
> > + port->base_lid = node->smalid;
> > + port->lmc = node->smalmc;
> > + }
> > +
> > + add_to_portguid_hash(port, fabric->portstbl);
> > +
> > + debug_port(&smp->path, port);
> >
> > - mad_dump_node_type(type, 64, &node->type, sizeof(int));
> > - printf("%s -> %s %s {%016" PRIx64 "} portnum %d base lid %d-%d\"%s\"\n",
> > - portid2str(path), prompt, type, node->guid,
> > - node->type == IB_NODE_SWITCH ? 0 : port->portnum,
> > - port->base_lid,
> > - port->base_lid + (1 << port->lmc) - 1, node->nodedesc);
> > + if (port_num &&
> > + (mad_get_field(port->info, 0, IB_PORT_PHYS_STATE_F)
> > + == IB_PORT_PHYS_STATE_LINKUP)
> > + &&
> > + (node->type == IB_NODE_SWITCH || node == fabric->from_node)) {
>
> What will happen when ibnetdiscover runs from host connected by two
> ports to different subnets? Wouldn't it run over both fabrics (following
> PortInfo querying loop in recv_node_info() below)?
Looking at the code I think you are right... In practice, this does not happen...
I am still trying to figure out exactly why.
>
> > +
> > + ib_portid_t path = smp->path;
> > + if (extend_dpath(engine, &path, port_num) != -1)
> > + query_node_info(engine, &path, node);
> > + }
> > +
> > + return 0;
> > +}
>
> [snip]
>
> > +static int recv_node_info(smp_engine_t *engine, ibnd_smp_t * smp,
> > + uint8_t *mad, void *cb_data)
> > +{
> > + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> > + int i = 0;
> > + uint8_t *node_info = mad + IB_SMP_DATA_OFFS;
> > + ibnd_node_t * rem_node = (ibnd_node_t *)cb_data;
> > ibnd_node_t *node;
> > + int node_is_new = 0;
> > + uint64_t node_guid = mad_get_field64(node_info, 0, IB_NODE_GUID_F);
> > + uint64_t port_guid = mad_get_field64(node_info, 0, IB_NODE_PORT_GUID_F);
> > + int port_num = mad_get_field(node_info, 0, IB_NODE_LOCAL_PORT_F);
> > + ibnd_port_t *port = NULL;
> >
> > - for (node = fabric->nodestbl[hash]; node; node = node->htnext)
> > - if (node->guid == new->guid)
> > - return node;
> > + node = ibnd_find_node_guid(fabric, node_guid);
> > + if (!node) {
> > + node = create_node(engine, &smp->path, node_info);
> > + if (!node)
> > + return -1;
> > + node_is_new = 1;
> > + }
> > + IBND_DEBUG("Found %s node GUID %lx (%s)\n",
> > + (node_is_new) ? "new": "old", node->guid,
> > + portid2str(&smp->path));
> >
> > - return NULL;
> > + port = node->ports[port_num];
> > + if (!port) {
> > + /* If we have not see this port before create a shell for it */
> > + port = node->ports[port_num] = calloc(1, sizeof(*port));
> > + port->node = node;
> > + port->portnum = port_num;
> > + }
> > + port->guid = port_guid;
> > +
> > + if (rem_node == NULL) /* this is the start node */
> > + fabric->from_node = node;
> > + else {
> > + /* link ports... */
> > + int rem_port_num = get_last_port(&smp->path);
> > +
> > + if (!rem_node->ports[rem_port_num]) {
> > + IBND_ERROR("Internal Error; "
> > + "Node(%p) %lx Port %d no port created!?!?!?\n\n",
> > + rem_node, rem_node->guid, rem_port_num);
> > + return (-1);
> > + }
> > +
> > + link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
> > + }
> > +
> > + if (!node_is_new)
> > + return 0;
> > +
> > + query_node_desc(engine, &smp->path, node);
> > +
> > + if (node->type == IB_NODE_SWITCH)
> > + query_switch_info(engine, &smp->path, node);
> > +
> > + /* process all the ports on this node */
> > + for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
> > + i <= node->numports; i++) {
> > + query_port_info(engine, &smp->path, node, i);
> > + }
>
> This one.
>
> > +
> > + return 0;
> > +}
>
> [snip]
>
> > diff --git a/infiniband-diags/libibnetdisc/src/internal.h b/infiniband-diags/libibnetdisc/src/internal.h
> > index 348bd0f..61b644d 100644
> > --- a/infiniband-diags/libibnetdisc/src/internal.h
> > +++ b/infiniband-diags/libibnetdisc/src/internal.h
> > @@ -39,6 +39,7 @@
> > #define _INTERNAL_H_
> >
> > #include <infiniband/ibnetdisc.h>
> > +#include <complib/cl_qmap.h>
>
> I'm not really happy with complib dependency introduction - we had an
> issues with it in the past.
I don't recall. infiniband-diags is already dependant on opensm-libs. We have, at least, the node name map dependency from complib already, so I did not think it was that big of a deal.
>
> Also I think that we can implement simpler transaction tracker (likely
> even more efficient) by storing sent MADs in cyclic array and encoding
> its index as part of TRID.
I just thought code reuse was the way to go here.
>
> >
> > #define IBND_DEBUG(fmt, ...) \
> > if (ibdebug) { \
> > @@ -52,16 +53,44 @@
> >
> > #define MAXHOPS 63
> >
> > -typedef struct ibnd_node_scan {
> > - ibnd_node_t *node;
> > - struct ibnd_node_scan *dnext; /* nodesdist next */
> > -} ibnd_node_scan_t;
> > +#define DEFAULT_MAX_SMP_ON_WIRE 2
> >
> > typedef struct ibnd_scan {
> > - ibnd_node_scan_t *nodesdist[MAXHOPS + 1];
> > ib_portid_t selfportid;
> > + ibnd_fabric_t *fabric;
> > } ibnd_scan_t;
> >
> > +
> > +typedef struct ibnd_smp ibnd_smp_t;
> > +typedef struct smp_engine smp_engine_t;
> > +typedef int (*smp_comp_cb_t)(smp_engine_t *engine, ibnd_smp_t * smp,
> > + uint8_t *mad_resp, void *cb_data);
> > +struct ibnd_smp {
> > + cl_map_item_t on_wire;
> > + struct ibnd_smp * qnext;
> > + smp_comp_cb_t cb;
> > + void * cb_data;
> > + ib_portid_t path;
> > + ib_rpc_t rpc;
> > +};
> > +struct smp_engine {
> > + struct ibmad_port *ibmad_port;
> > + ibnd_smp_t *smp_queue_head;
> > + ibnd_smp_t *smp_queue_tail;
> > + void * user_data;
> > + cl_qmap_t smps_on_wire;
> > + int num_smps_outstanding;
> > + int max_smps_on_wire;
> > +};
> > +
> > +void smp_engine_init(smp_engine_t * engine, struct ibmad_port *ibmad_port,
> > + void * user_data, int max_smps_on_wire);
> > +int issue_smp(smp_engine_t *engine, ib_portid_t * portid,
> > + unsigned attrid, unsigned mod,
> > + smp_comp_cb_t cb, void * cb_data);
> > +int process_mads(smp_engine_t *engine);
> > +void smp_engine_destroy(smp_engine_t *engine);
> > +
> > void add_to_nodeguid_hash(ibnd_node_t * node, ibnd_node_t * hash[]);
> >
> > void add_to_portguid_hash(ibnd_port_t * port, ibnd_port_t * hash[]);
> > diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
> > new file mode 100644
> > index 0000000..5571314
> > --- /dev/null
> > +++ b/infiniband-diags/libibnetdisc/src/query_smp.c
> > @@ -0,0 +1,249 @@
> > +/*
> > + * Copyright (c) 2010 Lawrence Livermore National Laboratory
> > + *
> > + * This software is available to you under a choice of one of two
> > + * licenses. You may choose to be licensed under the terms of the GNU
> > + * General Public License (GPL) Version 2, available from the file
> > + * COPYING in the main directory of this source tree, or the
> > + * OpenIB.org BSD license below:
> > + *
> > + * Redistribution and use in source and binary forms, with or
> > + * without modification, are permitted provided that the following
> > + * conditions are met:
> > + *
> > + * - Redistributions of source code must retain the above
> > + * copyright notice, this list of conditions and the following
> > + * disclaimer.
> > + *
> > + * - Redistributions in binary form must reproduce the above
> > + * copyright notice, this list of conditions and the following
> > + * disclaimer in the documentation and/or other materials
> > + * provided with the distribution.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> > + * SOFTWARE.
> > + *
> > + */
> > +
> > +#include <errno.h>
> > +#include <infiniband/ibnetdisc.h>
> > +#include <infiniband/umad.h>
> > +#include "internal.h"
> > +
> > +void
> > +queue_smp(smp_engine_t *engine, ibnd_smp_t *smp)
> > +{
> > + smp->qnext = NULL;
> > + if (!engine->smp_queue_head) {
> > + engine->smp_queue_head = smp;
> > + engine->smp_queue_tail = smp;
> > + } else {
> > + engine->smp_queue_tail->qnext = smp;
> > + engine->smp_queue_tail = smp;
> > + }
> > +}
> > +
> > +ibnd_smp_t *
> > +get_smp(smp_engine_t *engine)
> > +{
> > + ibnd_smp_t *head = engine->smp_queue_head;
> > + ibnd_smp_t *tail = engine->smp_queue_tail;
> > + ibnd_smp_t *rc = head;
> > + if (head) {
> > + if (tail == head)
> > + engine->smp_queue_tail = NULL;
> > + engine->smp_queue_head = head->qnext;
> > + }
> > + return rc;
> > +}
> > +
> > +int send_smp(ibnd_smp_t * smp, struct ibmad_port *srcport)
> > +{
> > + int rc = 0;
> > + uint8_t umad[1024];
> > + ib_rpc_t *rpc = &smp->rpc;
> > +
> > + memset(umad, 0, umad_size() + IB_MAD_SIZE);
> > +
> > + if ((rc = mad_build_pkt(umad, &smp->rpc, &smp->path, NULL, NULL))
> > + < 0) {
> > + IBND_ERROR("mad_build_pkt failed; %d", rc);
> > + return rc;
> > + }
> > +
> > + if ((rc = umad_send(mad_rpc_portid(srcport),
> > + mad_rpc_class_agent(srcport, rpc->mgtclass),
> > + umad, IB_MAD_SIZE,
> > + mad_get_timeout(srcport, rpc->timeout),
> > + mad_get_retries(srcport))) < 0) {
> > + IBND_ERROR("send failed; %d", rc);
> > + return rc;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int process_smp_queue(smp_engine_t *engine)
> > +{
> > + int rc = 0;
> > + ibnd_smp_t *smp;
> > + while (cl_qmap_count(&engine->smps_on_wire) < engine->max_smps_on_wire) {
> > + smp = get_smp(engine);
> > + if (!smp)
> > + return 0;
> > +
> > + cl_qmap_insert(&engine->smps_on_wire, (uint32_t)smp->rpc.trid,
> > + (cl_map_item_t *)smp);
> > + if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
> > + return rc;
> > + }
> > + return 0;
> > +}
> > +
> > +int issue_smp(smp_engine_t *engine, ib_portid_t * portid,
> > + unsigned attrid, unsigned mod,
> > + smp_comp_cb_t cb, void * cb_data)
> > +{
> > + ibnd_smp_t *smp = calloc(1, sizeof *smp);
> > + if (!smp) {
> > + IBND_ERROR("OOM");
> > + return -ENOMEM;
> > + }
> > +
> > + smp->cb = cb;
> > + smp->cb_data = cb_data;
> > + smp->path = *portid;
> > + smp->rpc.method = IB_MAD_METHOD_GET;
> > + smp->rpc.attr.id = attrid;
> > + smp->rpc.attr.mod = mod;
> > + smp->rpc.timeout = mad_get_timeout(engine->ibmad_port, 0);
> > + smp->rpc.datasz = IB_SMP_DATA_SIZE;
> > + smp->rpc.dataoffs = IB_SMP_DATA_OFFS;
> > + smp->rpc.trid = mad_trid();
>
> BTW, by reviewing mad_trid() code in libibmad:
>
> uint64_t mad_trid(void)
> {
> static uint64_t base;
> static uint64_t trid;
> uint64_t next;
>
> if (!base) {
> srandom((int)time(0) * getpid());
> base = random();
> trid = random();
> }
> next = ++trid | (base << 32);
> return next;
> }
>
> For me it doesn't look as thread safe function.
It's not! The whole of libibmad is not thread safe! (Ok perhaps just the parts I was using before.) But we are not using threads here, right?
>
> > +
> > + if ((portid->lid <= 0) ||
> > + (portid->drpath.drslid == 0xffff) ||
> > + (portid->drpath.drdlid == 0xffff))
> > + smp->rpc.mgtclass = IB_SMI_DIRECT_CLASS; /* direct SMI */
> > + else
> > + smp->rpc.mgtclass = IB_SMI_CLASS; /* Lid routed SMI */
> > +
> > + portid->sl = 0;
> > + portid->qp = 0;
> > +
> > + engine->num_smps_outstanding++;
> > + queue_smp(engine, smp);
> > + return (process_smp_queue(engine));
> > +}
> > +
> > +int process_one_recv(smp_engine_t *engine)
> > +{
> > + int rc = 0;
> > + int status = 0;
> > + ibnd_smp_t *smp;
> > + uint8_t *mad;
> > + uint32_t trid;
> > + uint8_t umad[umad_size() + IB_MAD_SIZE];
> > + int length = umad_size() + IB_MAD_SIZE;
> > +
> > + memset(umad, 0, sizeof(umad));
> > +
> > + /* wait for the next message */
> > + if ((rc = umad_recv(mad_rpc_portid(engine->ibmad_port), umad, &length,
> > + 0)) < 0) {
> > + if (rc == -EWOULDBLOCK)
> > + return 0;
> > + IBND_ERROR("umad_recv failed: %d\n", rc);
> > + return -1;
> > + }
> > +
> > + rc = process_smp_queue(engine);
> > +
> > + mad = umad_get_mad(umad);
> > + trid = (uint32_t)mad_get_field64(mad, 0, IB_MAD_TRID_F);
> > +
> > + smp = (ibnd_smp_t *)cl_qmap_remove(&engine->smps_on_wire, trid);
> > + if ((cl_map_item_t *)smp == cl_qmap_end(&engine->smps_on_wire)) {
> > + IBND_ERROR("Failed to find matching smp for trid (%x)\n",
> > + trid);
> > + return -1;
> > + }
> > +
> > + if (rc)
> > + goto error;
> > +
> > + if ((status = umad_status(umad))) {
> > + IBND_ERROR("umad (%s Attr 0x%x:%u) bad status %d; %s\n",
> > + portid2str(&smp->path),
> > + smp->rpc.attr.id, smp->rpc.attr.mod,
> > + status, strerror(status));
> > + } else if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F))) {
> > + IBND_ERROR("mad (%s Attr 0x%x:%u) bad status 0x%x\n",
> > + portid2str(&smp->path),
> > + smp->rpc.attr.id, smp->rpc.attr.mod,
> > + status);
> > + } else
> > + rc = smp->cb(engine, smp, mad, smp->cb_data);
> > +
> > +error:
> > + free(smp);
> > + engine->num_smps_outstanding--;
> > + return (rc);
> > +}
> > +
> > +void smp_engine_init(smp_engine_t * engine, struct ibmad_port *ibmad_port,
> > + void * user_data, int max_smps_on_wire)
> > +{
> > + memset(engine, '\0', sizeof(*engine));
> > + engine->ibmad_port = ibmad_port;
> > + engine->user_data = user_data;
> > + cl_qmap_init(&engine->smps_on_wire);
> > + engine->num_smps_outstanding = 0;
> > + engine->max_smps_on_wire = max_smps_on_wire;
> > +}
> > +
> > +void smp_engine_destroy(smp_engine_t *engine)
> > +{
> > + cl_map_item_t *item;
> > + ibnd_smp_t *smp;
> > +
> > + /* remove queued smps */
> > + smp = get_smp(engine);
> > + if (smp)
> > + IBND_ERROR("outstanding SMP's\n");
> > + for (/* */; smp; smp = get_smp(engine)) {
> > + free(smp);
> > + }
> > +
> > + /* remove smps from the wire queue */
> > + item = cl_qmap_head(&engine->smps_on_wire);
> > + if (item != cl_qmap_end(&engine->smps_on_wire))
> > + IBND_ERROR("outstanding SMP's on wire\n");
> > + for (/* */; item != cl_qmap_end(&engine->smps_on_wire);
> > + item = cl_qmap_head(&engine->smps_on_wire)) {
> > + cl_qmap_remove_item(&engine->smps_on_wire, item);
> > + free(item);
> > + }
> > +
> > + engine->num_smps_outstanding = 0;
> > +}
> > +
> > +int process_mads(smp_engine_t *engine)
> > +{
> > + int rc = 0;
> > + while (engine->num_smps_outstanding > 0) {
> > + if ((rc = process_smp_queue(engine)) != 0)
> > + return rc;
>
> Is it really needed to run process_smp_queue() here (assuming that it is
> already executed in issue_smp() and process_one_recv())? Or this is just
> for handling process_one_recv() error case? If so wouldn't it better to
> run it there?
Yes I was looking at that yesterday. Since we are not threaded I just wanted to make sure that there were some messages on the wire before we went into the processing loop. I will send a patch after I have tested everything out.
Ira
>
> Sasha
>
> > + while (!cl_is_qmap_empty(&engine->smps_on_wire))
> > + if ((rc = process_one_recv(engine)) != 0)
> > + return rc;
> > + }
> > + return 0;
> > +}
> > +
> > --
> > 1.5.4.5
> >
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] libibnetdisc: don't try to cross discovery over CA
From: Sasha Khapyorsky @ 2010-04-13 16:58 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413165325.GN10830@me>
When discovery is running from CA node it shouldn't try to cross over
all ports, but only via local one (send over non-local ports will fail
since CA doesn't route MADs).
Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/ibnetdisc.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
index 03b678e..2ce15b7 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
@@ -200,7 +200,7 @@ static int recv_port_info(smp_engine_t * engine, ibnd_smp_t * smp,
if (port_num && mad_get_field(port->info, 0, IB_PORT_PHYS_STATE_F)
== IB_PORT_PHYS_STATE_LINKUP
&& ((node->type == IB_NODE_SWITCH && port_num != local_port) ||
- node == fabric->from_node)) {
+ (node == fabric->from_node && port_num == local_port))) {
ib_portid_t path = smp->path;
if (extend_dpath(engine, &path, port_num) > 0)
query_node_info(engine, &path, node);
--
1.7.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Sasha Khapyorsky @ 2010-04-13 16:53 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413104658.GA10830@me>
On 13:46 Tue 13 Apr , Sasha Khapyorsky wrote:
>
> > +static int recv_port_info(smp_engine_t *engine, ibnd_smp_t * smp,
> > + uint8_t *mad, void *cb_data)
> > +{
> > + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> > + ibnd_node_t *node = (ibnd_node_t *)cb_data;
> > + ibnd_port_t *port;
> > + uint8_t *port_info = mad + IB_SMP_DATA_OFFS;
> > + uint8_t port_num, local_port;
> > +
> > + port_num = mad_get_field(mad, 0, IB_MAD_ATTRMOD_F);
> > + local_port = mad_get_field(port_info, 0, IB_PORT_LOCAL_PORT_F);
> > +
> > + /* this may have been created before */
> > + port = node->ports[port_num];
> > + if (!port) {
> > + port = node->ports[port_num] = calloc(1, sizeof(*port));
> > + if (!port) {
> > + IBND_ERROR("Failed to allocate port\n");
> > + return -1;
> > + }
> > + }
> > +
> > + memcpy(port->info, port_info, sizeof(port->info));
> > + port->node = node;
> > + port->portnum = port_num;
> > + port->ext_portnum = 0;
> > + port->base_lid = (uint16_t) mad_get_field(port->info, 0, IB_PORT_LID_F);
> > + port->lmc = (uint8_t) mad_get_field(port->info, 0, IB_PORT_LMC_F);
> > +
> > + if (port_num == 0) {
> > + node->smalid = port->base_lid;
> > + node->smalmc = port->lmc;
> > + } else if (node->type == IB_NODE_SWITCH) {
> > + port->base_lid = node->smalid;
> > + port->lmc = node->smalmc;
> > + }
> > +
> > + add_to_portguid_hash(port, fabric->portstbl);
> > +
> > + debug_port(&smp->path, port);
> >
> > - mad_dump_node_type(type, 64, &node->type, sizeof(int));
> > - printf("%s -> %s %s {%016" PRIx64 "} portnum %d base lid %d-%d\"%s\"\n",
> > - portid2str(path), prompt, type, node->guid,
> > - node->type == IB_NODE_SWITCH ? 0 : port->portnum,
> > - port->base_lid,
> > - port->base_lid + (1 << port->lmc) - 1, node->nodedesc);
> > + if (port_num &&
> > + (mad_get_field(port->info, 0, IB_PORT_PHYS_STATE_F)
> > + == IB_PORT_PHYS_STATE_LINKUP)
> > + &&
> > + (node->type == IB_NODE_SWITCH || node == fabric->from_node)) {
>
> What will happen when ibnetdiscover runs from host connected by two
> ports to different subnets? Wouldn't it run over both fabrics (following
> PortInfo querying loop in recv_node_info() below)?
I tested with two connected ports. This doesn't work - it tries to cross
another fabric from another port of local node, but failed to send (CA
doesn't route MADs). So this case should be fixed as:
if (port_num && port_state == IB_PORT_PHYS_STATE_LINKUP &&
((node->type == IB_NODE_SWITCH && port_num != local_port) ||
(node == fabric->from_node && port_num == local_port)))
Sasha
>
> > +
> > + ib_portid_t path = smp->path;
> > + if (extend_dpath(engine, &path, port_num) != -1)
> > + query_node_info(engine, &path, node);
> > + }
> > +
> > + return 0;
> > +}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-13 16:39 UTC (permalink / raw)
To: Tung, Chien Tin
Cc: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <2EFBCAEF10980645BBCFB605689E08E9047C7402DA-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
> NE020 supports 4 SGEs. I don't know enough about SDP to know why it is
> using this calculation for # of send_sge:
>
> #define SDP_MAX_RECV_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE)
>
>
Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5 code
and it appears to do nothing in the map_phys_fmr functions.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Sasha Khapyorsky @ 2010-04-13 16:38 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100218124933.c018a23d.weiny2-i2BcT+NCU+M@public.gmane.org>
send_smp() failure blocks whole discovery execution (it never returns).
Fix this by updating all outstanding SMPs related counters only after
successful MAD sending.
Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/query_smp.c | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
index b4322bc..08e3ef7 100644
--- a/infiniband-diags/libibnetdisc/src/query_smp.c
+++ b/infiniband-diags/libibnetdisc/src/query_smp.c
@@ -96,10 +96,12 @@ static int process_smp_queue(smp_engine_t * engine)
if (!smp)
return 0;
- cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
- (cl_map_item_t *) smp);
if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
return rc;
+ engine->num_smps_outstanding++;
+ cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
+ (cl_map_item_t *) smp);
+ engine->total_smps++;
}
return 0;
}
@@ -133,8 +135,6 @@ int issue_smp(smp_engine_t * engine, ib_portid_t * portid,
portid->sl = 0;
portid->qp = 0;
- engine->total_smps++;
- engine->num_smps_outstanding++;
queue_smp(engine, smp);
return process_smp_queue(engine);
}
--
1.7.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-13 16:28 UTC (permalink / raw)
To: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, rolandd
In-Reply-To: <1246722.1271082830120.SLOX.WebMail.wwwrun-XDIR3SKYeFbgKi2NxijLtw@public.gmane.org>
>I add that in kernel space SDP debug the error is:
>
>command line: dmesg
>sdp_init_qp:95 sdp_sock( 2100:2 40720:0): recv sge's. capability: 4
>needed: 9
>sdp_init_qp:95 sdp_sock( 2100:2 41203:0): recv sge's. capability: 4
>needed: 9
>
>The structure sdp_init_qp() is defined in
>/usr/src/ofa_kernel-1.5.1/drivers/infiniband/ulp/sdp/sdp_cma.c (lines 76
>- 141)
NE020 supports 4 SGEs. I don't know enough about SDP to know why it is
using this calculation for # of send_sge:
#define SDP_MAX_RECV_SKB_FRAGS (PAGE_SIZE > 0x8000 ? 1 : 0x8000 / PAGE_SIZE)
>Could be a firmware problem?
This is not a firmware problem.
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: two questions about RDMA-WRITE
From: Sean Hefty @ 2010-04-13 15:48 UTC (permalink / raw)
To: '丁定华',
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <q2v7bb361261004130308za7b4d6ccj6fdb657b7c7cc7e9-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
>(2) It takes a long time to finish rdma_disconnect work (about 10
>seconds), so is it a reasonable time?
This is long. How is your disconnect code structured?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] libibnetdisc: add \n at end of error messages
From: Sasha Khapyorsky @ 2010-04-13 14:56 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100410005245.GH4808@me>
Add '\n' new line character at end of error messages printouts.
Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/chassis.c | 6 +++---
infiniband-diags/libibnetdisc/src/query_smp.c | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/chassis.c b/infiniband-diags/libibnetdisc/src/chassis.c
index 1a839d9..80e034b 100644
--- a/infiniband-diags/libibnetdisc/src/chassis.c
+++ b/infiniband-diags/libibnetdisc/src/chassis.c
@@ -378,7 +378,7 @@ static int get_sfb_slot(ibnd_node_t * n, ibnd_port_t * lineport)
n->ch_slotnum = spine4_slot_2_slb[lineport->portnum];
n->ch_anafanum = anafa_spine4_slot_2_slb[lineport->portnum];
} else {
- IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64,
+ IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64 "\n",
n->guid);
return -1;
}
@@ -419,7 +419,7 @@ static int get_router_slot(ibnd_node_t * n, ibnd_port_t * spineport)
n->ch_slotnum = line_slot_2_sfb4[spineport->portnum];
n->ch_anafanum = ipr_slot_2_sfb4_port[spineport->portnum];
} else {
- IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64,
+ IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64 "\n",
spineport->node->guid);
return -1;
}
@@ -446,7 +446,7 @@ static int get_slb_slot(ibnd_node_t * n, ibnd_port_t * spineport)
n->ch_slotnum = line_slot_2_sfb4[spineport->portnum];
n->ch_anafanum = anafa_line_slot_2_sfb4[spineport->portnum];
} else {
- IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64,
+ IBND_ERROR("Unexpected node found: guid 0x%016" PRIx64 "\n",
spineport->node->guid);
return -1;
}
diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
index 1ec9752..b4322bc 100644
--- a/infiniband-diags/libibnetdisc/src/query_smp.c
+++ b/infiniband-diags/libibnetdisc/src/query_smp.c
@@ -71,7 +71,7 @@ static int send_smp(ibnd_smp_t * smp, struct ibmad_port *srcport)
if ((rc = mad_build_pkt(umad, &smp->rpc, &smp->path, NULL, NULL))
< 0) {
- IBND_ERROR("mad_build_pkt failed; %d", rc);
+ IBND_ERROR("mad_build_pkt failed; %d\n", rc);
return rc;
}
@@ -80,7 +80,7 @@ static int send_smp(ibnd_smp_t * smp, struct ibmad_port *srcport)
umad, IB_MAD_SIZE,
mad_get_timeout(srcport, rpc->timeout),
mad_get_retries(srcport))) < 0) {
- IBND_ERROR("send failed; %d", rc);
+ IBND_ERROR("send failed; %d\n", rc);
return rc;
}
@@ -109,7 +109,7 @@ int issue_smp(smp_engine_t * engine, ib_portid_t * portid,
{
ibnd_smp_t *smp = calloc(1, sizeof *smp);
if (!smp) {
- IBND_ERROR("OOM");
+ IBND_ERROR("OOM\n");
return -ENOMEM;
}
--
1.7.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: two questions about RDMA-WRITE
From: Steve Wise @ 2010-04-13 14:50 UTC (permalink / raw)
To: 丁定华; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <q2v7bb361261004130308za7b4d6ccj6fdb657b7c7cc7e9-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
丁定华 wrote:
> Hi guys:
> I'm doing a project using RDMA to transfer data between servers. I'm
> using RDMA_PS_TCP type and do RDMA_WRITE operations from client node
> to server node. While the program succeed in setting up connection and
> transferring data, I 'm still very confused about two problems:
> (1) After transfer the data using IB_WR_RDMA_WRITE , the complete
> queue entry found that the opcode turned from IB_WR_RDMA_WRITE to
> IB_WR_RDMA_WRITE_WITH_IMM, I don't know the meaning of this opcode and
> the reason of the change.
>
The opcode in the completion is of type enum ib_wc_opcode, not enum
ib_wr_opcode. You are mis-interpreting the opcode in the completion.
> (2) It takes a long time to finish rdma_disconnect work (about 10
> seconds), so is it a reasonable time?
>
What device are you using?
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] Dimension port order file support
From: Sasha Khapyorsky @ 2010-04-13 14:02 UTC (permalink / raw)
To: Dale Purdy; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100324133221.GK4808@me>
On 15:32 Wed 24 Mar , Sasha Khapyorsky wrote:
> Hi Dale,
>
> On 18:06 Wed 03 Mar , Dale Purdy wrote:
> >
> > Provide a means to specify on a per switch basis the mapping (order)
> > between switch ports and dimensions for Dimension Order Routing. This
> > allows the DOR routing engine to be used when the cabling is not
> > properly aligned for DOR, either initially, or for an upgrade.
>
> Nice stuff.
>
> Is this something useful with ! '-R dor'?
>
> >
> > Signed-off-by: Dale Purdy <purdy-sJ/iWh9BUns@public.gmane.org>
Applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Sasha Khapyorsky @ 2010-04-13 13:25 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413104658.GA10830@me>
On 13:46 Tue 13 Apr , Sasha Khapyorsky wrote:
>
> However see some comments and questions below.
Another thought. What about API like:
ibnd_discover_fabric(cosnt char *ca_name, unsigned port_num,
struct ibnd_config *cfg);
So libibnetdisc will be responsible for opening and closing its own port
and in this way will be fully reenterable?
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [infiniband-diags] [0/3] support --diff and --diffcheck in ibnetdiscover
From: Sasha Khapyorsky @ 2010-04-13 13:10 UTC (permalink / raw)
To: Al Chu; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100413130803.GG10830@me>
On 16:08 Tue 13 Apr , Sasha Khapyorsky wrote:
>
> Would it be better to keep cache in a regular human readable
> ibnetdiscover output format, so '--diff' will be usable not just against
> cache, but also against a regular ibnetdiscover output files?
BTW, such implementation could be simplified by the fact that we have
already ibnetdiscover output parser in 'ibsim'.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox