* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Sasha Khapyorsky @ 2010-04-16 12:05 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <0EEE4F40-F1DD-46A6-B756-3C46DA06B403-i2BcT+NCU+M@public.gmane.org>
On 09:52 Wed 14 Apr , Ira Weiny wrote:
>
> > But then it blocks process_mads() to loop forever after single
> > send_smp() failure (with all empty queues and umad_recv() running
> > without timeout).
>
> But moving the cl_qmap_insert below the send call fixes that.
It doesn't:
int process_mads(smp_engine_t * engine)
{
int rc = 0;
while (engine->num_smps_outstanding > 0) {
if ((rc = process_smp_queue(engine)) != 0)
return rc;
while (!cl_is_qmap_empty(&engine->smps_on_wire))
if ((rc = process_one_recv(engine)) != 0)
return rc;
}
return 0;
}
After send_smp() failure engine->num_smps_outstanding still be > 0 and
will be never decreased (tested).
> However, it does cause a memory leak because the smp is no longer in
> the smp_queue_head list.
This is correct about leaking.
> It needs to be put back on that list to be
> retried with a limit on the retries (to prevent what you are saying
> here.)
We have already retries mechanism implemented in umad_send(), so likely
failed MAD should be just dropped and freed:
diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
index 08e3ef7..89c0b05 100644
--- a/infiniband-diags/libibnetdisc/src/query_smp.c
+++ b/infiniband-diags/libibnetdisc/src/query_smp.c
@@ -96,8 +96,10 @@ static int process_smp_queue(smp_engine_t * engine)
if (!smp)
return 0;
- if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
+ if ((rc = send_smp(smp, engine->ibmad_port)) != 0) {
+ free(smp);
return rc;
+ }
engine->num_smps_outstanding++;
cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
(cl_map_item_t *) smp);
> Are you seeing a hang?
I'm seeing endless loop.
> I have seen a hang when running "iblinkinfo -S <guid>".
What do you mean "hang"? Endless loop?
> However, the
> problem is not with send_smp. I am seeing the mad going on the wire
> and returning (according to madeye) but I am not receiving it from
> umad_recv. I don't know why. If I run with 1 outstanding mad it
> works???
Do you see this with current master (for me 'iblinkinfo -S' works fine,
but I have only two switches).
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH] rdma/cm: Randomize local port allocation.
From: Cong Wang @ 2010-04-16 2:22 UTC (permalink / raw)
To: Sean Hefty
Cc: 'Tetsuo Handa', opurdila, eric.dumazet, netdev, nhorman,
davem, ebiederm, linux-kernel, rolandd, linux-rdma
In-Reply-To: <5CE6BC7B954643FAAFFD2AD66F673CBD@amr.corp.intel.com>
Sean Hefty wrote:
> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
>
>> Randomize local port allocation in a way sctp_get_port_local() does.
>> Update rover at the end of loop since we're likely to pick a valid port
>> on the first try.
>>
>> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Reviewed-by: Sean Hefty <sean.hefty@intel.com>
>
Thanks, everyone!
>
> I like this version, thanks! I'm not sure which tree to merge it through.
> Are you needing this for 2.6.34, or is 2.6.35 okay?
>
As soon as possible, so 2.6.34. :)
^ permalink raw reply
* [infiniband-diags] [2/2] add ibcacheedit tool
From: Al Chu @ 2010-04-15 23:52 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 505 bytes --]
Hey Sasha,
This patch adds a new ibcacheedit tool to infiniband-diags. As the name
suggests, it offers users options to edit a stored cache. This tool is
primarily necessary to allow system administrators to update caches as
they make minor modifications to a cluster (e.g. node dies, thus HCA is
replaced) rather than regularly regenerating a new cache.
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0002-add-ibcacheedit-tool.patch --]
[-- Type: message/rfc822, Size: 13093 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] add ibcacheedit tool
Date: Thu, 15 Apr 2010 09:38:07 -0700
Message-ID: <1271375429.17987.174.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
infiniband-diags/Makefile.am | 6 +-
infiniband-diags/man/ibcacheedit.8 | 57 ++++++
infiniband-diags/src/ibcacheedit.c | 361 ++++++++++++++++++++++++++++++++++++
3 files changed, 422 insertions(+), 2 deletions(-)
create mode 100644 infiniband-diags/man/ibcacheedit.8
create mode 100644 infiniband-diags/src/ibcacheedit.c
diff --git a/infiniband-diags/Makefile.am b/infiniband-diags/Makefile.am
index 1cdb60e..af90b05 100644
--- a/infiniband-diags/Makefile.am
+++ b/infiniband-diags/Makefile.am
@@ -13,7 +13,7 @@ sbin_PROGRAMS = src/ibaddr src/ibnetdiscover src/ibping src/ibportstate \
src/ibroute src/ibstat src/ibsysstat src/ibtracert \
src/perfquery src/sminfo src/smpdump src/smpquery \
src/saquery src/vendstat src/iblinkinfo \
- src/ibqueryerrors
+ src/ibqueryerrors src/ibcacheedit
if ENABLE_TEST_UTILS
sbin_PROGRAMS += src/ibsendtrap src/mcm_rereg_test
@@ -62,6 +62,8 @@ src_iblinkinfo_SOURCES = src/iblinkinfo.c
src_iblinkinfo_LDFLAGS = -L$(top_builddir)/libibnetdisc -libnetdisc
src_ibqueryerrors_SOURCES = src/ibqueryerrors.c
src_ibqueryerrors_LDFLAGS = -L$(top_builddir)/libibnetdisc -libnetdisc
+src_ibcacheedit_SOURCES = src/ibcacheedit.c
+src_ibcacheedit_LDFLAGS = -L$(top_builddir)/libibnetdisc -libnetdisc
man_MANS = man/ibaddr.8 man/ibcheckerrors.8 man/ibcheckerrs.8 \
man/ibchecknet.8 man/ibchecknode.8 man/ibcheckport.8 \
@@ -76,7 +78,7 @@ man_MANS = man/ibaddr.8 man/ibcheckerrors.8 man/ibcheckerrs.8 \
man/ibprintswitch.8 man/ibprintca.8 man/ibfindnodesusing.8 \
man/ibdatacounts.8 man/ibdatacounters.8 \
man/ibrouters.8 man/ibprintrt.8 man/ibidsverify.8 \
- man/check_lft_balance.8
+ man/check_lft_balance.8 man/ibcacheedit.8
BUILT_SOURCES = ibdiag_version
ibdiag_version:
diff --git a/infiniband-diags/man/ibcacheedit.8 b/infiniband-diags/man/ibcacheedit.8
new file mode 100644
index 0000000..b977827
--- /dev/null
+++ b/infiniband-diags/man/ibcacheedit.8
@@ -0,0 +1,57 @@
+.TH IBCACHEEDIT 8 "Apr 15, 2010" "OpenIB" "OpenIB Diagnostics"
+
+.SH NAME
+ibcacheedit \- edit an ibnetdiscover cache
+
+.SH SYNOPSIS
+.B ibcacheedit
+[\-\-switchguid BEFOREGUID:AFTERGUID] [\-\-caguid BEFORE:AFTER]
+[\-\-sysimgguid BEFOREGUID:AFTERGUID] [\-\-portguid NODEGUID:BEFOREGUID:AFTERGUID]
+[\-h(elp)] <orig.cache> <new.cache>
+
+.SH DESCRIPTION
+.PP
+ibcacheedit allows users to edit an ibnetdiscover cache created through the
+\fB\-\-cache\fR option in
+.B ibnetdiscover(8).
+
+.SH OPTIONS
+
+.PP
+.TP
+\fB\-\-switchguid\fR BEFOREGUID:AFTERGUID
+Specify a switchguid that should be changed. The before and after guid
+should be separated by a colon. On switches, port guids are identical
+to the switch guid, so port guids will be adjusted as well on switches.
+.TP
+\fB\-\-caguid\fR BEFOREGUID:AFTERGUID
+Specify a caguid that should be changed. The before and after guid
+should be separated by a colon.
+.TP
+\fB\-\-sysimgguid\fR BEFOREGUID:AFTERGUID
+Specify a sysimgguid that should be changed. The before and after guid
+should be spearated by a colon.
+.TP
+\fB\-\-portguid\fR NODEGUID:BEFOREGUID:AFTERGUID
+Specify a portguid that should be changed. The nodeguid of the port
+(e.g. switchguid or caguid) should be specified first, followed by a
+colon, the before port guid, another colon, then the after port guid.
+On switches, port guids are identical to the switch guid, so the
+switch guid will be adjusted as well on switches.
+
+.SH COMMON OPTIONS
+
+Most OpenIB diagnostics take the following common flags. The exact list of
+supported flags per utility can be found in the usage message and can be shown
+using the util_name -h syntax.
+
+# Debugging flags
+.PP
+\-h show the usage message
+.PP
+\-V show the version info.
+
+.SH AUTHORS
+.TP
+Albert Chu
+.RI < chu11-i2BcT+NCU+M@public.gmane.org >
diff --git a/infiniband-diags/src/ibcacheedit.c b/infiniband-diags/src/ibcacheedit.c
new file mode 100644
index 0000000..1f2558b
--- /dev/null
+++ b/infiniband-diags/src/ibcacheedit.c
@@ -0,0 +1,361 @@
+/*
+ * Copyright (c) 2010 Lawrence Livermore National Lab. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ */
+
+#if HAVE_CONFIG_H
+# include <config.h>
+#endif /* HAVE_CONFIG_H */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <getopt.h>
+#include <inttypes.h>
+
+#include <infiniband/mad.h>
+#include <infiniband/ibnetdisc.h>
+
+#include "ibdiag_common.h"
+
+uint64_t switchguid_before = 0;
+uint64_t switchguid_after = 0;
+int switchguid_flag = 0;
+
+uint64_t caguid_before = 0;
+uint64_t caguid_after = 0;
+int caguid_flag = 0;
+
+uint64_t sysimgguid_before = 0;
+uint64_t sysimgguid_after = 0;
+int sysimgguid_flag = 0;
+
+uint64_t portguid_nodeguid = 0;
+uint64_t portguid_before = 0;
+uint64_t portguid_after = 0;
+int portguid_flag = 0;
+
+struct guids {
+ uint64_t searchguid;
+ int searchguid_found;
+ uint64_t before;
+ uint64_t after;
+ int found;
+};
+
+static int parse_beforeafter(char *arg, uint64_t *before, uint64_t *after)
+{
+ char *ptr;
+ char *before_str;
+ char *after_str;
+
+ ptr = strchr(optarg, ':');
+ if (!ptr || !(*(ptr + 1))) {
+ fprintf(stderr, "invalid input '%s'\n", arg);
+ return -1;
+ }
+ (*ptr) = '\0';
+ before_str = arg;
+ after_str = ptr + 1;
+
+ (*before) = strtoull(before_str, 0, 0);
+ (*after) = strtoull(after_str, 0, 0);
+ return 0;
+}
+
+static int parse_guidbeforeafter(char *arg,
+ uint64_t *guid,
+ uint64_t *before,
+ uint64_t *after)
+{
+ char *ptr1;
+ char *ptr2;
+ char *guid_str;
+ char *before_str;
+ char *after_str;
+
+ ptr1 = strchr(optarg, ':');
+ if (!ptr1 || !(*(ptr1 + 1))) {
+ fprintf(stderr, "invalid input '%s'\n", arg);
+ return -1;
+ }
+ guid_str = arg;
+ before_str = ptr1 + 1;
+
+ ptr2 = strchr(before_str, ':');
+ if (!ptr2 || !(*(ptr2 + 1))) {
+ fprintf(stderr, "invalid input '%s'\n", arg);
+ return -1;
+ }
+ (*ptr1) = '\0';
+ (*ptr2) = '\0';
+ after_str = ptr2 + 1;
+
+ (*guid) = strtoull(guid_str, 0, 0);
+ (*before) = strtoull(before_str, 0, 0);
+ (*after) = strtoull(after_str, 0, 0);
+ return 0;
+}
+
+static int process_opt(void *context, int ch, char *optarg)
+{
+ switch (ch) {
+ case 1:
+ if (parse_beforeafter(optarg,
+ &switchguid_before,
+ &switchguid_after) < 0)
+ return -1;
+ switchguid_flag++;
+ break;
+ case 2:
+ if (parse_beforeafter(optarg,
+ &caguid_before,
+ &caguid_after) < 0)
+ return -1;
+ caguid_flag++;
+ break;
+ case 3:
+ if (parse_beforeafter(optarg,
+ &sysimgguid_before,
+ &sysimgguid_after) < 0)
+ return -1;
+ sysimgguid_flag++;
+ break;
+ case 4:
+ if (parse_guidbeforeafter(optarg,
+ &portguid_nodeguid,
+ &portguid_before,
+ &portguid_after) < 0)
+ return -1;
+ portguid_flag++;
+ break;
+ default:
+ return -1;
+ }
+
+ return 0;
+}
+
+static void update_switchportguids(ibnd_node_t *node, uint64_t guid)
+{
+ ibnd_port_t *port;
+ int p;
+
+ for (p = 0; p <= node->numports; p++) {
+ port = node->ports[p];
+ if (port)
+ port->guid = node->guid;
+ }
+}
+
+static void replace_node_guid(ibnd_node_t *node, void *user_data)
+{
+ struct guids *guids;
+
+ guids = (struct guids *)user_data;
+
+ if (node->guid == guids->before) {
+
+ node->guid = guids->after;
+
+ /* port guids are identical to switch guids on
+ * switches, so update port guids too
+ */
+ if (node->type == IB_NODE_SWITCH)
+ update_switchportguids(node, guids->after);
+
+ guids->found++;
+ }
+}
+
+static void replace_sysimgguid(ibnd_node_t *node, void *user_data)
+{
+ struct guids *guids;
+ uint64_t sysimgguid;
+
+ guids = (struct guids *)user_data;
+
+ sysimgguid = mad_get_field64(node->info, 0, IB_NODE_SYSTEM_GUID_F);
+ if (sysimgguid == guids->before) {
+ mad_set_field64(node->info, 0, IB_NODE_SYSTEM_GUID_F,
+ guids->after);
+ guids->found++;
+ }
+}
+
+static void replace_portguid(ibnd_node_t *node, void *user_data)
+{
+ struct guids *guids;
+
+ guids = (struct guids *)user_data;
+
+ if (node->guid != guids->searchguid)
+ return;
+
+ guids->searchguid_found++;
+
+ if (node->type == IB_NODE_SWITCH) {
+ /* port guids are identical to switch guids on
+ * switches, so update switch guid too
+ */
+ if (node->guid == guids->before) {
+ node->guid = guids->after;
+ update_switchportguids(node, guids->after);
+ guids->found++;
+ }
+ }
+ else {
+ ibnd_port_t *port;
+ int p;
+
+ for (p = 1; p <= node->numports; p++) {
+ port = node->ports[p];
+ if (port
+ && port->guid == guids->before) {
+ port->guid = guids->after;
+ guids->found++;
+ break;
+ }
+ }
+ }
+}
+
+int main(int argc, char **argv)
+{
+ ibnd_fabric_t *fabric = NULL;
+ char *orig_cache_file = NULL;
+ char *new_cache_file = NULL;
+ struct guids guids;
+
+ const struct ibdiag_opt opts[] = {
+ {"switchguid", 1, 1, "BEFOREGUID:AFTERGUID",
+ "Specify before and after switchguid to edit"},
+ {"caguid", 2, 1, "BEFOREGUID:AFTERGUID",
+ "Specify before and after caguid to edit"},
+ {"sysimgguid", 3, 1, "BEFOREGUID:AFTERGUID",
+ "Specify before and after sysimgguid to edit"},
+ {"portguid", 4, 1, "NODEGUID:BEFOREGUID:AFTERGUID",
+ "Specify before and after port guid to edit"},
+ {0}
+ };
+ char *usage_args = "<orig.cache> <new.cache>";
+
+ ibdiag_process_opts(argc, argv, NULL, "edCPDLGtsv",
+ opts, process_opt, usage_args,
+ NULL);
+
+ argc -= optind;
+ argv += optind;
+
+ orig_cache_file = argv[0];
+ new_cache_file = argv[1];
+
+ if (!orig_cache_file)
+ IBERROR("original cache file not specified");
+
+ if (!new_cache_file)
+ IBERROR("new cache file not specified");
+
+ if ((fabric = ibnd_load_fabric(orig_cache_file, 0)) == NULL)
+ IBERROR("loading original cached fabric failed");
+
+ if (switchguid_flag) {
+ guids.before = switchguid_before;
+ guids.after = switchguid_after;
+ guids.found = 0;
+ ibnd_iter_nodes_type(fabric,
+ replace_node_guid,
+ IB_NODE_SWITCH,
+ &guids);
+
+ if (!guids.found)
+ IBERROR("switchguid = %" PRIx64 " not found",
+ switchguid_before);
+ }
+
+ if (caguid_flag) {
+ guids.before = caguid_before;
+ guids.after = caguid_after;
+ guids.found = 0;
+ ibnd_iter_nodes_type(fabric,
+ replace_node_guid,
+ IB_NODE_CA,
+ &guids);
+
+ if (!guids.found)
+ IBERROR("caguid = %" PRIx64 " not found",
+ caguid_before);
+ }
+
+ if (sysimgguid_flag) {
+ guids.before = sysimgguid_before;
+ guids.after = sysimgguid_after;
+ guids.found = 0;
+ ibnd_iter_nodes(fabric,
+ replace_sysimgguid,
+ &guids);
+
+ if (!guids.found)
+ IBERROR("sysimgguid = %" PRIx64 " not found",
+ sysimgguid_before);
+ }
+
+ if (portguid_flag) {
+ guids.searchguid = portguid_nodeguid;
+ guids.searchguid_found = 0;
+ guids.before = portguid_before;
+ guids.after = portguid_after;
+ guids.found = 0;
+ ibnd_iter_nodes(fabric,
+ replace_portguid,
+ &guids);
+
+ if (!guids.searchguid_found)
+ IBERROR("nodeguid = %" PRIx64 " not found",
+ portguid_nodeguid);
+
+ if (!guids.found)
+ IBERROR("portguid = %" PRIx64 " not found",
+ portguid_before);
+ }
+
+ /* set IBND_CACHE_FABRIC_FLAG_OVERWRITE in case user wants
+ * to use same cache filename
+ */
+ if (ibnd_cache_fabric(fabric,
+ new_cache_file,
+ IBND_CACHE_FABRIC_FLAG_OVERWRITE) < 0)
+ IBERROR("caching new cache data failed");
+
+ ibnd_destroy_fabric(fabric);
+ exit(0);
+}
--
1.5.4.5
^ permalink raw reply related
* [infiniband-diags] [1/2] support libibnetdisc caching overwrite flag
From: Al Chu @ 2010-04-15 23:52 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 249 bytes --]
Hi Sasha,
This patch supports a flag in the libibnetdisc caching code to overwrite
a previous cache.
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0001-support-libibnetdisc-caching-overwrite-flag.patch --]
[-- Type: message/rfc822, Size: 2109 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] support libibnetdisc caching overwrite flag
Date: Thu, 15 Apr 2010 09:37:44 -0700
Message-ID: <1271375429.17987.176.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
.../libibnetdisc/include/infiniband/ibnetdisc.h | 3 +++
.../libibnetdisc/src/ibnetdisc_cache.c | 17 ++++++++++++++---
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/include/infiniband/ibnetdisc.h b/infiniband-diags/libibnetdisc/include/infiniband/ibnetdisc.h
index 136282c..c83bd0b 100644
--- a/infiniband-diags/libibnetdisc/include/infiniband/ibnetdisc.h
+++ b/infiniband-diags/libibnetdisc/include/infiniband/ibnetdisc.h
@@ -181,6 +181,9 @@ MAD_EXPORT void ibnd_destroy_fabric(ibnd_fabric_t * fabric);
MAD_EXPORT ibnd_fabric_t *ibnd_load_fabric(const char *file,
unsigned int flags);
+#define IBND_CACHE_FABRIC_FLAG_DEFAULT 0x0000
+#define IBND_CACHE_FABRIC_FLAG_OVERWRITE 0x0001
+
MAD_EXPORT int ibnd_cache_fabric(ibnd_fabric_t * fabric, const char *file,
unsigned int flags);
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
index 480a0a2..6cf7d4d 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
@@ -876,9 +876,20 @@ int ibnd_cache_fabric(ibnd_fabric_t * fabric, const char *file,
return -1;
}
- if (!stat(file, &statbuf)) {
- IBND_DEBUG("file '%s' already exists\n", file);
- return -1;
+ if (flags & IBND_CACHE_FABRIC_FLAG_OVERWRITE) {
+ if (!stat(file, &statbuf)) {
+ if (unlink(file) < 0) {
+ IBND_DEBUG("error removing '%s': %s\n",
+ file, strerror(errno));
+ return -1;
+ }
+ }
+ }
+ else {
+ if (!stat(file, &statbuf)) {
+ IBND_DEBUG("file '%s' already exists\n", file);
+ return -1;
+ }
}
if ((fd = open(file, O_CREAT | O_EXCL | O_WRONLY, 0644)) < 0) {
--
1.5.4.5
^ permalink raw reply related
* [PATCH] rdma/cm: Randomize local port allocation.
From: Sean Hefty @ 2010-04-15 19:55 UTC (permalink / raw)
To: 'Tetsuo Handa'
Cc: amwang-H+wXaHxf7aLQT0dZR+AlfA, opurdila-+zzKsuq53OdBDgjK7y7TUQ,
eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w,
netdev-u79uwXL29TY76Z2rM5mHXA, nhorman-2XuSBdqkA4R54TAoqtyWWQ,
davem-fT/PcQaiUtIeIZ0/mPfg9Q, ebiederm-aS9lmoZGLiVWk0Htik3J/w,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w, linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <201004150229.o3F2T4dZ054768-etx+eQDEXHD7nzcFbJAaVXf5DAMn2ifp@public.gmane.org>
From: Tetsuo Handa <penguin-kernel-JPay3/Yim36HaxMnTkn67Xf5DAMn2ifp@public.gmane.org>
>Randomize local port allocation in a way sctp_get_port_local() does.
>Update rover at the end of loop since we're likely to pick a valid port
>on the first try.
>
>Signed-off-by: Tetsuo Handa <penguin-kernel-JPay3/Yim36HaxMnTkn67Xf5DAMn2ifp@public.gmane.org>
Reviewed-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>---
I like this version, thanks! I'm not sure which tree to merge it through.
Are you needing this for 2.6.34, or is 2.6.35 okay?
> drivers/infiniband/core/cma.c | 70 +++++++++++++++--------------------------
>-
> 1 file changed, 25 insertions(+), 45 deletions(-)
>
>--- linux-2.6.34-rc4.orig/drivers/infiniband/core/cma.c
>+++ linux-2.6.34-rc4/drivers/infiniband/core/cma.c
>@@ -79,7 +79,6 @@ static DEFINE_IDR(sdp_ps);
> static DEFINE_IDR(tcp_ps);
> static DEFINE_IDR(udp_ps);
> static DEFINE_IDR(ipoib_ps);
>-static int next_port;
>
> struct cma_device {
> struct list_head list;
>@@ -1970,47 +1969,33 @@ err1:
>
> static int cma_alloc_any_port(struct idr *ps, struct rdma_id_private *id_priv)
> {
>- struct rdma_bind_list *bind_list;
>- int port, ret, low, high;
>-
>- bind_list = kzalloc(sizeof *bind_list, GFP_KERNEL);
>- if (!bind_list)
>- return -ENOMEM;
>-
>-retry:
>- /* FIXME: add proper port randomization per like inet_csk_get_port */
>- do {
>- ret = idr_get_new_above(ps, bind_list, next_port, &port);
>- } while ((ret == -EAGAIN) && idr_pre_get(ps, GFP_KERNEL));
>-
>- if (ret)
>- goto err1;
>+ static unsigned int last_used_port;
>+ int low, high, remaining;
>+ unsigned int rover;
>
> inet_get_local_port_range(&low, &high);
>- if (port > high) {
>- if (next_port != low) {
>- idr_remove(ps, port);
>- next_port = low;
>- goto retry;
>- }
>- ret = -EADDRNOTAVAIL;
>- goto err2;
>+ remaining = (high - low) + 1;
>+ rover = net_random() % remaining + low;
>+retry:
>+ if (last_used_port != rover &&
>+ !idr_find(ps, (unsigned short) rover)) {
>+ int ret = cma_alloc_port(ps, id_priv, rover);
>+ /*
>+ * Remember previously used port number in order to avoid
>+ * re-using same port immediately after it is closed.
>+ */
>+ if (!ret)
>+ last_used_port = rover;
>+ if (ret != -EADDRNOTAVAIL)
>+ return ret;
> }
>-
>- if (port == high)
>- next_port = low;
>- else
>- next_port = port + 1;
>-
>- bind_list->ps = ps;
>- bind_list->port = (unsigned short) port;
>- cma_bind_port(bind_list, id_priv);
>- return 0;
>-err2:
>- idr_remove(ps, port);
>-err1:
>- kfree(bind_list);
>- return ret;
>+ if (--remaining) {
>+ rover++;
>+ if ((rover < low) || (rover > high))
>+ rover = low;
>+ goto retry;
>+ }
>+ return -EADDRNOTAVAIL;
> }
>
> static int cma_use_port(struct idr *ps, struct rdma_id_private *id_priv)
>@@ -2995,12 +2980,7 @@ static void cma_remove_one(struct ib_dev
>
> static int __init cma_init(void)
> {
>- int ret, low, high, remaining;
>-
>- get_random_bytes(&next_port, sizeof next_port);
>- inet_get_local_port_range(&low, &high);
>- remaining = (high - low) + 1;
>- next_port = ((unsigned int) next_port % remaining) + low;
>+ int ret;
>
> cma_wq = create_singlethread_workqueue("rdma_cm");
> if (!cma_wq)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: two questions about RDMA-WRITE
From: Sean Hefty @ 2010-04-15 17:17 UTC (permalink / raw)
To: 'Ding Dinghua'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100414034142.GA5567-6UwBTAaVBkEeWBJJgrHO4P1unptsbck7@public.gmane.org>
>static void jm_cq_comp_handler(struct ib_cq *cq, void *context) {
> struct jm_rdma_conn *conn = context;
> struct ib_wc wc;
> struct jm_send_ctx *send;
>
> /* No idea why it should be called twice. */
> printk("cq comp for id %p\n", conn->jc_id);
> ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
> while (ib_poll_cq(cq, 1, &wc) == 1) {
> if (wc.opcode != IB_WC_RDMA_WRITE) {
> printk("completed unknown opcode %d\n", wc.opcode);
> /* continue; */
> }
> send = (struct jm_send_ctx *)wc.wr_id;
> printk("got send=%p\n", send);
> printk("completed RDMA_WRITE of IO(%Lu, %u)\n",
> send->s_offset, send->s_size);
> send->s_done = wc.status == IB_WC_SUCCESS ? 1 : -EIO;
> wake_up_all(&send->s_wait);
> }
> ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
unrelated to your problem, but this second call to ib_req_notify_cq isn't
necessary.
>static int jm_rdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event
>*event) {
..
> case RDMA_CM_EVENT_DISCONNECTED:
> connstate = -ECONNABORTED;
> goto connected;
..
>connected:
> printk("%pI4:%u (event 0x%x)\n",
> &conn->jc_remoteaddr.sin_addr.s_addr,
> ntohs(conn->jc_remoteaddr.sin_port),
> event->event << 11);
> conn->jc_connstate = connstate;
> wake_up_all(&conn->jc_connect_wait);
> break;
How quickly do you respond to the disconnect event? The remote side will wait
until it receives a response or times out, which may be several seconds or
minutes.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Peter Zijlstra @ 2010-04-15 13:48 UTC (permalink / raw)
To: Gleb Natapov
Cc: Roland Dreier, Håkon Bugge, Jason Gunthorpe, Andrew Morton,
Eric B Munson, linux-kernel, linux-rdma, rolandd, pavel, mingo,
jsquyres
In-Reply-To: <20100414085228.GL23554@redhat.com>
On Wed, 2010-04-14 at 11:52 +0300, Gleb Natapov wrote:
> On Tue, Apr 13, 2010 at 08:02:54PM +0200, Peter Zijlstra wrote:
> > On Tue, 2010-04-13 at 10:57 -0700, Roland Dreier wrote:
> > > Are those system calls the only possible way that virtual to physical
> > > mappings can change? Can't page migration or something like that
> > > potentially affect things? And even if you did have hooks into every
> > > system call that mattered (keep in mind that relying on glibc is not
> > > enough, since an MPI application may not use glibc) would decoding them
> > > and figuring out what happened really be preferable to a single event
> > > type that tells you exactly what address range was affected?
> >
> > Yeah, virtual<->physical maps can change through swapping, page
> > migration, memory compaction, huge-page aggregation (the latter two not
> > yet being upstream).
> >
> > Even mlock() doesn't pin virtual<->physical maps.
> Pages registered for RDMA are GUPed so no method above should touch
> them. Fork+cow or unmap/map on the other hand can change
> virtual<->physical maps. GUPed pages are still GUPed, but they are no
> longer mapped into process' virtual address space. MPI copes with
> Fork+cow by marking registered memory as MADV_DONTFORK.
Sure, holding a page-ref will pin the physical page, but that is not
something userspace usually has means to.
^ permalink raw reply
* Re: [PATCH 1/4] opensm: added function that dumps PathRecords
From: Yevgeny Kliteynik @ 2010-04-15 11:54 UTC (permalink / raw)
To: Jim Schutt; +Cc: Sasha Khapyorsky, Linux RDMA
In-Reply-To: <1271283366.4092.145.camel-mgfCWIlwujvg4c9jKm7R2O1ftBKYq+Ku@public.gmane.org>
Hi Jim,
On 15/Apr/10 01:16, Jim Schutt wrote:
> Hi Yevgeny,
>
> On Thu, 2010-04-08 at 07:29 -0600, Yevgeny Kliteynik wrote:
>> Dumping SL, MTU and Rate for all the
>> non-switch-2-non-switch paths in the subnet.
>>
>> PRs that are dumped:
>>
>> for every non-switch source port
>> for every non-switch target LID in the subnet
>> dump PR between source port and target LID
>>
>> This way number of sources is equal to number of physical
>> non-switch ports in the subnet, and only number of targets
>> depends on LMC that is used.
>>
>> Signed-off-by: Yevgeny Kliteynik<kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
>> ---
[snip]
>> + p_node = p_src_port->p_node;
>> + if (p_node->node_info.node_type == IB_NODE_TYPE_SWITCH)
>> + return;
>
> - return;
> + continue;
>
> Otherwise we stop dumping at the first switch we encounter?
Correct, thanks :)
-- Yevgeny
> -- Jim
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-15 8:38 UTC (permalink / raw)
To: Andrea Gozzelino
Cc: Tung, Chien Tin, Steve Wise,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <1205877.1271315264088.SLOX.WebMail.wwwrun-XDIR3SKYeFbgKi2NxijLtw@public.gmane.org>
It should be a simple fix and I plan to do soon - just add yourself as
CC in bugzilla - that way I won't forget to notify you.
- amir
On 04/15/2010 10:07 AM, Andrea Gozzelino wrote:
> On Apr 15, 2010 08:24 AM, Amir Vadai <amirv-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org> wrote:
>
>
>> I hope to have a fix next week for the first one.
>>
>> Thanks,
>> Amir
>>
>> On 04/14/2010 09:48 PM, Tung, Chien Tin wrote:
>>
>>>> Tung, Chien Tin wrote:
>>>>
>>>>
>>>>>> One more thing - Please open a bug regarding the num_sge
>>>>>> limitation at:
>>>>>> https://bugs.openfabrics.org/
>>>>>>
>>>>>>
>>>>>>
>>>>> Done, Bug 2027.
>>>>>
>>>>> Chien
>>>>>
>>>>>
>>>>>
>>>> And 2028 opened to request fastreg support.
>>>>
>>>>
>>>>
>>> I am open to test fixes for these two bugs.
>>>
>>> Chien
>>>
>>>
>>>
>>
> Hi Amir,
> Hi Chien,
>
> I understand that the bug 2027 could be solved next week, so I will test
> SDP protocol performance on NE020 cards.
> Is it correct?
> If yes, could you point out the code modifies?
>
> Keep in touch and take care.
> Regards,
> Andrea
>
>
> Andrea Gozzelino
>
> INFN - Laboratori Nazionali di Legnaro (LNL)
> Viale dell'Universita' 2
> I-35020 - Legnaro (PD)- ITALIA
> Tel: +39 049 8068346
> Fax: +39 049 641925
> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Andrea Gozzelino @ 2010-04-15 7:07 UTC (permalink / raw)
To: Amir Vadai
Cc: Tung, Chien Tin, Steve Wise, Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC6B135.2060004-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
On Apr 15, 2010 08:24 AM, Amir Vadai <amirv-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org> wrote:
> I hope to have a fix next week for the first one.
>
> Thanks,
> Amir
>
> On 04/14/2010 09:48 PM, Tung, Chien Tin wrote:
> >> Tung, Chien Tin wrote:
> >>
> >>>> One more thing - Please open a bug regarding the num_sge
> >>>> limitation at:
> >>>> https://bugs.openfabrics.org/
> >>>>
> >>>>
> >>> Done, Bug 2027.
> >>>
> >>> Chien
> >>>
> >>>
> >> And 2028 opened to request fastreg support.
> >>
> >>
> >
> > I am open to test fixes for these two bugs.
> >
> > Chien
> >
> >
>
Hi Amir,
Hi Chien,
I understand that the bug 2027 could be solved next week, so I will test
SDP protocol performance on NE020 cards.
Is it correct?
If yes, could you point out the code modifies?
Keep in touch and take care.
Regards,
Andrea
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2
I-35020 - Legnaro (PD)- ITALIA
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-15 6:24 UTC (permalink / raw)
To: Tung, Chien Tin
Cc: Steve Wise, Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <2EFBCAEF10980645BBCFB605689E08E9047C79BD12-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
I hope to have a fix next week for the first one.
Thanks,
Amir
On 04/14/2010 09:48 PM, Tung, Chien Tin wrote:
>> Tung, Chien Tin wrote:
>>
>>>> One more thing - Please open a bug regarding the num_sge limitation at:
>>>> https://bugs.openfabrics.org/
>>>>
>>>>
>>> Done, Bug 2027.
>>>
>>> Chien
>>>
>>>
>> And 2028 opened to request fastreg support.
>>
>>
>
> I am open to test fixes for these two bugs.
>
> Chien
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [infiniband-diags] [2/2] check for duplicate port guids in libibnetdisc cache
From: Al Chu @ 2010-04-14 22:48 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 296 bytes --]
Hey Sasha,
This patch checks for duplicate port guids in a libibnetdisc cache when
it is loaded and report an error back to the user appropriately.
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0002-check-for-duplicate-port-guids-in-libibnetdisc-cache.patch --]
[-- Type: message/rfc822, Size: 1101 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] check for duplicate port guids in libibnetdisc cache
Date: Wed, 14 Apr 2010 15:25:11 -0700
Message-ID: <1271285070.17987.138.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
.../libibnetdisc/src/ibnetdisc_cache.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
index 2ec353d..a02fcba 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
@@ -505,6 +505,11 @@ static int _fill_port(ibnd_fabric_cache_t * fabric_cache, ibnd_node_t * node,
return -1;
}
+ if (port_cache->port_stored_to_fabric) {
+ IBND_DEBUG("Cache invalid: duplicate port discovered\n");
+ return -1;
+ }
+
node->ports[port_cache->port->portnum] = port_cache->port;
port_cache->port_stored_to_fabric++;
--
1.5.4.5
^ permalink raw reply related
* [infiniband-diags] [1/2] fix libibnetdisc cache error path memleak
From: Al Chu @ 2010-04-14 22:47 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 360 bytes --]
Hey Sasha,
This patch fixes a mem-leak through error paths in the libibnetdisc
cache loading. If some data had not yet been "copied over" to the
fabric struct and an error occurred, that memory would be leaked.
Al
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0001-fix-libibnetdisc-cache-error-path-memleak.patch --]
[-- Type: message/rfc822, Size: 3710 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] fix libibnetdisc cache error path memleak
Date: Wed, 14 Apr 2010 14:11:27 -0700
Message-ID: <1271284770.17987.131.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/ibnetdisc.c | 2 +-
.../libibnetdisc/src/ibnetdisc_cache.c | 13 +++++++++++--
infiniband-diags/libibnetdisc/src/internal.h | 2 ++
3 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
index c895de9..dfd7dce 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
@@ -510,7 +510,7 @@ error:
return NULL;
}
-static void destroy_node(ibnd_node_t * node)
+void destroy_node(ibnd_node_t * node)
{
int p = 0;
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
index 480a0a2..2ec353d 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc_cache.c
@@ -113,6 +113,7 @@ typedef struct ibnd_node_cache {
ibnd_port_cache_key_t *port_cache_keys;
struct ibnd_node_cache *next;
struct ibnd_node_cache *htnext;
+ int node_stored_to_fabric;
} ibnd_node_cache_t;
typedef struct ibnd_port_cache {
@@ -122,6 +123,7 @@ typedef struct ibnd_port_cache {
ibnd_port_cache_key_t remoteport_cache_key;
struct ibnd_port_cache *next;
struct ibnd_port_cache *htnext;
+ int port_stored_to_fabric;
} ibnd_port_cache_t;
typedef struct ibnd_fabric_cache {
@@ -257,6 +259,9 @@ static int _load_header_info(int fd, ibnd_fabric_cache_t * fabric_cache,
static void _destroy_ibnd_node_cache(ibnd_node_cache_t * node_cache)
{
free(node_cache->port_cache_keys);
+ if (!node_cache->node_stored_to_fabric
+ && node_cache->node)
+ destroy_node(node_cache->node);
free(node_cache);
}
@@ -283,6 +288,9 @@ static void _destroy_ibnd_fabric_cache(ibnd_fabric_cache_t * fabric_cache)
while (port_cache) {
port_cache_next = port_cache->next;
+ if (!port_cache->port_stored_to_fabric
+ && port_cache->port)
+ free(port_cache->port);
free(port_cache);
port_cache = port_cache_next;
@@ -387,8 +395,6 @@ static int _load_node(int fd, ibnd_fabric_cache_t * fabric_cache)
return 0;
cleanup:
- /* note, no need to destroy node through destroy_node(), nothing else malloced */
- free(node);
_destroy_ibnd_node_cache(node_cache);
return -1;
}
@@ -500,6 +506,7 @@ static int _fill_port(ibnd_fabric_cache_t * fabric_cache, ibnd_node_t * node,
}
node->ports[port_cache->port->portnum] = port_cache->port;
+ port_cache->port_stored_to_fabric++;
/* achu: needed if user wishes to re-cache a loaded fabric.
* Otherwise, mostly unnecessary to do this.
@@ -532,6 +539,8 @@ static int _rebuild_nodes(ibnd_fabric_cache_t * fabric_cache)
add_to_type_list(node_cache->node, fabric_cache->fabric);
+ node_cache->node_stored_to_fabric++;
+
/* Rebuild node ports array */
if (!(node->ports =
diff --git a/infiniband-diags/libibnetdisc/src/internal.h b/infiniband-diags/libibnetdisc/src/internal.h
index 57034f9..32d567e 100644
--- a/infiniband-diags/libibnetdisc/src/internal.h
+++ b/infiniband-diags/libibnetdisc/src/internal.h
@@ -98,4 +98,6 @@ void add_to_portguid_hash(ibnd_port_t * port, ibnd_port_t * hash[]);
void add_to_type_list(ibnd_node_t * node, ibnd_fabric_t * fabric);
+void destroy_node(ibnd_node_t * node);
+
#endif /* _INTERNAL_H_ */
--
1.5.4.5
^ permalink raw reply related
* Re: [PATCH 1/4] opensm: added function that dumps PathRecords
From: Jim Schutt @ 2010-04-14 22:16 UTC (permalink / raw)
To: Yevgeny Kliteynik; +Cc: Sasha Khapyorsky, Linux RDMA
In-Reply-To: <4BBDDA1E.7030102-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
Hi Yevgeny,
On Thu, 2010-04-08 at 07:29 -0600, Yevgeny Kliteynik wrote:
> Dumping SL, MTU and Rate for all the
> non-switch-2-non-switch paths in the subnet.
>
> PRs that are dumped:
>
> for every non-switch source port
> for every non-switch target LID in the subnet
> dump PR between source port and target LID
>
> This way number of sources is equal to number of physical
> non-switch ports in the subnet, and only number of targets
> depends on LMC that is used.
>
> Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
> ---
[snip]
> +
> /****f* OpenSM: MC Member Record Receiver/osm_mcmr_rcv_find_or_create_new_mgrp
> * NAME
> * osm_mcmr_rcv_find_or_create_new_mgrp
> diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c
> index 8aab548..83da258 100644
> --- a/opensm/opensm/osm_sa.c
> +++ b/opensm/opensm/osm_sa.c
> @@ -718,6 +718,87 @@ int osm_sa_db_file_dump(osm_opensm_t * p_osm)
> return res;
> }
>
> +typedef struct _path_parms {
> + ib_net16_t pkey;
> + uint8_t mtu;
> + uint8_t rate;
> + uint8_t sl;
> + uint8_t pkt_life;
> + boolean_t reversible;
> +} path_parms_t;
> +
> +extern ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
> + IN const osm_port_t * p_src_port,
> + IN const osm_port_t * p_dest_port,
> + IN const uint16_t dlid_ho,
> + OUT path_parms_t * p_parms);
> +
> +static void sa_dump_path_records(osm_opensm_t * p_osm, FILE * file)
> +{
> + osm_port_t *p_src_port;
> + osm_port_t *p_dest_port;
> + osm_node_t *p_node;
> + uint16_t dlid_ho;
> + uint32_t vector_size;
> + osm_physp_t *p_physp;
> + path_parms_t path_parms;
> + ib_api_status_t status;
> +
> + vector_size = cl_ptr_vector_get_size(&p_osm->subn.port_lid_tbl);
> + for (p_src_port = (osm_port_t *) cl_qmap_head(&p_osm->subn.port_guid_tbl);
> + p_src_port != (osm_port_t *) cl_qmap_end(&p_osm->subn.port_guid_tbl);
> + p_src_port = (osm_port_t *) cl_qmap_next(&p_src_port->map_item)) {
> +
> + p_node = p_src_port->p_node;
> + if (p_node->node_info.node_type == IB_NODE_TYPE_SWITCH)
> + return;
- return;
+ continue;
Otherwise we stop dumping at the first switch we encounter?
-- Jim
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-14 19:24 UTC (permalink / raw)
To: Andrea Gozzelino, Steve Wise
Cc: linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <7037536.1271235105513.SLOX.WebMail.wwwrun-XDIR3SKYeFbgKi2NxijLtw@public.gmane.org>
>I work on NE020 cards from February 2010 for an INFN experimental
>proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>acquisition and movement systems. The covergence of storage protocols
>around 10 Gigabits/s Ethernet protocols shows that one way could be the
>Remote Direct Memory Access (RDMA). The goals are the investigations of
>latency time, the throughput, the buffer size schemes and finally the
>global event building bandwidth.
>
>Do you know if NE020 cards have problems with librdma (RDMA procedures,
>in general) and / or with MPI versions?
If you run into any issues, please email me. NE020 supports Mvapich,
Mvapich2, OpenMPI, Intel MPI and HP MPI.
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* OFED Cross Reference Server update
From: John Groves @ 2010-04-14 19:09 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
As some of you know, System Fabric Works hosts an indexed, hyper
linked and search-able cross reference of the OFED code - based on the
LXR cross referencing engine. I have just added and indexed
OFED-1.5.1, as well as variant of the 1.5.1 with our new Soft RoCEE
driver and library included (OFED-1.5.1-rxe).
You can access the cross reference here:
http://SystemFabricWorks.com/ofed-xr.html. The code for all recent
OFED releases can be browsed and searched, and symbols appear as hyper
links that retrieve all references to the symbols.
I have personally found this to be very useful, and I hope others will as well.
Regards,
John Groves
John-1EUXpbKPehdhfwPrcV3yrZqQE7yCjDx5@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-14 18:48 UTC (permalink / raw)
To: Steve Wise
Cc: Amir Vadai, Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D9D4.7020702-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>Tung, Chien Tin wrote:
>>> One more thing - Please open a bug regarding the num_sge limitation at:
>>> https://bugs.openfabrics.org/
>>>
>>
>> Done, Bug 2027.
>>
>> Chien
>>
>
>And 2028 opened to request fastreg support.
>
I am open to test fixes for these two bugs.
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [infiniband-diags] fix libibnetdisc portguid hashing corner case
From: Al Chu @ 2010-04-14 17:48 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
[-- Attachment #1: Type: text/plain, Size: 664 bytes --]
Hey Sasha,
This patch fixes a corner case in libibnetdisc that was storing
portguids w/ a guid of 0.
This bug was relatively innoucuous for ibnetdiscover b/c ibnetdiscover
does not output these ports. However, it became a problem for me in the
caching library as I attempted to reconstruct a fabric, and multiple
ports were identifying themselves with identical guids [1].
Al
[1] - The fact the caching code assumes duplicate guids can't exist is
also a bug. But that's for another patch. This is a bug by itself.
--
Albert Chu
chu11-i2BcT+NCU+M@public.gmane.org
Computer Scientist
High Performance Systems Division
Lawrence Livermore National Laboratory
[-- Attachment #2: 0001-fix-libibnetdisc-portguid-hashing-corner-case.patch --]
[-- Type: message/rfc822, Size: 996 bytes --]
From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
Subject: [PATCH] fix libibnetdisc portguid hashing corner case
Date: Tue, 13 Apr 2010 14:01:43 -0700
Message-ID: <1271264633.17987.112.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/ibnetdisc.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
index c895de9..6010a28 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
@@ -173,6 +173,7 @@ static int recv_port_info(smp_engine_t * engine, ibnd_smp_t * smp,
IBND_ERROR("Failed to allocate port\n");
return -1;
}
+ port->guid = mad_get_field64(node->info, 0, IB_NODE_PORT_GUID_F);
}
memcpy(port->info, port_info, sizeof(port->info));
--
1.5.4.5
^ permalink raw reply related
* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Ira Weiny @ 2010-04-14 16:52 UTC (permalink / raw)
To: Sasha Khapyorsky
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100414102335.GT10830@me>
On Apr 14, 2010, at 3:23 AM, Sasha Khapyorsky wrote:
> On 13:44 Tue 13 Apr , Ira Weiny wrote:
>>
>>> This changes the logic. "num_smps_outstanding" is NOT the number
>>> on the wire, but it appears you have made it so.
>
> Actually yes, it made it so.
>
>>> This is the number which will cause process_smp_queue to continue
>>> being called.
>>>
>>> If you are going to do this I think you need to change
>>> process_mads as well as process_one_recv. We discussed
>>> process_one_recv in the error case.
>
> process_one_recv() failure breaks the loop anyway.
>
>>> What were you trying to fix?
>>
>> Ok, I think I see. We should move cl_qmap_insert to after a
>> successful umad_send and putting total_smps here is ok. But
>> num_smps_outstanding should be put back I think.
>
> But then it blocks process_mads() to loop forever after single
> send_smp() failure (with all empty queues and umad_recv() running
> without timeout).
But moving the cl_qmap_insert below the send call fixes that.
However, it does cause a memory leak because the smp is no longer in
the smp_queue_head list. It needs to be put back on that list to be
retried with a limit on the retries (to prevent what you are saying
here.) Are you seeing a hang?
I have seen a hang when running "iblinkinfo -S <guid>". However, the
problem is not with send_smp. I am seeing the mad going on the wire
and returning (according to madeye) but I am not receiving it from
umad_recv. I don't know why. If I run with 1 outstanding mad it
works???
Ira
>
> Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] ummunotify: fix umn-test build
From: Randy Dunlap @ 2010-04-14 16:43 UTC (permalink / raw)
To: Eric B Munson
Cc: akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI
In-Reply-To: <1271053337-7121-1-git-send-email-ebmunson-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
From: Randy Dunlap <randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Add ummunotify.h to Kbuild list for export to userspace,
fixing 27 build errors in umn-test.c when O=builddir is used.
Signed-off-by: Randy Dunlap <randy.dunlap-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
---
include/linux/Kbuild | 1 +
1 file changed, 1 insertion(+)
maybe another item to add to SubmitChecklist :(
--- lnx-2634-rc4.orig/include/linux/Kbuild
+++ lnx-2634-rc4/include/linux/Kbuild
@@ -163,6 +163,7 @@ header-y += tipc_config.h
header-y += toshiba.h
header-y += udf_fs_i.h
header-y += ultrasound.h
+header-y += ummunotify.h
header-y += un.h
header-y += utime.h
header-y += veth.h
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [ANNOUNCE] libcxgb4 1.0.0 Release
From: Steve Wise @ 2010-04-14 16:19 UTC (permalink / raw)
To: linux-rdma
The libcxgb4 package is a userspace driver for the new Chelsio T4 iWARP
RNICs. It is a plug-in module for libibverbs that allows programs to
use Chelsio RDMA T4 hardware directly from userspace.
The initial release of this library is now available at:
http://www.openfabrics.org/downloads/cxgb4/libcxgb4-1.0.0.tar.gz
with md5sum:
21c3505414e861837d4d54d501d918e9 libcxgb4-1.0.0.tar.gz
Also, I welcome code review comments if anyone cares to review the code.
My development git tree is at:
git://www.openfabrics.org/~swise/libcxgb4
<http://www.openfabrics.org/git/?p=%7Eswise/libcxgb4.git;a=summary>
Thanks,
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RDMA CM problems, ib_find_cached_gid() fails
From: Mike M @ 2010-04-14 15:55 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <CC1A47DBF3084F98B391E73238EAE221-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Tue, Apr 13, 2010 at 5:41 PM, Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
>>Putting in lots of printk messages, I see that the server machine
>>indeed gets a connection request MAD, but that the
>>ib_find_cached_gid() call inside of cma_acquire_dev() fails.
>>
>>Any ideas?
>
> What kernel version are you using, and are you trying to connect using IPv4 or
> IPv6 addresses?
>
A modified 2.6.26-based kernel patched with updated OFED drivers based
on an rc version of 1.4.1. IPv4 addresses.
I reverted to standard 2.6.26 and everything worked fine. I will
further investigate.
Regards,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-14 15:16 UTC (permalink / raw)
To: Steve Wise
Cc: Andrea Gozzelino, Tung, Chien Tin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5DA5F.6010607-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
ok - actually I used it in an early version of SDP before changing to FMR...
- Amir
On 04/14/2010 06:08 PM, Steve Wise wrote:
> Amir Vadai wrote:
>
>> You are right - I missed it.
>>
>> Andrea, Please open a bug at bugzilla (https://bugs.openfabrics.org) -
>> so that you will be notified as soon as I will fix SDP not use FMR if
>> not supported.
>>
>> As to fastreg_mrs support - I don't know this mechanism. Do you mean FRWR?
>>
>>
>>
> ib_alloc_fast_reg_mr(), ib_alloc_fast_reg_page_list() and friends, plus
> the IB_WR_FAST_REG_MR work request.
>
>
>
>> Thanks,
>> Amir
>>
>>
>> On 04/14/2010 05:54 PM, Steve Wise wrote:
>>
>>
>>> Hey Amir,
>>>
>>> I don't think this helps because sdp_add_device() will not add rdma
>>> devices that fail to create fmr pools.
>>>
>>> So I guess you could key off of fmr pool failures and set
>>> sdp_zcopy_thresh to 0 and allow the device to be used?
>>>
>>> But what we really need is sdp support for fastreg_mrs as an
>>> alternative to fmrs.
>>>
>>>
>>> Steve.
>>>
>>>
>>> Amir Vadai wrote:
>>>
>>>
>>>
>>>> Hi,
>>>>
>>>> FMR are being used only in a special mode called ZCopy.
>>>>
>>>> You could disable this mode by setting the module paramter
>>>> sdp_zcopy_thresh to 0, or by issuing:
>>>> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>>>>
>>>> This means that you won't get the benefits of Zero-copy.
>>>>
>>>> - Amir
>>>>
>>>> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>>>>> code
>>>>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Good to know. Thanks.
>>>>>>
>>>>>> Chien
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Hi Steve and Chien,
>>>>>
>>>>> I understand that NE020 cards have problem with SDP connected with
>>>>> map_phy_fmr (FMR stands for Fast Memory Region).
>>>>> Is it possible to solve/fix this point?
>>>>> If yes, have you an idea about the time that is necessary to code
>>>>> development/build?
>>>>> If no, can you suggest me a card that supports SDP protocol?
>>>>>
>>>>> I work on NE020 cards from February 2010 for an INFN experimental
>>>>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>>>>> acquisition and movement systems. The covergence of storage protocols
>>>>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>>>>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>>>>> latency time, the throughput, the buffer size schemes and finally the
>>>>> global event building bandwidth.
>>>>>
>>>>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>>>>> in general) and / or with MPI versions?
>>>>>
>>>>> Thank you very much,
>>>>> Andrea
>>>>>
>>>>> Andrea Gozzelino
>>>>>
>>>>> INFN - Laboratori Nazionali di Legnaro (LNL)
>>>>> Viale dell'Universita' 2
>>>>> I-35020 - Legnaro (PD)- ITALIA
>>>>> Tel: +39 049 8068346
>>>>> Fax: +39 049 641925
>>>>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>> Please read the FAQ at http://www.tux.org/lkml/
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Andrea Gozzelino @ 2010-04-14 15:14 UTC (permalink / raw)
To: Steve Wise
Cc: Tung, Chien Tin, Amir Vadai, Andrea Gozzelino,
linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
rolandd@cisco.com, peterz@infradead.org, pavel@ucw.cz,
mingo@elte.hu, Eric B Munson
In-Reply-To: <4BC5D9D4.7020702@opengridcomputing.com>
Thank you very much.
I check the status of 2027 and 2028 bugs.
Andrea
n Apr 14, 2010 05:05 PM, Steve Wise <swise@opengridcomputing.com> wrote:
> Tung, Chien Tin wrote:
> >> One more thing - Please open a bug regarding the num_sge limitation
> >> at:
> >> https://bugs.openfabrics.org/
> >>
> >
> > Done, Bug 2027.
> >
> > Chien
> >
>
> And 2028 opened to request fastreg support.
>
> Steve.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2
I-35020 - Legnaro (PD)- ITALIA
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzelino@lnl.infn.it
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-14 15:08 UTC (permalink / raw)
To: Amir Vadai
Cc: Andrea Gozzelino, Tung, Chien Tin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D95D.3010108-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Amir Vadai wrote:
> You are right - I missed it.
>
> Andrea, Please open a bug at bugzilla (https://bugs.openfabrics.org) -
> so that you will be notified as soon as I will fix SDP not use FMR if
> not supported.
>
> As to fastreg_mrs support - I don't know this mechanism. Do you mean FRWR?
>
>
ib_alloc_fast_reg_mr(), ib_alloc_fast_reg_page_list() and friends, plus
the IB_WR_FAST_REG_MR work request.
> Thanks,
> Amir
>
>
> On 04/14/2010 05:54 PM, Steve Wise wrote:
>
>> Hey Amir,
>>
>> I don't think this helps because sdp_add_device() will not add rdma
>> devices that fail to create fmr pools.
>>
>> So I guess you could key off of fmr pool failures and set
>> sdp_zcopy_thresh to 0 and allow the device to be used?
>>
>> But what we really need is sdp support for fastreg_mrs as an
>> alternative to fmrs.
>>
>>
>> Steve.
>>
>>
>> Amir Vadai wrote:
>>
>>
>>> Hi,
>>>
>>> FMR are being used only in a special mode called ZCopy.
>>>
>>> You could disable this mode by setting the module paramter
>>> sdp_zcopy_thresh to 0, or by issuing:
>>> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>>>
>>> This means that you won't get the benefits of Zero-copy.
>>>
>>> - Amir
>>>
>>> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>>>
>>>
>>>
>>>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>>>> code
>>>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Good to know. Thanks.
>>>>>
>>>>> Chien
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Hi Steve and Chien,
>>>>
>>>> I understand that NE020 cards have problem with SDP connected with
>>>> map_phy_fmr (FMR stands for Fast Memory Region).
>>>> Is it possible to solve/fix this point?
>>>> If yes, have you an idea about the time that is necessary to code
>>>> development/build?
>>>> If no, can you suggest me a card that supports SDP protocol?
>>>>
>>>> I work on NE020 cards from February 2010 for an INFN experimental
>>>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>>>> acquisition and movement systems. The covergence of storage protocols
>>>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>>>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>>>> latency time, the throughput, the buffer size schemes and finally the
>>>> global event building bandwidth.
>>>>
>>>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>>>> in general) and / or with MPI versions?
>>>>
>>>> Thank you very much,
>>>> Andrea
>>>>
>>>> Andrea Gozzelino
>>>>
>>>> INFN - Laboratori Nazionali di Legnaro (LNL)
>>>> Viale dell'Universita' 2
>>>> I-35020 - Legnaro (PD)- ITALIA
>>>> Tel: +39 049 8068346
>>>> Fax: +39 049 641925
>>>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at http://www.tux.org/lkml/
>>>
>>>
>>>
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox