* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-08 16:00 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <r2le2e108261004080807i36ad0755gad9843224b0c42fc-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> It would be great if someone could explain the impact of this issue --
> the impact is not yet clear to me.
Um, it's a one-line patch -- pretty clear if the allocation fails then
we check the wrong value and probably crash.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch] infiniband: checking the wrong variable
From: Bart Van Assche @ 2010-04-08 15:07 UTC (permalink / raw)
To: Roland Dreier
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <ada7hoixaqw.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, Apr 8, 2010 at 4:46 PM, Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > Have you already had the time to check whether this fix should be
> > backported to already released kernels, and if so, to which kernel
> > versions ? The easiest way to ask for backporting is to add "Cc:
> > <stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
>
> It backports but I don't see it as fixing a big enough problem to be
> worth it for stable.
It would be great if someone could explain the impact of this issue --
the impact is not yet clear to me.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-08 14:46 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <g2ge2e108261004080313t32c2e1besd8a95e2f9bf418f9-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> Have you already had the time to check whether this fix should be
> backported to already released kernels, and if so, to which kernel
> versions ? The easiest way to ask for backporting is to add "Cc:
> <stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
It backports but I don't see it as fixing a big enough problem to be
worth it for stable.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 4/4] opensm: add command line argument to dump PR file
From: Yevgeny Kliteynik @ 2010-04-08 13:30 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Added command line argument to dump PathRecord
file and updated the man page.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/man/opensm.8.in | 8 +++++++-
opensm/opensm/main.c | 9 +++++++++
2 files changed, 16 insertions(+), 1 deletions(-)
diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in
index 7aca8f9..8d3ce97 100644
--- a/opensm/man/opensm.8.in
+++ b/opensm/man/opensm.8.in
@@ -1,4 +1,4 @@
-.TH OPENSM 8 "October 22, 2009" "OpenIB" "OpenIB Management"
+.TH OPENSM 8 "April 8, 2010" "OpenIB" "OpenIB Management"
.SH NAME
opensm \- InfiniBand subnet manager and administration (SM/SA)
@@ -28,6 +28,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA)
[\-H | \-\-max_reverse_hops <max reverse hops allowed>]
[\-X | \-\-guid_routing_order_file <path to file>]
[\-m | \-\-ids_guid_file <path to file>]
+[\-\-dump_path_records]
[\-o(nce)]
[\-s(weep) <interval>]
[\-t(imeout) <milliseconds>]
@@ -218,6 +219,11 @@ Set the order port guids will be routed for the MinHop
and Up/Down routing algorithms to the guids provided in the
given file (one to a line).
.TP
+\fB\-\-dump_path_records\fR
+This option causes OpenSM to dump Path Records file.
+This file can be used by applications such as MPI to
+eliminate the need to query SA.
+.TP
\fB\-o\fR, \fB\-\-once\fR
This option causes OpenSM to configure the subnet
once, then exit. Ports remain in the ACTIVE state.
diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c
index f9a33af..9d5c886 100644
--- a/opensm/opensm/main.c
+++ b/opensm/opensm/main.c
@@ -231,6 +231,10 @@ static void show_usage(void)
" Set the order port guids will be routed for the MinHop\n"
" and Up/Down routing algorithms to the guids provided in the\n"
" given file (one to a line)\n\n");
+ printf("--dump_path_records\n"
+ " This option causes OpenSM to dump Path Records file.\n"
+ " This file can be used by applications such as MPI to\n"
+ " eliminate the need to query SA.\n\n");
printf("--once, -o\n"
" This option causes OpenSM to configure the subnet\n"
" once, then exit. Ports remain in the ACTIVE state.\n\n");
@@ -610,6 +614,7 @@ int main(int argc, char *argv[])
{"sm_sl", 1, NULL, 7},
{"retries", 1, NULL, 8},
{"log_prefix", 1, NULL, 9},
+ {"dump_path_records", 0, NULL, 10},
{NULL, 0, NULL, 0} /* Required at the end of the array */
};
@@ -992,6 +997,10 @@ int main(int argc, char *argv[])
SET_STR_OPT(opt.log_prefix, optarg);
printf("Log prefix = %s\n", opt.log_prefix);
break;
+ case 10:
+ opt.path_rec_dump = TRUE;
+ printf(" Dumping Path Records file\n");
+ break;
case 'h':
case '?':
case ':':
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 3/4] opensm: dump PRs after every heavy sweep and after reroute
From: Yevgeny Kliteynik @ 2010-04-08 13:30 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_dump.c | 3 +++
opensm/opensm/osm_state_mgr.c | 2 ++
2 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c
index 86e9c00..cedff24 100644
--- a/opensm/opensm/osm_dump.c
+++ b/opensm/opensm/osm_dump.c
@@ -634,6 +634,9 @@ void osm_dump_all(osm_opensm_t * osm)
osm_dump_qmap_to_file(osm, "opensm-subnet.lst",
&osm->subn.node_guid_tbl, dump_topology_node,
osm);
+ if (osm->subn.opt.path_rec_dump)
+ osm_sa_path_records_file_dump(osm);
+
if (osm_log_is_active(&osm->log, OSM_LOG_VERBOSE))
print_report(osm, stdout);
}
diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c
index e43463f..e32ff18 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1149,6 +1149,8 @@ static void do_sweep(osm_sm_t * sm)
return;
if (!sm->p_subn->subnet_initialization_error) {
+ if (sm->p_subn->opt.path_rec_dump)
+ osm_sa_path_records_file_dump(sm->p_subn->p_osm);
OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE,
"REROUTE COMPLETE");
return;
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 2/4] opensm: added 2 options: dump PRs and filename
From: Yevgeny Kliteynik @ 2010-04-08 13:29 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_subnet.h | 10 ++++++++++
opensm/opensm/osm_subnet.c | 14 ++++++++++++++
2 files changed, 24 insertions(+), 0 deletions(-)
diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h
index 3970e98..8d7182b 100644
--- a/opensm/include/opensm/osm_subnet.h
+++ b/opensm/include/opensm/osm_subnet.h
@@ -201,6 +201,8 @@ typedef struct osm_subn_opt {
char *guid_routing_order_file;
char *sa_db_file;
boolean_t sa_db_dump;
+ boolean_t path_rec_dump;
+ char *path_rec_file;
boolean_t do_mesh_analysis;
boolean_t exit_on_fatal;
boolean_t honor_guid2lid_file;
@@ -418,6 +420,14 @@ typedef struct osm_subn_opt {
* When TRUE causes OpenSM to dump SA DB at the end of every
* light sweep regardless the current verbosity level.
*
+* path_rec_file
+* Name of the PathRecords dump file.
+*
+* path_rec_dump
+* When TRUE causes OpenSM to dump PathRecords for all the
+* CA-2-CA paths at the end of every heavy sweep, regardless
+* of the verbosity level.
+*
* exit_on_fatal
* If TRUE (default) - SM will exit on fatal subnet initialization
* issues.
diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c
index e4126bc..c87ed2a 100644
--- a/opensm/opensm/osm_subnet.c
+++ b/opensm/opensm/osm_subnet.c
@@ -757,6 +757,8 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt)
p_opt->guid_routing_order_file = NULL;
p_opt->sa_db_file = NULL;
p_opt->sa_db_dump = FALSE;
+ p_opt->path_rec_file = NULL;
+ p_opt->path_rec_dump = FALSE;
p_opt->do_mesh_analysis = FALSE;
p_opt->exit_on_fatal = TRUE;
p_opt->enable_quirks = FALSE;
@@ -1464,6 +1466,18 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)
p_opts->sa_db_dump ? "TRUE" : "FALSE");
fprintf(out,
+ "# PathRecords dump file name\n"
+ "path_rec_file %s\n\n",
+ p_opts->path_rec_file ? p_opts->path_rec_file : null_str);
+
+ fprintf(out,
+ "# If TRUE causes OpenSM to dump PathRecords for all the\n"
+ "# CA-2-CA paths at the end of every heavy sweep, regardless\n"
+ "# of the verbosity level\n"
+ "path_rec_dump %s\n\n",
+ p_opts->path_rec_dump ? "TRUE" : "FALSE");
+
+ fprintf(out,
"#\n# HANDOVER - MULTIPLE SMs OPTIONS\n#\n"
"# SM priority used for deciding who is the master\n"
"# Range goes from 0 (lowest priority) to 15 (highest).\n"
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 1/4] opensm: added function that dumps PathRecords
From: Yevgeny Kliteynik @ 2010-04-08 13:29 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Dumping SL, MTU and Rate for all the
non-switch-2-non-switch paths in the subnet.
PRs that are dumped:
for every non-switch source port
for every non-switch target LID in the subnet
dump PR between source port and target LID
This way number of sources is equal to number of physical
non-switch ports in the subnet, and only number of targets
depends on LMC that is used.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_sa.h | 21 +++++++++
opensm/opensm/osm_sa.c | 81 ++++++++++++++++++++++++++++++++++++
opensm/opensm/osm_sa_path_record.c | 12 +++++
3 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/opensm/include/opensm/osm_sa.h b/opensm/include/opensm/osm_sa.h
index d516310..40622ff 100644
--- a/opensm/include/opensm/osm_sa.h
+++ b/opensm/include/opensm/osm_sa.h
@@ -462,6 +462,27 @@ int osm_sa_db_file_load(struct osm_opensm *p_osm);
*
*********/
+/****f* OpenSM: SA/osm_sa_path_records_file_dump
+* NAME
+* osm_sa_path_records_file_dump
+*
+* DESCRIPTION
+* Dumps the SA Path Records to the dump file.
+*
+* SYNOPSIS
+*/
+int osm_sa_path_records_file_dump(struct osm_opensm *p_osm);
+/*
+* PARAMETERS
+* p_osm
+* [in] Pointer to an osm_opensm_t object.
+*
+* RETURN VALUES
+* 0 on success
+* !0 if some error occurred.
+*
+*********/
+
/****f* OpenSM: MC Member Record Receiver/osm_mcmr_rcv_find_or_create_new_mgrp
* NAME
* osm_mcmr_rcv_find_or_create_new_mgrp
diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c
index 8aab548..83da258 100644
--- a/opensm/opensm/osm_sa.c
+++ b/opensm/opensm/osm_sa.c
@@ -718,6 +718,87 @@ int osm_sa_db_file_dump(osm_opensm_t * p_osm)
return res;
}
+typedef struct _path_parms {
+ ib_net16_t pkey;
+ uint8_t mtu;
+ uint8_t rate;
+ uint8_t sl;
+ uint8_t pkt_life;
+ boolean_t reversible;
+} path_parms_t;
+
+extern ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
+ IN const osm_port_t * p_src_port,
+ IN const osm_port_t * p_dest_port,
+ IN const uint16_t dlid_ho,
+ OUT path_parms_t * p_parms);
+
+static void sa_dump_path_records(osm_opensm_t * p_osm, FILE * file)
+{
+ osm_port_t *p_src_port;
+ osm_port_t *p_dest_port;
+ osm_node_t *p_node;
+ uint16_t dlid_ho;
+ uint32_t vector_size;
+ osm_physp_t *p_physp;
+ path_parms_t path_parms;
+ ib_api_status_t status;
+
+ vector_size = cl_ptr_vector_get_size(&p_osm->subn.port_lid_tbl);
+ for (p_src_port = (osm_port_t *) cl_qmap_head(&p_osm->subn.port_guid_tbl);
+ p_src_port != (osm_port_t *) cl_qmap_end(&p_osm->subn.port_guid_tbl);
+ p_src_port = (osm_port_t *) cl_qmap_next(&p_src_port->map_item)) {
+
+ p_node = p_src_port->p_node;
+ if (p_node->node_info.node_type == IB_NODE_TYPE_SWITCH)
+ return;
+
+ p_physp = p_src_port->p_physp;
+ CL_ASSERT(p_physp->p_remote_physp);
+
+ fprintf(file, "%s 0x%016" PRIx64 ", base LID %d, "
+ "\"%s\", port %d\n# LID : SL : MTU : RATE\n",
+ ib_get_node_type_str(p_node->node_info.node_type),
+ cl_ntoh64(p_src_port->guid),
+ cl_ntoh16(osm_port_get_base_lid(p_src_port)),
+ p_node->print_desc, p_physp->port_num);
+
+ memset(&path_parms, 0, sizeof(path_parms_t));
+
+ for (dlid_ho = 1; dlid_ho < vector_size; dlid_ho++) {
+
+ p_dest_port = (osm_port_t *) cl_ptr_vector_get(
+ &p_osm->subn.port_lid_tbl, dlid_ho);
+
+ if (!p_dest_port || !p_dest_port->p_node ||
+ p_dest_port->p_node->node_info.node_type ==
+ IB_NODE_TYPE_SWITCH)
+ continue;
+
+ status = osm_get_path_params(&p_osm->sa,
+ p_src_port, p_dest_port, dlid_ho,
+ (void*)&path_parms);
+
+ if (!status)
+ fprintf(file, "0x%04X : %-2d : %-3d : %-4d\n",
+ dlid_ho, path_parms.sl,
+ path_parms.mtu, path_parms.rate);
+ else
+ fprintf(file, "0x%04X : UNREACHABLE\n",
+ dlid_ho);
+ }
+ fprintf(file, "\n");
+ }
+}
+
+
+int osm_sa_path_records_file_dump(osm_opensm_t * p_osm)
+{
+ return opensm_dump_to_file(p_osm, p_osm->subn.opt.path_rec_file ?
+ p_osm->subn.opt.path_rec_file : "opensm-path-records.dump",
+ sa_dump_path_records);
+}
+
/*
* SA DB Loader
*/
diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c
index c4c3f86..b168428 100644
--- a/opensm/opensm/osm_sa_path_record.c
+++ b/opensm/opensm/osm_sa_path_record.c
@@ -743,6 +743,18 @@ Exit:
return status;
}
+ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
+ IN const osm_port_t * p_src_port,
+ IN const osm_port_t * p_dest_port,
+ IN const uint16_t dlid_ho,
+ OUT osm_path_parms_t * p_parms)
+{
+ ib_path_rec_t pr;
+ memset(&pr, 0, sizeof(ib_path_rec_t));
+ return pr_rcv_get_path_parms(sa, &pr,
+ p_src_port, p_dest_port, dlid_ho, 0, p_parms);
+}
+
static void pr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t * p_src_port,
IN const osm_port_t * p_dest_port,
IN const ib_gid_t * p_dgid,
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/4] opensm: dump PathRecords file
From: Yevgeny Kliteynik @ 2010-04-08 13:28 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Hi Sasha,
One of the major bottlenecks of the OpenSM is handling
SA queries storm. In order to allow all-2-all communication,
SA must process N^2 PathRecord queries.
On large clusters this takes way too much time.
Currently, most of the MPI implementations do not query SA
at all in order to avoid this problem, but use predefined
parameters when communicating between fabric nodes.
This works fine as long as MPI doesn't need to use different
parameters for different path, but it's a problem when we're
talking about non-trivial QoS settings or routing engines that
are using IB VLs (such as Torus-2QoS).
The following patch series enables OpenSM to dump core
information from PathRecords that is needed for opening
communication channel: SL, MTU and Rate.
This information is dumped for for all the non-switch-2-non-switch
paths in the subnet in the following way:
for every non-switch source port
for every non-switch target LID in the subnet
dump PR between source port and target LID
This way number of sources is equal to number of physical
non-switch ports in the subnet, and only number of targets
depends on LMC that is used.
Patches:
[PATCH 1/4] opensm: added function that dumps PathRecords
[PATCH 2/4] opensm: added 2 options: dump PRs and filename
[PATCH 3/4] opensm: dump PRs after every heavy sweep and
after reroute
[PATCH 4/4] opensm: add command line argument to dump PR file
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] opensm/osm_sa.c: enhance opensm_dump_to_file() to accept full path file name
From: Yevgeny Kliteynik @ 2010-04-08 13:14 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA
Hi Sasha,
Small patch that enhances opensm_dump_to_file() function
and allows it to accept full path file names as argument.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_sa.c | 9 +++++++--
1 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c
index 3473e4c..8aab548 100644
--- a/opensm/opensm/osm_sa.c
+++ b/opensm/opensm/osm_sa.c
@@ -510,8 +510,13 @@ opensm_dump_to_file(osm_opensm_t * p_osm, const char *file_name,
char path[1024];
FILE *file;
- snprintf(path, sizeof(path), "%s/%s",
- p_osm->subn.opt.dump_files_dir, file_name);
+ if (*file_name == '/')
+ /* file name was provided as an absolute path */
+ snprintf(path, sizeof(path), "%s", file_name);
+ else
+ /* file name is relative to dump_files_dir */
+ snprintf(path, sizeof(path), "%s/%s",
+ p_osm->subn.opt.dump_files_dir, file_name);
file = fopen(path, "w");
if (!file) {
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [patch] infiniband: checking the wrong variable
From: Bart Van Assche @ 2010-04-08 10:13 UTC (permalink / raw)
To: Dan Carpenter
Cc: Roland Dreier, Sean Hefty, Hal Rosenstock, Tejun Heo,
Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407093901.GG5157@bicker>
On Wed, Apr 7, 2010 at 11:39 AM, Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> The intent here was to check the "mfrpl->mapped_page_list" allocation.
> We checked "mfrpl->ibfrpl.page_list" earlier.
>
> Signed-off-by: Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>
> diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
> index 56147b2..1d27b9a 100644
> --- a/drivers/infiniband/hw/mlx4/mr.c
> +++ b/drivers/infiniband/hw/mlx4/mr.c
> @@ -240,7 +240,7 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
> mfrpl->mapped_page_list = dma_alloc_coherent(&dev->dev->pdev->dev,
> size, &mfrpl->map,
> GFP_KERNEL);
> - if (!mfrpl->ibfrpl.page_list)
> + if (!mfrpl->mapped_page_list)
> goto err_free;
>
> WARN_ON(mfrpl->map & 0x3f);
Hello Dan,
Have you already had the time to check whether this fix should be
backported to already released kernels, and if so, to which kernel
versions ? The easiest way to ask for backporting is to add "Cc:
<stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* ibdev2netdev optimization [ PATCH 1/1 ]
From: Aleksey Senin @ 2010-04-08 9:22 UTC (permalink / raw)
To: linux-rdma
[-- Attachment #1: Type: text/plain, Size: 70 bytes --]
Call just once all these sed/awk/cut etc. when finding mac addresses.
[-- Attachment #2: 0001-Optimized-when-looking-ethernet-data.patch --]
[-- Type: text/x-patch, Size: 1443 bytes --]
From 2437e980e6a8f0bcd123ddab077f00908069d379 Mon Sep 17 00:00:00 2001
From: Aleksey Senin <alekseys@voltaire.com>
Date: Thu, 8 Apr 2010 11:53:07 +0300
Subject: [PATCH] Optimized when looking ethernet data.
I have two adapters with two ports on each one. This optimization gives me
6 sec. run-time instead of 19 sec.
Signed-off-by: Aleksey Senin <alekseys@voltaire.com>
---
ofed_scripts/ibdev2netdev | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/ofed_scripts/ibdev2netdev b/ofed_scripts/ibdev2netdev
index b436c05..adb5eba 100755
--- a/ofed_scripts/ibdev2netdev
+++ b/ofed_scripts/ibdev2netdev
@@ -30,12 +30,12 @@ function find_mac()
for port in $ports; do
gids=$(ls /sys/class/infiniband/$ibdev/ports/$port/gids)
for gid in $gids; do
- first=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21-22)
+ pmac=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21-24,26-27,33-34,36-)
+ first=${pmac:0:2}
+ pmac=${pmac:2}
first=$(( first ^ 2 ))
first=$(printf "%02x" $first)
- second=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21- | sed -e 's/://g' | cut -b 3-6)
- third=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21- | sed -e 's/://g' | cut -b 11-)
- pmac=$first$second$third
+ pmac=$first$pmac
if [ x$pmac == x$1 ]; then
print_line $ibdev $port $2
fi
--
1.6.4.2
^ permalink raw reply related
* [PATCH 31/37 v2] librdmacm: provide abstracted verb calls
From: Sean Hefty @ 2010-04-08 0:41 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <27A4CFF94530491F88C8E528428FF93E-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
librdmacm: provide abstracted verb calls
From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Provide abstractions to the verb calls to simplify the user
interface for more casual verbs consumers. Users still have
access to the full range of verbs functionality by calling
verbs directly.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes from v1:
rdma_get_send/recv_comp now always return a work completion.
I decided that the get completion helpers did not need to worry about
queuing multiple events in the kernel. That condition should be rare,
but is really a kernel level issue to avoid endlessly queuing
completion events for a single CQ.
Makefile.am | 5 -
include/rdma/rdma_verbs.h | 285 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 288 insertions(+), 2 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 8d86045..8aef24a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -31,7 +31,8 @@ librdmacmincludedir = $(includedir)/rdma $(includedir)/infiniband
librdmacminclude_HEADERS = include/rdma/rdma_cma_abi.h \
include/rdma/rdma_cma.h \
- include/infiniband/ib.h
+ include/infiniband/ib.h \
+ include/rdma/rdma_verbs.h
man_MANS = \
man/rdma_accept.3 \
@@ -69,7 +70,7 @@ man_MANS = \
man/rdma_cm.7
EXTRA_DIST = include/rdma/rdma_cma_abi.h include/rdma/rdma_cma.h \
- include/infiniband/ib.h \
+ include/infiniband/ib.h include/rdma/rdma_verbs.h \
src/cma.h src/librdmacm.map librdmacm.spec.in $(man_MANS)
dist-hook: librdmacm.spec
diff --git a/include/rdma/rdma_verbs.h b/include/rdma/rdma_verbs.h
new file mode 100644
index 0000000..beeb189
--- /dev/null
+++ b/include/rdma/rdma_verbs.h
@@ -0,0 +1,285 @@
+/*
+ * Copyright (c) 2010 Intel Corporation. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#if !defined(RDMA_VERBS_H)
+#define RDMA_VERBS_H
+
+#include <assert.h>
+#include <infiniband/verbs.h>
+#include <rdma/rdma_cma.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Memory registration helpers.
+ */
+static inline struct ibv_mr *
+rdma_reg_msgs(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE);
+}
+
+static inline struct ibv_mr *
+rdma_reg_read(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE|
+ IBV_ACCESS_REMOTE_READ);
+}
+
+static inline struct ibv_mr *
+rdma_reg_write(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE |
+ IBV_ACCESS_REMOTE_WRITE);
+}
+
+static inline int
+rdma_dereg_mr(struct ibv_mr *mr)
+{
+ return ibv_dereg_mr(mr);
+}
+
+
+/*
+ * Vectored send, receive, and RDMA operations.
+ * Support multiple scatter-gather entries.
+ */
+static inline int
+rdma_post_recvv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge)
+{
+ struct ibv_recv_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+
+ return ibv_post_recv(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_sendv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_SEND;
+ wr.send_flags = flags;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_readv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_RDMA_READ;
+ wr.send_flags = flags;
+ wr.wr.rdma.remote_addr = remote_addr;
+ wr.wr.rdma.rkey = rkey;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_writev(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_RDMA_WRITE;
+ wr.send_flags = flags;
+ wr.wr.rdma.remote_addr = remote_addr;
+ wr.wr.rdma.rkey = rkey;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+/*
+ * Simple send, receive, and RDMA calls.
+ */
+static inline int
+rdma_post_recv(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr)
+{
+ struct ibv_sge sge;
+
+ assert((addr >= mr->addr) && ((addr + length) <= (mr->addr + mr->length)));
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr->lkey;
+
+ return rdma_post_recvv(id, context, &sge, 1);
+}
+
+static inline int
+rdma_post_send(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ return rdma_post_sendv(id, context, &sge, 1, flags);
+}
+
+static inline int
+rdma_post_read(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr->lkey;
+
+ return rdma_post_readv(id, context, &sge, 1, flags, remote_addr, rkey);
+}
+
+static inline int
+rdma_post_write(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ return rdma_post_writev(id, context, &sge, 1, flags, remote_addr, rkey);
+}
+
+static inline int
+rdma_post_ud_send(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ struct ibv_ah *ah, uint32_t remote_qpn)
+{
+ struct ibv_send_wr wr, *bad;
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = &sge;
+ wr.num_sge = 1;
+ wr.opcode = IBV_WR_SEND;
+ wr.send_flags = flags;
+ wr.wr.ud.ah = ah;
+ wr.wr.ud.remote_qpn = remote_qpn;
+ wr.wr.ud.remote_qkey = RDMA_UDP_QKEY;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
+{
+ struct ibv_cq *cq;
+ void *context;
+ int ret;
+
+ ret = ibv_poll_cq(id->send_cq, 1, wc);
+ if (ret)
+ return ret;
+
+ ret = ibv_req_notify_cq(id->send_cq, 0);
+ if (ret)
+ return ret;
+
+ while (!(ret = ibv_poll_cq(id->send_cq, 1, wc))) {
+ ret = ibv_get_cq_event(id->send_cq_channel, &cq, &context);
+ if (ret)
+ break;
+
+ assert(cq == id->send_cq && context == id);
+ ibv_ack_cq_events(id->send_cq, 1);
+ }
+
+ return ret;
+}
+
+static inline int
+rdma_get_recv_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
+{
+ struct ibv_cq *cq;
+ void *context;
+ int ret;
+
+ ret = ibv_poll_cq(id->recv_cq, 1, wc);
+ if (ret)
+ return ret;
+
+ ret = ibv_req_notify_cq(id->recv_cq, 0);
+ if (ret)
+ return ret;
+
+ while (!(ret = ibv_poll_cq(id->recv_cq, 1, wc))) {
+ ret = ibv_get_cq_event(id->recv_cq_channel, &cq, &context);
+ if (ret)
+ break;
+
+ assert(cq == id->recv_cq && context == id);
+ ibv_ack_cq_events(id->recv_cq, 1);
+ }
+
+ return ret;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RDMA_CMA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH v2 13/51] IB/qib: Add qib_driver.c
From: John A. Gregor @ 2010-04-07 22:58 UTC (permalink / raw)
To: swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
rdreier-FYB4Gu1CFyUAvxtiuMwx3w
Cc: ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <ada7hoj2h35.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> > Where can I find information on trace events? Something in Documentation/*?
>
> Yep, Documentation/trace/events.txt.
LWN just did a really good writeup on using the TRACE_EVENT macro:
http://lwn.net/Articles/379903/
Part 2 is still behind the paywall.
-John Gregor
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Roland Dreier @ 2010-04-07 22:37 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBD0807.10001-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Shouldn't it be __u16? These structs are part of the firmware to host
> driver/lib API.
Yes, if this header is used by userspace too then you want __u16.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Steve Wise @ 2010-04-07 22:32 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <adahbnn0wxl.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier wrote:
> The intention is that the device just treats wrid as opaque I assume so
> I think the correct fix is to go from __be16 to u16 in the structure
> declarations.
>
>
Shouldn't it be __u16? These structs are part of the firmware to host
driver/lib API.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: librdmacm meets libiwarp
From: Sean Hefty @ 2010-04-07 22:26 UTC (permalink / raw)
To: philip.frey-gy3b+zu4XSAfv37vnLkPlQ
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bernard Metzler
In-Reply-To: <942482910-1270678218-cardhu_decombobulator_blackberry.rim.net-106991531--vREwkvsugevjL2gL5RxOE1pMnUE8n+ufF6Q+nnUHFY8@public.gmane.org>
>it's nice to see, that you seem to have liked most of
>my ideas and intentions of libiwarp :) thanks for
>making that work a bit more sustainable!
Thanks for the input. I think there could still be a little more work done to
handle completions across shared CQs, plus add in SRQ support.
>I was wondering where I could get a copy of your
>latest code to look at it as a whole and (maybe)
>comment on it.
The code is available from my git tree, in the af_ib branch:
git://git.openfabrics.org/~shefty/librdmacm.git af_ib
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 26/37] librdmacm: set src_addr in rdma_getaddrinfo
From: Jason Gunthorpe @ 2010-04-07 22:22 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <569EDABAE9F84A029ED7AC84AA5767D6-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Wed, Apr 07, 2010 at 03:10:36PM -0700, Sean Hefty wrote:
> >Not sure what you mean that ACM requires it? Doesn't ACM plug in at
> >the rdma_getaddrinfo stage? If so it can get the source on its own
> >like you did in this patch. I agree that ACM should always return
> >results with the source set, because it is providing path records
> >relative to a specific adaptor.
>
> Yes - the code to set the source could move from librdmacm into ACM.
>
> I can change rdma_getaddrinfo to only set the source address if
> either the user provides one through a hint, or if resolved through
> ACM.
That would be my preference. I think the kernel calls should use a
null source address in the common case and a set source should be an
exceptional case. This matches sockets very well.
I'd see two cases for setting a source address, an app that wants to
control the bind port - this is similar to socket cases, and is
generally an exceptional case.
The other is that an app wants the connection to be usable with a
certain PD. This is more like the DAPL case, as far as I understand it
(ie resources have been allocated against a PD prior to the addresses
being known). This would be best served by having the hints include a
PD and have rdma_getaddrinfo generate a source address that works with
that PD. A PD is more general than a source address - in single HCA
cases a PD will be usable with all ports.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* librdmacm meets libiwarp
From: philip.frey-gy3b+zu4XSAfv37vnLkPlQ @ 2010-04-07 22:11 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bernard Metzler
Hi Sean,
it's nice to see, that you seem to have liked most of
my ideas and intentions of libiwarp :) thanks for
making that work a bit more sustainable!
I was wondering where I could get a copy of your
latest code to look at it as a whole and (maybe)
comment on it.
Thanks,
Philip
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 26/37] librdmacm: set src_addr in rdma_getaddrinfo
From: Sean Hefty @ 2010-04-07 22:10 UTC (permalink / raw)
To: 'Jason Gunthorpe'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407201417.GJ15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>I don't have a problem with ai_src_addr being set, when necessary, but
>setting it unconditionally seems wrong to me. In most cases the kernel
>should select the source during route resolution, not be forced to
>something in userspace.
Just to be precise, the source is selected during address resolution, and the
existing APIs allow the user to indicate that a specific source should be used.
This is a requirement of some applications.
>Not sure what you mean that ACM requires it? Doesn't ACM plug in at
>the rdma_getaddrinfo stage? If so it can get the source on its own
>like you did in this patch. I agree that ACM should always return
>results with the source set, because it is providing path records
>relative to a specific adaptor.
Yes - the code to set the source could move from librdmacm into ACM.
I can change rdma_getaddrinfo to only set the source address if either the user
provides one through a hint, or if resolved through ACM.
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 31/37] librdmacm: provide abstracted verb calls
From: Steve Wise @ 2010-04-07 22:07 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <3F9969F0569A431A9B40B8DED7E18FB4-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
Sean Hefty wrote:
>>> I think it's possible for the function to return without having filled in a
>>>
>> wc.
>>
>> So its busted? Or is this intended behavior?
>>
>
> Depends on the point of view, I guess. :) It would be nice to avoid that
> situation.
>
>
>> Is rdma_get_send_completion() supposed to return exactly one wc? If so
>> then the 2 polls can cause a wc to get silently discarded. I must
>> still not be understanding the intended use?
>>
>
> How can a wc get discarded? Maybe the return code from ibv_poll_cq is confusing
> you? If the first poll finds a wc, ibv_poll_cq returns 1, and we exit the
> function. Otherwise, we rearm the cq, then poll again to make sure that nothing
> got missed.
>
>
Right. I missed that. poll will return 1 if there's a completion
returned. Nevermind :)
>> I would think this should just be:
>>
>> get_cq_event()
>> notify_cq()
>> poll()
>>
>
> This requires arming the CQ up front. I was also trying to avoid the overhead
> of always calling get_cq_event and notify_cq to just pull a completed request
> off of the work queue.
>
>
I was confused on the poll_cq return code (and I've been working in this
code for umpteen years :) ).
>>> The only drawback I see is that it's theoretically possible to build up a
>>>
>> queue
>>
>>> of cq events in the kernel. Not sure how to fix that. Any ideas?
>>>
>>>
>> That can always happen, yes?
>>
>
> It seems like it should be avoidable. Maybe 1 event can queue up, but I think
> we can prevent more by not rearming until that event gets pulled.
>
> If nothing else, I think this discussion shows why we need this sort of wrapper.
> :)
>
Indeed! I like the wrappers.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 31/37] librdmacm: provide abstracted verb calls
From: Sean Hefty @ 2010-04-07 22:01 UTC (permalink / raw)
To: 'Steve Wise'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBCED37.9040708-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
>> I think it's possible for the function to return without having filled in a
>wc.
>>
>
>So its busted? Or is this intended behavior?
Depends on the point of view, I guess. :) It would be nice to avoid that
situation.
>Is rdma_get_send_completion() supposed to return exactly one wc? If so
>then the 2 polls can cause a wc to get silently discarded. I must
>still not be understanding the intended use?
How can a wc get discarded? Maybe the return code from ibv_poll_cq is confusing
you? If the first poll finds a wc, ibv_poll_cq returns 1, and we exit the
function. Otherwise, we rearm the cq, then poll again to make sure that nothing
got missed.
>I would think this should just be:
>
>get_cq_event()
>notify_cq()
>poll()
This requires arming the CQ up front. I was also trying to avoid the overhead
of always calling get_cq_event and notify_cq to just pull a completed request
off of the work queue.
>> The only drawback I see is that it's theoretically possible to build up a
>queue
>> of cq events in the kernel. Not sure how to fix that. Any ideas?
>>
>That can always happen, yes?
It seems like it should be avoidable. Maybe 1 event can queue up, but I think
we can prevent more by not rearming until that event gets pulled.
If nothing else, I think this discussion shows why we need this sort of wrapper.
:)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 05/10] iw_cxgb4: Add connection management functions.
From: Roland Dreier @ 2010-04-07 21:55 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBCFDF9.70306-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Both iw_cxgb3 and iw_cxgb4 load ok concurrently when compiled as modules.
Oh, right. The peer2peer symbol isn't exported so the clash is only if
you try to build them both into the kernel (as I often do as part of my
quick build tests).
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Steve Wise @ 2010-04-07 21:50 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <adahbnn0wxl.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier wrote:
> You have:
>
> > +struct fw_ri_send_wr {
> ...
> > + __be16 wrid;
>
> > +struct fw_ri_recv_wr {
> ...
> > + __be16 wrid;
>
> But also:
>
> > +static inline void init_wr_hdr(union t4_wr *wqe, u16 wrid,
> > + enum fw_wr_opcodes opcode, u8 flags, u8 len16)
> ...
> > + wqe->send.wrid = wrid;
>
> and similar for recv.wrid in qp.c. sparse correctly warns about this
> endianness clash.
>
> The intention is that the device just treats wrid as opaque I assume so
> I think the correct fix is to go from __be16 to u16 in the structure
> declarations.
>
> - R.
>
Yes, it should be a u16 in the wr structs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 05/10] iw_cxgb4: Add connection management functions.
From: Steve Wise @ 2010-04-07 21:49 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <adaljcz0x0g.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier wrote:
> > +int peer2peer = 0;
> > +module_param(peer2peer, int, 0644);
> > +MODULE_PARM_DESC(peer2peer, "Support peer2peer ULPs (default=0)");
>
> If you build iw_cxgb3 and iw_cxgb4 into the kernel, the peer2peer symbol
> names clash. (Same problem occurs if you try to load cxgb3 and cxgb4
> modules at the same time, I think).
Both iw_cxgb3 and iw_cxgb4 load ok concurrently when compiled as modules.
The option was originally intended to be used in more than just cm.c.
So there's a piece of code missing in qp.c. I'll clean this up. I
might make an attribute in c4iw_endpoint that indicates this mode. Then
the qp code won't need the global option and can key off the endpoint
attribute. So I can make this a static as you suggest.
> We can fix it here in cxgb4 by just
> making peer2peer static (and deleting the extern declaration).
>
> However peer2peer is not that great of a name for a global symbol; might
> be good to add a patch to cxgb3 to rename peer2peer to something like
> iwch_peer2peer and using module_param_named()...
>
I'll do this for cxgb3.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 06/10] iw_cxgb4: Add memory management functions.
From: Roland Dreier @ 2010-04-07 21:40 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100402192419.14117.40759.stgit-T4OLL4TyM9aNDNWfRnPdfg@public.gmane.org>
> + req->wr.wr_lo = (u64)&wr_wait;
wr_lo is __be64. The cast should be to __force __be64 here I think.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox