* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Steve Wise @ 2010-04-07 22:32 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <adahbnn0wxl.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier wrote:
> The intention is that the device just treats wrid as opaque I assume so
> I think the correct fix is to go from __be16 to u16 in the structure
> declarations.
>
>
Shouldn't it be __u16? These structs are part of the firmware to host
driver/lib API.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH 02/10] iw_cxgb4: Add driver, fw, and hw headers.
From: Roland Dreier @ 2010-04-07 22:37 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BBD0807.10001-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
> Shouldn't it be __u16? These structs are part of the firmware to host
> driver/lib API.
Yes, if this header is used by userspace too then you want __u16.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 13/51] IB/qib: Add qib_driver.c
From: John A. Gregor @ 2010-04-07 22:58 UTC (permalink / raw)
To: swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW,
rdreier-FYB4Gu1CFyUAvxtiuMwx3w
Cc: ralph.campbell-h88ZbnxC6KDQT0dZR+AlfA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <ada7hoj2h35.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> > Where can I find information on trace events? Something in Documentation/*?
>
> Yep, Documentation/trace/events.txt.
LWN just did a really good writeup on using the TRACE_EVENT macro:
http://lwn.net/Articles/379903/
Part 2 is still behind the paywall.
-John Gregor
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 31/37 v2] librdmacm: provide abstracted verb calls
From: Sean Hefty @ 2010-04-08 0:41 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <27A4CFF94530491F88C8E528428FF93E-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
librdmacm: provide abstracted verb calls
From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Provide abstractions to the verb calls to simplify the user
interface for more casual verbs consumers. Users still have
access to the full range of verbs functionality by calling
verbs directly.
Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes from v1:
rdma_get_send/recv_comp now always return a work completion.
I decided that the get completion helpers did not need to worry about
queuing multiple events in the kernel. That condition should be rare,
but is really a kernel level issue to avoid endlessly queuing
completion events for a single CQ.
Makefile.am | 5 -
include/rdma/rdma_verbs.h | 285 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 288 insertions(+), 2 deletions(-)
diff --git a/Makefile.am b/Makefile.am
index 8d86045..8aef24a 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -31,7 +31,8 @@ librdmacmincludedir = $(includedir)/rdma $(includedir)/infiniband
librdmacminclude_HEADERS = include/rdma/rdma_cma_abi.h \
include/rdma/rdma_cma.h \
- include/infiniband/ib.h
+ include/infiniband/ib.h \
+ include/rdma/rdma_verbs.h
man_MANS = \
man/rdma_accept.3 \
@@ -69,7 +70,7 @@ man_MANS = \
man/rdma_cm.7
EXTRA_DIST = include/rdma/rdma_cma_abi.h include/rdma/rdma_cma.h \
- include/infiniband/ib.h \
+ include/infiniband/ib.h include/rdma/rdma_verbs.h \
src/cma.h src/librdmacm.map librdmacm.spec.in $(man_MANS)
dist-hook: librdmacm.spec
diff --git a/include/rdma/rdma_verbs.h b/include/rdma/rdma_verbs.h
new file mode 100644
index 0000000..beeb189
--- /dev/null
+++ b/include/rdma/rdma_verbs.h
@@ -0,0 +1,285 @@
+/*
+ * Copyright (c) 2010 Intel Corporation. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#if !defined(RDMA_VERBS_H)
+#define RDMA_VERBS_H
+
+#include <assert.h>
+#include <infiniband/verbs.h>
+#include <rdma/rdma_cma.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/*
+ * Memory registration helpers.
+ */
+static inline struct ibv_mr *
+rdma_reg_msgs(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE);
+}
+
+static inline struct ibv_mr *
+rdma_reg_read(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE|
+ IBV_ACCESS_REMOTE_READ);
+}
+
+static inline struct ibv_mr *
+rdma_reg_write(struct rdma_cm_id *id, void *addr, size_t length)
+{
+ return ibv_reg_mr(id->qp->pd, addr, length, IBV_ACCESS_LOCAL_WRITE |
+ IBV_ACCESS_REMOTE_WRITE);
+}
+
+static inline int
+rdma_dereg_mr(struct ibv_mr *mr)
+{
+ return ibv_dereg_mr(mr);
+}
+
+
+/*
+ * Vectored send, receive, and RDMA operations.
+ * Support multiple scatter-gather entries.
+ */
+static inline int
+rdma_post_recvv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge)
+{
+ struct ibv_recv_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+
+ return ibv_post_recv(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_sendv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_SEND;
+ wr.send_flags = flags;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_readv(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_RDMA_READ;
+ wr.send_flags = flags;
+ wr.wr.rdma.remote_addr = remote_addr;
+ wr.wr.rdma.rkey = rkey;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_post_writev(struct rdma_cm_id *id, void *context, struct ibv_sge *sgl,
+ int nsge, int flags, uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_send_wr wr, *bad;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = sgl;
+ wr.num_sge = nsge;
+ wr.opcode = IBV_WR_RDMA_WRITE;
+ wr.send_flags = flags;
+ wr.wr.rdma.remote_addr = remote_addr;
+ wr.wr.rdma.rkey = rkey;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+/*
+ * Simple send, receive, and RDMA calls.
+ */
+static inline int
+rdma_post_recv(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr)
+{
+ struct ibv_sge sge;
+
+ assert((addr >= mr->addr) && ((addr + length) <= (mr->addr + mr->length)));
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr->lkey;
+
+ return rdma_post_recvv(id, context, &sge, 1);
+}
+
+static inline int
+rdma_post_send(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ return rdma_post_sendv(id, context, &sge, 1, flags);
+}
+
+static inline int
+rdma_post_read(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr->lkey;
+
+ return rdma_post_readv(id, context, &sge, 1, flags, remote_addr, rkey);
+}
+
+static inline int
+rdma_post_write(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ uint64_t remote_addr, uint32_t rkey)
+{
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ return rdma_post_writev(id, context, &sge, 1, flags, remote_addr, rkey);
+}
+
+static inline int
+rdma_post_ud_send(struct rdma_cm_id *id, void *context, void *addr,
+ size_t length, struct ibv_mr *mr, int flags,
+ struct ibv_ah *ah, uint32_t remote_qpn)
+{
+ struct ibv_send_wr wr, *bad;
+ struct ibv_sge sge;
+
+ sge.addr = (uint64_t) addr;
+ sge.length = (uint32_t) length;
+ sge.lkey = mr ? mr->lkey : 0;
+
+ wr.wr_id = (uintptr_t) context;
+ wr.next = NULL;
+ wr.sg_list = &sge;
+ wr.num_sge = 1;
+ wr.opcode = IBV_WR_SEND;
+ wr.send_flags = flags;
+ wr.wr.ud.ah = ah;
+ wr.wr.ud.remote_qpn = remote_qpn;
+ wr.wr.ud.remote_qkey = RDMA_UDP_QKEY;
+
+ return ibv_post_send(id->qp, &wr, &bad);
+}
+
+static inline int
+rdma_get_send_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
+{
+ struct ibv_cq *cq;
+ void *context;
+ int ret;
+
+ ret = ibv_poll_cq(id->send_cq, 1, wc);
+ if (ret)
+ return ret;
+
+ ret = ibv_req_notify_cq(id->send_cq, 0);
+ if (ret)
+ return ret;
+
+ while (!(ret = ibv_poll_cq(id->send_cq, 1, wc))) {
+ ret = ibv_get_cq_event(id->send_cq_channel, &cq, &context);
+ if (ret)
+ break;
+
+ assert(cq == id->send_cq && context == id);
+ ibv_ack_cq_events(id->send_cq, 1);
+ }
+
+ return ret;
+}
+
+static inline int
+rdma_get_recv_comp(struct rdma_cm_id *id, struct ibv_wc *wc)
+{
+ struct ibv_cq *cq;
+ void *context;
+ int ret;
+
+ ret = ibv_poll_cq(id->recv_cq, 1, wc);
+ if (ret)
+ return ret;
+
+ ret = ibv_req_notify_cq(id->recv_cq, 0);
+ if (ret)
+ return ret;
+
+ while (!(ret = ibv_poll_cq(id->recv_cq, 1, wc))) {
+ ret = ibv_get_cq_event(id->recv_cq_channel, &cq, &context);
+ if (ret)
+ break;
+
+ assert(cq == id->recv_cq && context == id);
+ ibv_ack_cq_events(id->recv_cq, 1);
+ }
+
+ return ret;
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* RDMA_CMA_H */
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* ibdev2netdev optimization [ PATCH 1/1 ]
From: Aleksey Senin @ 2010-04-08 9:22 UTC (permalink / raw)
To: linux-rdma
[-- Attachment #1: Type: text/plain, Size: 70 bytes --]
Call just once all these sed/awk/cut etc. when finding mac addresses.
[-- Attachment #2: 0001-Optimized-when-looking-ethernet-data.patch --]
[-- Type: text/x-patch, Size: 1443 bytes --]
From 2437e980e6a8f0bcd123ddab077f00908069d379 Mon Sep 17 00:00:00 2001
From: Aleksey Senin <alekseys@voltaire.com>
Date: Thu, 8 Apr 2010 11:53:07 +0300
Subject: [PATCH] Optimized when looking ethernet data.
I have two adapters with two ports on each one. This optimization gives me
6 sec. run-time instead of 19 sec.
Signed-off-by: Aleksey Senin <alekseys@voltaire.com>
---
ofed_scripts/ibdev2netdev | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/ofed_scripts/ibdev2netdev b/ofed_scripts/ibdev2netdev
index b436c05..adb5eba 100755
--- a/ofed_scripts/ibdev2netdev
+++ b/ofed_scripts/ibdev2netdev
@@ -30,12 +30,12 @@ function find_mac()
for port in $ports; do
gids=$(ls /sys/class/infiniband/$ibdev/ports/$port/gids)
for gid in $gids; do
- first=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21-22)
+ pmac=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21-24,26-27,33-34,36-)
+ first=${pmac:0:2}
+ pmac=${pmac:2}
first=$(( first ^ 2 ))
first=$(printf "%02x" $first)
- second=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21- | sed -e 's/://g' | cut -b 3-6)
- third=$(cat /sys/class/infiniband/$ibdev/ports/$port/gids/$gid | cut -b 21- | sed -e 's/://g' | cut -b 11-)
- pmac=$first$second$third
+ pmac=$first$pmac
if [ x$pmac == x$1 ]; then
print_line $ibdev $port $2
fi
--
1.6.4.2
^ permalink raw reply related
* Re: [patch] infiniband: checking the wrong variable
From: Bart Van Assche @ 2010-04-08 10:13 UTC (permalink / raw)
To: Dan Carpenter
Cc: Roland Dreier, Sean Hefty, Hal Rosenstock, Tejun Heo,
Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407093901.GG5157@bicker>
On Wed, Apr 7, 2010 at 11:39 AM, Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> The intent here was to check the "mfrpl->mapped_page_list" allocation.
> We checked "mfrpl->ibfrpl.page_list" earlier.
>
> Signed-off-by: Dan Carpenter <error27-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>
> diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
> index 56147b2..1d27b9a 100644
> --- a/drivers/infiniband/hw/mlx4/mr.c
> +++ b/drivers/infiniband/hw/mlx4/mr.c
> @@ -240,7 +240,7 @@ struct ib_fast_reg_page_list *mlx4_ib_alloc_fast_reg_page_list(struct ib_device
> mfrpl->mapped_page_list = dma_alloc_coherent(&dev->dev->pdev->dev,
> size, &mfrpl->map,
> GFP_KERNEL);
> - if (!mfrpl->ibfrpl.page_list)
> + if (!mfrpl->mapped_page_list)
> goto err_free;
>
> WARN_ON(mfrpl->map & 0x3f);
Hello Dan,
Have you already had the time to check whether this fix should be
backported to already released kernels, and if so, to which kernel
versions ? The easiest way to ask for backporting is to add "Cc:
<stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] opensm/osm_sa.c: enhance opensm_dump_to_file() to accept full path file name
From: Yevgeny Kliteynik @ 2010-04-08 13:14 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA
Hi Sasha,
Small patch that enhances opensm_dump_to_file() function
and allows it to accept full path file names as argument.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_sa.c | 9 +++++++--
1 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c
index 3473e4c..8aab548 100644
--- a/opensm/opensm/osm_sa.c
+++ b/opensm/opensm/osm_sa.c
@@ -510,8 +510,13 @@ opensm_dump_to_file(osm_opensm_t * p_osm, const char *file_name,
char path[1024];
FILE *file;
- snprintf(path, sizeof(path), "%s/%s",
- p_osm->subn.opt.dump_files_dir, file_name);
+ if (*file_name == '/')
+ /* file name was provided as an absolute path */
+ snprintf(path, sizeof(path), "%s", file_name);
+ else
+ /* file name is relative to dump_files_dir */
+ snprintf(path, sizeof(path), "%s/%s",
+ p_osm->subn.opt.dump_files_dir, file_name);
file = fopen(path, "w");
if (!file) {
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 0/4] opensm: dump PathRecords file
From: Yevgeny Kliteynik @ 2010-04-08 13:28 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Hi Sasha,
One of the major bottlenecks of the OpenSM is handling
SA queries storm. In order to allow all-2-all communication,
SA must process N^2 PathRecord queries.
On large clusters this takes way too much time.
Currently, most of the MPI implementations do not query SA
at all in order to avoid this problem, but use predefined
parameters when communicating between fabric nodes.
This works fine as long as MPI doesn't need to use different
parameters for different path, but it's a problem when we're
talking about non-trivial QoS settings or routing engines that
are using IB VLs (such as Torus-2QoS).
The following patch series enables OpenSM to dump core
information from PathRecords that is needed for opening
communication channel: SL, MTU and Rate.
This information is dumped for for all the non-switch-2-non-switch
paths in the subnet in the following way:
for every non-switch source port
for every non-switch target LID in the subnet
dump PR between source port and target LID
This way number of sources is equal to number of physical
non-switch ports in the subnet, and only number of targets
depends on LMC that is used.
Patches:
[PATCH 1/4] opensm: added function that dumps PathRecords
[PATCH 2/4] opensm: added 2 options: dump PRs and filename
[PATCH 3/4] opensm: dump PRs after every heavy sweep and
after reroute
[PATCH 4/4] opensm: add command line argument to dump PR file
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 1/4] opensm: added function that dumps PathRecords
From: Yevgeny Kliteynik @ 2010-04-08 13:29 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Dumping SL, MTU and Rate for all the
non-switch-2-non-switch paths in the subnet.
PRs that are dumped:
for every non-switch source port
for every non-switch target LID in the subnet
dump PR between source port and target LID
This way number of sources is equal to number of physical
non-switch ports in the subnet, and only number of targets
depends on LMC that is used.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_sa.h | 21 +++++++++
opensm/opensm/osm_sa.c | 81 ++++++++++++++++++++++++++++++++++++
opensm/opensm/osm_sa_path_record.c | 12 +++++
3 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/opensm/include/opensm/osm_sa.h b/opensm/include/opensm/osm_sa.h
index d516310..40622ff 100644
--- a/opensm/include/opensm/osm_sa.h
+++ b/opensm/include/opensm/osm_sa.h
@@ -462,6 +462,27 @@ int osm_sa_db_file_load(struct osm_opensm *p_osm);
*
*********/
+/****f* OpenSM: SA/osm_sa_path_records_file_dump
+* NAME
+* osm_sa_path_records_file_dump
+*
+* DESCRIPTION
+* Dumps the SA Path Records to the dump file.
+*
+* SYNOPSIS
+*/
+int osm_sa_path_records_file_dump(struct osm_opensm *p_osm);
+/*
+* PARAMETERS
+* p_osm
+* [in] Pointer to an osm_opensm_t object.
+*
+* RETURN VALUES
+* 0 on success
+* !0 if some error occurred.
+*
+*********/
+
/****f* OpenSM: MC Member Record Receiver/osm_mcmr_rcv_find_or_create_new_mgrp
* NAME
* osm_mcmr_rcv_find_or_create_new_mgrp
diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c
index 8aab548..83da258 100644
--- a/opensm/opensm/osm_sa.c
+++ b/opensm/opensm/osm_sa.c
@@ -718,6 +718,87 @@ int osm_sa_db_file_dump(osm_opensm_t * p_osm)
return res;
}
+typedef struct _path_parms {
+ ib_net16_t pkey;
+ uint8_t mtu;
+ uint8_t rate;
+ uint8_t sl;
+ uint8_t pkt_life;
+ boolean_t reversible;
+} path_parms_t;
+
+extern ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
+ IN const osm_port_t * p_src_port,
+ IN const osm_port_t * p_dest_port,
+ IN const uint16_t dlid_ho,
+ OUT path_parms_t * p_parms);
+
+static void sa_dump_path_records(osm_opensm_t * p_osm, FILE * file)
+{
+ osm_port_t *p_src_port;
+ osm_port_t *p_dest_port;
+ osm_node_t *p_node;
+ uint16_t dlid_ho;
+ uint32_t vector_size;
+ osm_physp_t *p_physp;
+ path_parms_t path_parms;
+ ib_api_status_t status;
+
+ vector_size = cl_ptr_vector_get_size(&p_osm->subn.port_lid_tbl);
+ for (p_src_port = (osm_port_t *) cl_qmap_head(&p_osm->subn.port_guid_tbl);
+ p_src_port != (osm_port_t *) cl_qmap_end(&p_osm->subn.port_guid_tbl);
+ p_src_port = (osm_port_t *) cl_qmap_next(&p_src_port->map_item)) {
+
+ p_node = p_src_port->p_node;
+ if (p_node->node_info.node_type == IB_NODE_TYPE_SWITCH)
+ return;
+
+ p_physp = p_src_port->p_physp;
+ CL_ASSERT(p_physp->p_remote_physp);
+
+ fprintf(file, "%s 0x%016" PRIx64 ", base LID %d, "
+ "\"%s\", port %d\n# LID : SL : MTU : RATE\n",
+ ib_get_node_type_str(p_node->node_info.node_type),
+ cl_ntoh64(p_src_port->guid),
+ cl_ntoh16(osm_port_get_base_lid(p_src_port)),
+ p_node->print_desc, p_physp->port_num);
+
+ memset(&path_parms, 0, sizeof(path_parms_t));
+
+ for (dlid_ho = 1; dlid_ho < vector_size; dlid_ho++) {
+
+ p_dest_port = (osm_port_t *) cl_ptr_vector_get(
+ &p_osm->subn.port_lid_tbl, dlid_ho);
+
+ if (!p_dest_port || !p_dest_port->p_node ||
+ p_dest_port->p_node->node_info.node_type ==
+ IB_NODE_TYPE_SWITCH)
+ continue;
+
+ status = osm_get_path_params(&p_osm->sa,
+ p_src_port, p_dest_port, dlid_ho,
+ (void*)&path_parms);
+
+ if (!status)
+ fprintf(file, "0x%04X : %-2d : %-3d : %-4d\n",
+ dlid_ho, path_parms.sl,
+ path_parms.mtu, path_parms.rate);
+ else
+ fprintf(file, "0x%04X : UNREACHABLE\n",
+ dlid_ho);
+ }
+ fprintf(file, "\n");
+ }
+}
+
+
+int osm_sa_path_records_file_dump(osm_opensm_t * p_osm)
+{
+ return opensm_dump_to_file(p_osm, p_osm->subn.opt.path_rec_file ?
+ p_osm->subn.opt.path_rec_file : "opensm-path-records.dump",
+ sa_dump_path_records);
+}
+
/*
* SA DB Loader
*/
diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c
index c4c3f86..b168428 100644
--- a/opensm/opensm/osm_sa_path_record.c
+++ b/opensm/opensm/osm_sa_path_record.c
@@ -743,6 +743,18 @@ Exit:
return status;
}
+ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
+ IN const osm_port_t * p_src_port,
+ IN const osm_port_t * p_dest_port,
+ IN const uint16_t dlid_ho,
+ OUT osm_path_parms_t * p_parms)
+{
+ ib_path_rec_t pr;
+ memset(&pr, 0, sizeof(ib_path_rec_t));
+ return pr_rcv_get_path_parms(sa, &pr,
+ p_src_port, p_dest_port, dlid_ho, 0, p_parms);
+}
+
static void pr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t * p_src_port,
IN const osm_port_t * p_dest_port,
IN const ib_gid_t * p_dgid,
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 2/4] opensm: added 2 options: dump PRs and filename
From: Yevgeny Kliteynik @ 2010-04-08 13:29 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_subnet.h | 10 ++++++++++
opensm/opensm/osm_subnet.c | 14 ++++++++++++++
2 files changed, 24 insertions(+), 0 deletions(-)
diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h
index 3970e98..8d7182b 100644
--- a/opensm/include/opensm/osm_subnet.h
+++ b/opensm/include/opensm/osm_subnet.h
@@ -201,6 +201,8 @@ typedef struct osm_subn_opt {
char *guid_routing_order_file;
char *sa_db_file;
boolean_t sa_db_dump;
+ boolean_t path_rec_dump;
+ char *path_rec_file;
boolean_t do_mesh_analysis;
boolean_t exit_on_fatal;
boolean_t honor_guid2lid_file;
@@ -418,6 +420,14 @@ typedef struct osm_subn_opt {
* When TRUE causes OpenSM to dump SA DB at the end of every
* light sweep regardless the current verbosity level.
*
+* path_rec_file
+* Name of the PathRecords dump file.
+*
+* path_rec_dump
+* When TRUE causes OpenSM to dump PathRecords for all the
+* CA-2-CA paths at the end of every heavy sweep, regardless
+* of the verbosity level.
+*
* exit_on_fatal
* If TRUE (default) - SM will exit on fatal subnet initialization
* issues.
diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c
index e4126bc..c87ed2a 100644
--- a/opensm/opensm/osm_subnet.c
+++ b/opensm/opensm/osm_subnet.c
@@ -757,6 +757,8 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt)
p_opt->guid_routing_order_file = NULL;
p_opt->sa_db_file = NULL;
p_opt->sa_db_dump = FALSE;
+ p_opt->path_rec_file = NULL;
+ p_opt->path_rec_dump = FALSE;
p_opt->do_mesh_analysis = FALSE;
p_opt->exit_on_fatal = TRUE;
p_opt->enable_quirks = FALSE;
@@ -1464,6 +1466,18 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)
p_opts->sa_db_dump ? "TRUE" : "FALSE");
fprintf(out,
+ "# PathRecords dump file name\n"
+ "path_rec_file %s\n\n",
+ p_opts->path_rec_file ? p_opts->path_rec_file : null_str);
+
+ fprintf(out,
+ "# If TRUE causes OpenSM to dump PathRecords for all the\n"
+ "# CA-2-CA paths at the end of every heavy sweep, regardless\n"
+ "# of the verbosity level\n"
+ "path_rec_dump %s\n\n",
+ p_opts->path_rec_dump ? "TRUE" : "FALSE");
+
+ fprintf(out,
"#\n# HANDOVER - MULTIPLE SMs OPTIONS\n#\n"
"# SM priority used for deciding who is the master\n"
"# Range goes from 0 (lowest priority) to 15 (highest).\n"
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 3/4] opensm: dump PRs after every heavy sweep and after reroute
From: Yevgeny Kliteynik @ 2010-04-08 13:30 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_dump.c | 3 +++
opensm/opensm/osm_state_mgr.c | 2 ++
2 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c
index 86e9c00..cedff24 100644
--- a/opensm/opensm/osm_dump.c
+++ b/opensm/opensm/osm_dump.c
@@ -634,6 +634,9 @@ void osm_dump_all(osm_opensm_t * osm)
osm_dump_qmap_to_file(osm, "opensm-subnet.lst",
&osm->subn.node_guid_tbl, dump_topology_node,
osm);
+ if (osm->subn.opt.path_rec_dump)
+ osm_sa_path_records_file_dump(osm);
+
if (osm_log_is_active(&osm->log, OSM_LOG_VERBOSE))
print_report(osm, stdout);
}
diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c
index e43463f..e32ff18 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1149,6 +1149,8 @@ static void do_sweep(osm_sm_t * sm)
return;
if (!sm->p_subn->subnet_initialization_error) {
+ if (sm->p_subn->opt.path_rec_dump)
+ osm_sa_path_records_file_dump(sm->p_subn->p_osm);
OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE,
"REROUTE COMPLETE");
return;
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH 4/4] opensm: add command line argument to dump PR file
From: Yevgeny Kliteynik @ 2010-04-08 13:30 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA, Jim Schut
Added command line argument to dump PathRecord
file and updated the man page.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/man/opensm.8.in | 8 +++++++-
opensm/opensm/main.c | 9 +++++++++
2 files changed, 16 insertions(+), 1 deletions(-)
diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in
index 7aca8f9..8d3ce97 100644
--- a/opensm/man/opensm.8.in
+++ b/opensm/man/opensm.8.in
@@ -1,4 +1,4 @@
-.TH OPENSM 8 "October 22, 2009" "OpenIB" "OpenIB Management"
+.TH OPENSM 8 "April 8, 2010" "OpenIB" "OpenIB Management"
.SH NAME
opensm \- InfiniBand subnet manager and administration (SM/SA)
@@ -28,6 +28,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA)
[\-H | \-\-max_reverse_hops <max reverse hops allowed>]
[\-X | \-\-guid_routing_order_file <path to file>]
[\-m | \-\-ids_guid_file <path to file>]
+[\-\-dump_path_records]
[\-o(nce)]
[\-s(weep) <interval>]
[\-t(imeout) <milliseconds>]
@@ -218,6 +219,11 @@ Set the order port guids will be routed for the MinHop
and Up/Down routing algorithms to the guids provided in the
given file (one to a line).
.TP
+\fB\-\-dump_path_records\fR
+This option causes OpenSM to dump Path Records file.
+This file can be used by applications such as MPI to
+eliminate the need to query SA.
+.TP
\fB\-o\fR, \fB\-\-once\fR
This option causes OpenSM to configure the subnet
once, then exit. Ports remain in the ACTIVE state.
diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c
index f9a33af..9d5c886 100644
--- a/opensm/opensm/main.c
+++ b/opensm/opensm/main.c
@@ -231,6 +231,10 @@ static void show_usage(void)
" Set the order port guids will be routed for the MinHop\n"
" and Up/Down routing algorithms to the guids provided in the\n"
" given file (one to a line)\n\n");
+ printf("--dump_path_records\n"
+ " This option causes OpenSM to dump Path Records file.\n"
+ " This file can be used by applications such as MPI to\n"
+ " eliminate the need to query SA.\n\n");
printf("--once, -o\n"
" This option causes OpenSM to configure the subnet\n"
" once, then exit. Ports remain in the ACTIVE state.\n\n");
@@ -610,6 +614,7 @@ int main(int argc, char *argv[])
{"sm_sl", 1, NULL, 7},
{"retries", 1, NULL, 8},
{"log_prefix", 1, NULL, 9},
+ {"dump_path_records", 0, NULL, 10},
{NULL, 0, NULL, 0} /* Required at the end of the array */
};
@@ -992,6 +997,10 @@ int main(int argc, char *argv[])
SET_STR_OPT(opt.log_prefix, optarg);
printf("Log prefix = %s\n", opt.log_prefix);
break;
+ case 10:
+ opt.path_rec_dump = TRUE;
+ printf(" Dumping Path Records file\n");
+ break;
case 'h':
case '?':
case ':':
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-08 14:46 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <g2ge2e108261004080313t32c2e1besd8a95e2f9bf418f9-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> Have you already had the time to check whether this fix should be
> backported to already released kernels, and if so, to which kernel
> versions ? The easiest way to ask for backporting is to add "Cc:
> <stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
It backports but I don't see it as fixing a big enough problem to be
worth it for stable.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch] infiniband: checking the wrong variable
From: Bart Van Assche @ 2010-04-08 15:07 UTC (permalink / raw)
To: Roland Dreier
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <ada7hoixaqw.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, Apr 8, 2010 at 4:46 PM, Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
>
> > Have you already had the time to check whether this fix should be
> > backported to already released kernels, and if so, to which kernel
> > versions ? The easiest way to ask for backporting is to add "Cc:
> > <stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>" in a patch.
>
> It backports but I don't see it as fixing a big enough problem to be
> worth it for stable.
It would be great if someone could explain the impact of this issue --
the impact is not yet clear to me.
Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [patch] infiniband: checking the wrong variable
From: Roland Dreier @ 2010-04-08 16:00 UTC (permalink / raw)
To: Bart Van Assche
Cc: Dan Carpenter, Roland Dreier, Sean Hefty, Hal Rosenstock,
Tejun Heo, Christoph Lameter, Jack Morgenstein,
linux-rdma-u79uwXL29TY76Z2rM5mHXA,
kernel-janitors-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <r2le2e108261004080807i36ad0755gad9843224b0c42fc-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
> It would be great if someone could explain the impact of this issue --
> the impact is not yet clear to me.
Um, it's a one-line patch -- pretty clear if the allocation fails then
we check the wrong value and probably crash.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: [PATCH 11/37] librdmacm: add zalloc call
From: Sean Hefty @ 2010-04-08 16:14 UTC (permalink / raw)
To: 'Jason Gunthorpe'; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20100407191938.GF15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
>Maybe use calloc(,1 sizeof(x)); instead?
yes - I'll update the patches, but won't repost for this change.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* IB_CQ_VECTOR_LEAST_ATTACHED in mainline?
From: Andy Grover @ 2010-04-08 19:50 UTC (permalink / raw)
To: Yevgeny Petrilin, linux-rdma
Hi Yevgeny,
When will this be targeted for mainline?
I'm working on using multiple vectors, and it would be great to see it
there, since I need to work against both OFED and mainline. Otherwise I
need to have my ulp do round robin assignment up to
caps.num_comp_vectors, I guess? VECTOR_LEAST_ATTACHED seems better.
Thanks -- Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 12/51] IB/qib: Add qib_dma.c
From: Ralph Campbell @ 2010-04-08 20:40 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adaljcz43n9.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
OK. This will be in v3.
On Wed, 2010-03-31 at 16:18 -0700, Roland Dreier wrote:
> > +struct ib_dma_mapping_ops qib_dma_mapping_ops = {
> > + qib_mapping_error,
> > + qib_dma_map_single,
> > + qib_dma_unmap_single,
> > + qib_dma_map_page,
> > + qib_dma_unmap_page,
> > + qib_map_sg,
> > + qib_unmap_sg,
> > + qib_sg_dma_address,
> > + qib_sg_dma_len,
> > + qib_sync_single_for_cpu,
> > + qib_sync_single_for_device,
> > + qib_dma_alloc_coherent,
> > + qib_dma_free_coherent
> > +};
>
> I think it would be better to use designated initializers (".xxx = yyy,")
> here to avoid problems if this struct ever changes layout.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] IB/core: allow HCAs to create IB port sysfs files
From: Roland Dreier @ 2010-04-08 20:49 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma
In-Reply-To: <1258486848.992.407.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
Rather than have the device driver call a new function that then calls
it back, how about if we just add a parameter to the
ib_register_device() call that is the callback to add the sysfs stuff?
This would let us avoid the race where the device is registered but not
all the sysfs stuff is created yet.
- R.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Roland Dreier @ 2010-04-08 20:50 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <20091203190624.29507.22110.stgit-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
I didn't audit everything (so there may be more problems here too), but
certainly this violates the "one value per sysfs file" rule rather
egregiously:
> +static ssize_t show_stats(struct device *device, struct device_attribute *attr,
> + char *buf)
> +{
> + struct qib_ibdev *dev =
> + container_of(device, struct qib_ibdev, ibdev.dev);
> + struct qib_devdata *dd = dd_from_dev(dev);
> + unsigned pidx;
> + unsigned i;
> + int len = 0;
> + unsigned long flags;
> +
> + for (pidx = 0; pidx < dd->num_pports; ++pidx) {
> + struct qib_ibport *ibp = &dd->pport[pidx].ibport_data;
> +
> + len += sprintf(buf + len,
> + "Port %d:\n"
> + "RC timeouts %d\n"
> + "RC resends %d\n"
> + "RC QACKs %d\n"
> + "RC SEQ NAKs %d\n"
> + "RC RDMA seq %d\n"
> + "RC RNR NAKs %d\n"
> + "RC OTH NAKs %d\n"
> + "RC DComp %d\n"
> + "RCr dup req %d\n"
> + "RCr SEQ NAK %d\n"
> + "wait piobuf %d\n"
> + "wait DMA %d\n"
> + "wait TX %d\n"
> + "unaligned %d\n"
> + "loop pkts %d\n"
> + "PKT drops %d\n",
> + dd->pport[pidx].port,
> + ibp->n_timeouts, ibp->n_rc_resends,
> + ibp->n_rc_qacks, ibp->n_seq_naks,
> + ibp->n_rdma_seq, ibp->n_rnr_naks,
> + ibp->n_other_naks, ibp->n_rc_delayed_comp,
> + ibp->n_rc_dupreq, ibp->n_rc_seqnak,
> + dev->n_piowait, ibp->n_dmawait, dev->n_txwait,
> + ibp->n_unaligned, ibp->n_loop_pkts,
> + ibp->n_pkt_drops);
> + for (i = 0; i < ARRAY_SIZE(ibp->opstats); i++) {
> + const struct qib_opcode_stats *si = &ibp->opstats[i];
> +
> + if (!si->n_packets && !si->n_bytes)
> + continue;
> + len += sprintf(buf + len, "%02x %llu/%llu\n", i,
> + (unsigned long long) si->n_packets,
> + (unsigned long long) si->n_bytes);
> + }
> + }
> + len += sprintf(buf + len, "Ctx:npkts");
> + for (i = 0; i < dd->first_user_ctxt; i++) {
> + if (!dd->rcd[i])
> + continue;
> + len += sprintf(buf + len, " %u:%u", i,
> + dd->rcd[i]->pkt_count);
> + }
> + len += sprintf(buf + len, "\n");
> + spin_lock_irqsave(&dev->qpt_lock, flags);
> + for (i = 0; i < dev->qp_table_size; i++) {
> + struct qib_qp *qp;
> + for (qp = dev->qp_table[i]; qp != NULL; qp = qp->next) {
> + struct qib_swqe *wqe;
> +
> + if (qp->s_last == qp->s_acked &&
> + qp->s_acked == qp->s_cur &&
> + qp->s_cur == qp->s_tail &&
> + qp->s_tail == qp->s_head)
> + continue;
> + if (len + 128 >= PAGE_SIZE)
> + break;
> + wqe = get_swqe_ptr(qp, qp->s_last);
> + len += sprintf(buf + len,
> + "QP%u %s %u %u %u f=%x %u %u %u %u %u "
> + "PSN %x %x %x %x %x "
> + "(%u %u %u %u %u %u) QP%u LID %x\n",
> + qp->ibqp.qp_num,
> + qp_type_str[qp->ibqp.qp_type],
> + qp->state,
> + wqe->wr.opcode,
> + qp->s_hdrwords,
> + qp->s_flags,
> + atomic_read(&qp->s_dma_busy),
> + !list_empty(&qp->iowait),
> + qp->timeout,
> + wqe->ssn,
> + qp->s_lsn,
> + qp->s_last_psn,
> + qp->s_psn, qp->s_next_psn,
> + qp->s_sending_psn, qp->s_sending_hpsn,
> + qp->s_last, qp->s_acked, qp->s_cur,
> + qp->s_tail, qp->s_head, qp->s_size,
> + qp->remote_qpn,
> + qp->remote_ah_attr.dlid);
> + }
> + }
> + spin_unlock_irqrestore(&dev->qpt_lock, flags);
> + return len;
> +}
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2] IB/core: allow HCAs to create IB port sysfs files
From: Roland Dreier @ 2010-04-08 20:51 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma
In-Reply-To: <adamxxdwtys.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
> Rather than have the device driver call a new function that then calls
> it back, how about if we just add a parameter to the
> ib_register_device() call that is the callback to add the sysfs stuff?
> This would let us avoid the race where the device is registered but not
> all the sysfs stuff is created yet.
And as an improvement, maybe someday we could convert the existing
per-device driver-specific sysfs stuff to a similar approach.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-08 21:29 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <adaiq81wtw3.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
This was for debugging and clearly could use a better method.
Some stats are definitely useful and having files per value
would make scripting easier too.
I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
One other place that has multiple values is dumping the QSFP
data from the IB cable.
On Thu, 2010-04-08 at 13:50 -0700, Roland Dreier wrote:
> I didn't audit everything (so there may be more problems here too), but
> certainly this violates the "one value per sysfs file" rule rather
> egregiously:
>
> > +static ssize_t show_stats(struct device *device, struct device_attribute *attr,
> > + char *buf)
> > +{
> > + struct qib_ibdev *dev =
> > + container_of(device, struct qib_ibdev, ibdev.dev);
> > + struct qib_devdata *dd = dd_from_dev(dev);
> > + unsigned pidx;
> > + unsigned i;
> > + int len = 0;
> > + unsigned long flags;
> > +
> > + for (pidx = 0; pidx < dd->num_pports; ++pidx) {
> > + struct qib_ibport *ibp = &dd->pport[pidx].ibport_data;
> > +
> > + len += sprintf(buf + len,
> > + "Port %d:\n"
> > + "RC timeouts %d\n"
> > + "RC resends %d\n"
> > + "RC QACKs %d\n"
> > + "RC SEQ NAKs %d\n"
> > + "RC RDMA seq %d\n"
> > + "RC RNR NAKs %d\n"
> > + "RC OTH NAKs %d\n"
> > + "RC DComp %d\n"
> > + "RCr dup req %d\n"
> > + "RCr SEQ NAK %d\n"
> > + "wait piobuf %d\n"
> > + "wait DMA %d\n"
> > + "wait TX %d\n"
> > + "unaligned %d\n"
> > + "loop pkts %d\n"
> > + "PKT drops %d\n",
> > + dd->pport[pidx].port,
> > + ibp->n_timeouts, ibp->n_rc_resends,
> > + ibp->n_rc_qacks, ibp->n_seq_naks,
> > + ibp->n_rdma_seq, ibp->n_rnr_naks,
> > + ibp->n_other_naks, ibp->n_rc_delayed_comp,
> > + ibp->n_rc_dupreq, ibp->n_rc_seqnak,
> > + dev->n_piowait, ibp->n_dmawait, dev->n_txwait,
> > + ibp->n_unaligned, ibp->n_loop_pkts,
> > + ibp->n_pkt_drops);
> > + for (i = 0; i < ARRAY_SIZE(ibp->opstats); i++) {
> > + const struct qib_opcode_stats *si = &ibp->opstats[i];
> > +
> > + if (!si->n_packets && !si->n_bytes)
> > + continue;
> > + len += sprintf(buf + len, "%02x %llu/%llu\n", i,
> > + (unsigned long long) si->n_packets,
> > + (unsigned long long) si->n_bytes);
> > + }
> > + }
> > + len += sprintf(buf + len, "Ctx:npkts");
> > + for (i = 0; i < dd->first_user_ctxt; i++) {
> > + if (!dd->rcd[i])
> > + continue;
> > + len += sprintf(buf + len, " %u:%u", i,
> > + dd->rcd[i]->pkt_count);
> > + }
> > + len += sprintf(buf + len, "\n");
> > + spin_lock_irqsave(&dev->qpt_lock, flags);
> > + for (i = 0; i < dev->qp_table_size; i++) {
> > + struct qib_qp *qp;
> > + for (qp = dev->qp_table[i]; qp != NULL; qp = qp->next) {
> > + struct qib_swqe *wqe;
> > +
> > + if (qp->s_last == qp->s_acked &&
> > + qp->s_acked == qp->s_cur &&
> > + qp->s_cur == qp->s_tail &&
> > + qp->s_tail == qp->s_head)
> > + continue;
> > + if (len + 128 >= PAGE_SIZE)
> > + break;
> > + wqe = get_swqe_ptr(qp, qp->s_last);
> > + len += sprintf(buf + len,
> > + "QP%u %s %u %u %u f=%x %u %u %u %u %u "
> > + "PSN %x %x %x %x %x "
> > + "(%u %u %u %u %u %u) QP%u LID %x\n",
> > + qp->ibqp.qp_num,
> > + qp_type_str[qp->ibqp.qp_type],
> > + qp->state,
> > + wqe->wr.opcode,
> > + qp->s_hdrwords,
> > + qp->s_flags,
> > + atomic_read(&qp->s_dma_busy),
> > + !list_empty(&qp->iowait),
> > + qp->timeout,
> > + wqe->ssn,
> > + qp->s_lsn,
> > + qp->s_last_psn,
> > + qp->s_psn, qp->s_next_psn,
> > + qp->s_sending_psn, qp->s_sending_hpsn,
> > + qp->s_last, qp->s_acked, qp->s_cur,
> > + qp->s_tail, qp->s_head, qp->s_size,
> > + qp->remote_qpn,
> > + qp->remote_ah_attr.dlid);
> > + }
> > + }
> > + spin_unlock_irqrestore(&dev->qpt_lock, flags);
> > + return len;
> > +}
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Roland Dreier @ 2010-04-08 21:33 UTC (permalink / raw)
To: Ralph Campbell; +Cc: linux-rdma@vger.kernel.org
In-Reply-To: <1270762193.2278.25.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
> This was for debugging and clearly could use a better method.
> Some stats are definitely useful and having files per value
> would make scripting easier too.
> I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
>
> One other place that has multiple values is dumping the QSFP
> data from the IB cable.
You can stick debugging data under debugfs. Or just kill it for now and
figure out how to add it back later.
--
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Jason Gunthorpe @ 2010-04-08 22:08 UTC (permalink / raw)
To: Ralph Campbell
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1270762193.2278.25.camel-/vjeY7uYZjrPXfVEPVhPGq6RkeBMCJyt@public.gmane.org>
On Thu, Apr 08, 2010 at 02:29:53PM -0700, Ralph Campbell wrote:
> One other place that has multiple values is dumping the QSFP
> data from the IB cable.
I was going to comment that building your own I2C subsystem is kinda
strange, can't the in-kernel stuff be used?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ira Weiny @ 2010-04-08 22:12 UTC (permalink / raw)
To: Roland Dreier; +Cc: Ralph Campbell, linux-rdma@vger.kernel.org
In-Reply-To: <adask75vdbs.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Thu, 08 Apr 2010 14:33:59 -0700
Roland Dreier <rdreier-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> wrote:
> > This was for debugging and clearly could use a better method.
> > Some stats are definitely useful and having files per value
> > would make scripting easier too.
> > I could add /sys/class/infiniband/qib0/ports/1/diag_counters/*
> >
> > One other place that has multiple values is dumping the QSFP
> > data from the IB cable.
>
> You can stick debugging data under debugfs. Or just kill it for now and
> figure out how to add it back later.
I for one would rather not see this die. We have debugged some critical
issues using this data. The sysfs entries above are what Mellanox uses.
Should those also be changed?
Ira
> --
> Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
> http://*www.*cisco.com/web/about/doing_business/legal/cri/index.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://*vger.kernel.org/majordomo-info.html
>
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox