* Socket Direct Protocol: help (2)
From: Andrea Gozzelino @ 2010-04-12 14:33 UTC (permalink / raw)
To: linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI, Eric B Munson, Steve Wise <swise@
In-Reply-To: <13381_1271060138_o3C8FUH8013968_2977669.1271060085255.SLOX.WebMail.wwwrun-XDIR3SKYeFbgKi2NxijLtw@public.gmane.org>
On Apr 12, 2010 10:14 AM, Andrea Gozzelino
<Andrea.Gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org> wrote:
> Good morning,
>
> I'm testing some Neteffect cards (Intel code E10G81GP - Neteffect
> NE020.LP.1.SSR).
> PC has Linux| (kernel version) 2.6.18-164.15.1.el5 | x86_64 x86_64
> x86_64 GNU/Linux.
>
> In this phase, I measure the bandwidth with the netserver/nerperf
> (version netperf-2.4.5) ad hoc tests.
> They work fine with TCP protocol - as OFED 1.5.1 example programs -
> and
> they have some problems with SDP one.
>
> I'm trying test with the command lines below:
>
> server: LD_PRELOAD=/usr/local/lib64/libsdp.so netserver
>
> client: LD_PRELOAD=/usr/local/lib64/libsdp.so netperf -H
> server_address
> -c -C
> -- -m 65536
>
> The /etc/libsdp.conf file contains rules below:
> use both listen * *:*
> use both connect * *:*
> log min-level 9 destination file libsdp.log
>
> Client displays "Connection error: Can not allocate memory" and the
> connection fails.
> (original text on client log file:libsdp Error connect: failed for SDP
> fd:6 with error:Cannot allocate memory)
>
> The library path is:
> /usr/local/lib64/libsdp.so
>
>
> Could someone explain me how LD_PRELOAD environment variable must be
> set?
> I don't understand why the test work with TCP and not with SDP.
> Could I work with wrong Linux kernel environment or parameters?
>
> I don't know if there is a specific mailing list for SDP so I ask you
> help.
>
> Thank you very much,
> Andrea
>
>
>
>
>
>
>
> Andrea Gozzelino
>
> INFN - Laboratori Nazionali di Legnaro (LNL)
> Viale dell'Universita' 2
> I-35020 - Legnaro (PD)- ITALIA
> Tel: +39 049 8068346
> Fax: +39 049 641925
> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Hi all,
I add that in kernel space SDP debug the error is:
command line: dmesg
sdp_init_qp:95 sdp_sock( 2100:2 40720:0): recv sge's. capability: 4
needed: 9
sdp_init_qp:95 sdp_sock( 2100:2 41203:0): recv sge's. capability: 4
needed: 9
The structure sdp_init_qp() is defined in
/usr/src/ofa_kernel-1.5.1/drivers/infiniband/ulp/sdp/sdp_cma.c (lines 76
- 141).
Could be a firmware problem?
I have this situation:
command line: ethtool -i eth2
driver: iw_nes
version: 1.5.0.0
firmware-version: 3.16
bus-info: 0000:03:00.0
Thank you very much,
Andrea
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2
I-35020 - Legnaro (PD)- ITALIA
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v2 38/51] IB/qib: Add qib_sysfs.c
From: Ralph Campbell @ 2010-04-12 17:21 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Roland Dreier, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100410002705.GQ15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Fri, 2010-04-09 at 17:27 -0700, Jason Gunthorpe wrote:
> On Fri, Apr 09, 2010 at 05:13:24PM -0700, Ralph Campbell wrote:
>
> > For the QSFP data, I hope I can leave it as is since it is
> > related to the link state that the other files contain.
> > It is a read-only file so no issue with trying to set a value.
>
> There was some flack for other stuff like this a while back.
>
> IMHO, it would be appropriate to have a hex dump of the entire QFSP
> EEPROM and leave parsing to userspace, or put the parsed version in
> debugfs..
>
> Jason
OK. I will move it to our file system which is used
to export binary data.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* IPoIB performance benchmarking
From: Tom Ammon @ 2010-04-12 18:35 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; +Cc: Brian Haymore
Hi,
I'm trying to do some performance benchmarking of IPoIB on a DDR IB
cluster, and I am having a hard time understanding what I am seeing.
When I do a simple netperf, I get results like these:
[root@gateway3 ~]# netperf -H 192.168.23.252
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.23.252
(192.168.23.252) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 65536 65536 10.01 4577.70
Which is disappointing since it is simply two DDR IB-connected nodes
plugged in to a DDR switch - I would expect much higher throughput than
that. When I do a test with ibv_srq_pingpong (using the same message
size reported above), here's what I get:
[root@gateway3 ~]# ibv_srq_pingpong 192.168.23.252 -m 4096 -s 65536
local address: LID 0x012b, QPN 0x000337, PSN 0x19cc85
local address: LID 0x012b, QPN 0x000338, PSN 0x956fc2
...
[output omitted]
...
remote address: LID 0x0129, QPN 0x00032e, PSN 0x891ce3
131072000 bytes in 0.08 seconds = 12763.08 Mbit/sec
1000 iters in 0.08 seconds = 82.16 usec/iter
Which is much closer to what I would expect with DDR.
The MTU on both of the QLogic DDR HCAs is set to 4096, as it is on the
QLogic switch.
I know the above is not completely apples-to-apples, since the
ibv_srq_pingpong is layer2 and is using 16 QPs. So I ran it again with
only a single QP, to make it more roughly equivalent of my single-stream
netperf test, and I still get almost double the performance:
[root@gateway3 ~]# ibv_srq_pingpong 192.168.23.252 -m 4096 -s 65536 -q 1
local address: LID 0x012b, QPN 0x000347, PSN 0x65fb56
remote address: LID 0x0129, QPN 0x00032f, PSN 0x5e52f9
131072000 bytes in 0.13 seconds = 8323.22 Mbit/sec
1000 iters in 0.13 seconds = 125.98 usec/iter
Is there something that I am not understanding, here? Is there any way
to make single-stream TCP IPoIB performance better than 4.5Gb/s on a DDR
network? Am I just not using the benchmarking tools correctly?
Thanks,
Tom
--
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273
Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: IPoIB performance benchmarking
From: Dave Olson @ 2010-04-12 20:19 UTC (permalink / raw)
To: Tom Ammon
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Brian Haymore
In-Reply-To: <4BC367DD.30606-wbocuHtxKic@public.gmane.org>
On Mon, 12 Apr 2010, Tom Ammon wrote:
| I'm trying to do some performance benchmarking of IPoIB on a DDR IB
| cluster, and I am having a hard time understanding what I am seeing.
|
| When I do a simple netperf, I get results like these:
|
| [root@gateway3 ~]# netperf -H 192.168.23.252
| TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.23.252
| (192.168.23.252) port 0 AF_INET
| Recv Send Send
| Socket Socket Message Elapsed
| Size Size Size Time Throughput
| bytes bytes bytes secs. 10^6bits/sec
|
| 87380 65536 65536 10.01 4577.70
Are you using connected mode, or UD? Since you say you have a 4K MTU,
I'm guessing you are using UD. Change to use connected mode (edit
/etc/infiniband/openib.conf), or as a quick test
echo connected > /sys/class/net/ib0/mode
and then the mtu should show as 65520. That should help
the bandwidth a fair amount.
Dave Olson
dave.olson-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Pavel Machek @ 2010-04-12 20:20 UTC (permalink / raw)
To: Eric B Munson
Cc: akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, mingo-X9Un+BFzKDI
In-Reply-To: <1271053337-7121-1-git-send-email-ebmunson-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
Hi!
> I am resubmitting this patch because I believe that the discussion
> has shown this to be an acceptable solution. I have fixed the 32 bit
> build errors, but other than that change, the code is the same as
> Roland's V3 patch.
>
> From: Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
>
> As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
> and follow-up messages, libraries using RDMA would like to track
> precisely when application code changes memory mapping via free(),
> munmap(), etc. Current pure-userspace solutions using malloc hooks
> and other tricks are not robust, and the feeling among experts is that
> the issue is unfixable without kernel help.
I do not know. I still believe that this does not belong in the
kernel; application should not need to trace itself to know what it does.
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: IPoIB performance benchmarking
From: Tom Ammon @ 2010-04-12 20:52 UTC (permalink / raw)
To: Dave Olson
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Brian Dale Haymore
In-Reply-To: <alpine.LFD.1.10.1004121317270.21537-vxnkQ4oxbxUi9g6yJnKVd0EOCMrvLtNR@public.gmane.org>
Dave,
Thanks for the pointer. I thought it was running in connected mode, and
looking at that variable that you mentioned confirms it:
[root@gateway3 ~]# cat /sys/class/net/ib0/mode
connected
And the IP MTU shows up as:
[root@gateway3 ~]# ifconfig ib0
ib0 Link encap:InfiniBand HWaddr
80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.23.253 Bcast:192.168.23.255
Mask:255.255.254.0
inet6 addr: fe80::211:7500:ff:6edc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:2319010 errors:0 dropped:0 overruns:0 frame:0
TX packets:4512605 errors:0 dropped:33011 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:5450805352 (5.0 GiB) TX bytes:154353169896 (143.7 GiB)
This is partly why I'm stumped - I've seen threads about how connected
mode is supposed to improve IPoIB performance, but I'm not seeing as
much performance as I'd like.
Tom
On 04/12/2010 02:19 PM, Dave Olson wrote:
> On Mon, 12 Apr 2010, Tom Ammon wrote:
> | I'm trying to do some performance benchmarking of IPoIB on a DDR IB
> | cluster, and I am having a hard time understanding what I am seeing.
> |
> | When I do a simple netperf, I get results like these:
> |
> | [root@gateway3 ~]# netperf -H 192.168.23.252
> | TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.23.252
> | (192.168.23.252) port 0 AF_INET
> | Recv Send Send
> | Socket Socket Message Elapsed
> | Size Size Size Time Throughput
> | bytes bytes bytes secs. 10^6bits/sec
> |
> | 87380 65536 65536 10.01 4577.70
>
> Are you using connected mode, or UD? Since you say you have a 4K MTU,
> I'm guessing you are using UD. Change to use connected mode (edit
> /etc/infiniband/openib.conf), or as a quick test
>
> echo connected> /sys/class/net/ib0/mode
>
> and then the mtu should show as 65520. That should help
> the bandwidth a fair amount.
>
>
> Dave Olson
> dave.olson-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org
>
--
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273
Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] uDAPL v1.2 - cma: memory leak of FD's (pipe) created during dat_evd_create
From: Davis, Arlin R @ 2010-04-12 21:21 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, ofw_list
Add checking for pipe FD's during destroy and clean them up with close.
Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
dapl/openib_cma/dapl_ib_cq.c | 8 +++++++-
1 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/dapl/openib_cma/dapl_ib_cq.c b/dapl/openib_cma/dapl_ib_cq.c
index cf19f38..c54bbaf 100644
--- a/dapl/openib_cma/dapl_ib_cq.c
+++ b/dapl/openib_cma/dapl_ib_cq.c
@@ -462,8 +462,11 @@ dapls_ib_wait_object_create(IN DAPL_EVD *evd_ptr,
ibv_create_comp_channel(
evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle);
- if ((*p_cq_wait_obj_handle)->events == NULL)
+ if ((*p_cq_wait_obj_handle)->events == NULL) {
+ close((*p_cq_wait_obj_handle)->pipe[0]);
+ close((*p_cq_wait_obj_handle)->pipe[1]);
goto bail;
+ }
return DAT_SUCCESS;
bail:
@@ -483,6 +486,9 @@ dapls_ib_wait_object_destroy(IN ib_wait_obj_handle_t p_cq_wait_obj_handle)
ibv_destroy_comp_channel(p_cq_wait_obj_handle->events);
+ close(p_cq_wait_obj_handle->pipe[0]);
+ close(p_cq_wait_obj_handle->pipe[1]);
+
dapl_os_free(p_cq_wait_obj_handle,
sizeof(struct _ib_wait_obj_handle));
--
1.5.2.5
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH] uDAPL v1.2 - cma: memory leak of verbs CQ and completion channels created during dat_ia_open
From: Davis, Arlin R @ 2010-04-12 21:21 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, ofw_list
check/cleanup CQ and completion channels during dat_ia_close
Signed-off-by: Arlin Davis <arlin.r.davis-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
dapl/openib_cma/dapl_ib_util.c | 22 ++++++++++++++++------
1 files changed, 16 insertions(+), 6 deletions(-)
diff --git a/dapl/openib_cma/dapl_ib_util.c b/dapl/openib_cma/dapl_ib_util.c
index 9d97ae1..00aa5fb 100755
--- a/dapl/openib_cma/dapl_ib_util.c
+++ b/dapl/openib_cma/dapl_ib_util.c
@@ -373,12 +373,6 @@ DAT_RETURN dapls_ib_close_hca(IN DAPL_HCA *hca_ptr)
dapl_dbg_log(DAPL_DBG_TYPE_UTIL," close_hca: %p->%p\n",
hca_ptr,hca_ptr->ib_hca_handle);
- if (hca_ptr->ib_hca_handle != IB_INVALID_HANDLE) {
- if (rdma_destroy_id(hca_ptr->ib_trans.cm_id))
- return(dapl_convert_errno(errno,"ib_close_device"));
- hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
- }
-
dapl_os_lock(&g_hca_lock);
if (g_ib_thread_state != IB_THREAD_RUN) {
dapl_os_unlock(&g_hca_lock);
@@ -410,6 +404,22 @@ DAT_RETURN dapls_ib_close_hca(IN DAPL_HCA *hca_ptr)
nanosleep (&sleep, &remain);
}
bail:
+ if (hca_ptr->ib_trans.ib_cq)
+ ibv_destroy_comp_channel(hca_ptr->ib_trans.ib_cq);
+
+ if (hca_ptr->ib_trans.ib_cq_empty) {
+ struct ibv_comp_channel *channel;
+ channel = hca_ptr->ib_trans.ib_cq_empty->channel;
+ ibv_destroy_cq(hca_ptr->ib_trans.ib_cq_empty);
+ ibv_destroy_comp_channel(channel);
+ }
+
+ if (hca_ptr->ib_hca_handle != IB_INVALID_HANDLE) {
+ if (rdma_destroy_id(hca_ptr->ib_trans.cm_id))
+ return (dapl_convert_errno(errno, "ib_close_device"));
+ hca_ptr->ib_hca_handle = IB_INVALID_HANDLE;
+ }
+
return (DAT_SUCCESS);
}
--
1.5.2.5
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: IPoIB performance benchmarking
From: Dave Olson @ 2010-04-12 22:25 UTC (permalink / raw)
To: Tom Ammon
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Brian Dale Haymore
In-Reply-To: <4BC3882B.4070200-wbocuHtxKic@public.gmane.org>
On Mon, 12 Apr 2010, Tom Ammon wrote:
| Thanks for the pointer. I thought it was running in connected mode, and
| looking at that variable that you mentioned confirms it:
| [root@gateway3 ~]# ifconfig ib0
| ib0 Link encap:InfiniBand HWaddr
| 80:00:00:02:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
| inet addr:192.168.23.253 Bcast:192.168.23.255 Mask:255.255.254.0
| RX packets:2319010 errors:0 dropped:0 overruns:0 frame:0
| TX packets:4512605 errors:0 dropped:33011 overruns:0 carrier:0
That's a lot of packets dropped on the tx side.
If you have the qlogic software installed, running ipathstats -c1 while
you are running the test would be useful, otherwise perfquery -r at
start and another perfquery at the end on both nodes might point to
something.
Oh, and depending on your tcp stack tuning, setting the receive and/or
send buffer size might help. These are all ddr results, on a more
or less OFED 1.5.1 stack (completely unofficial, blah blah).
And yes, multi-thread will bring the results up (iperf, rather than netperf).
# netperf -H ib-host TCP_STREAM -- -m 65536
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ib-host (172.29.9.46) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 65536 65536 10.03 5150.24
# netperf -H ib-host TCP_STREAM -- -m 65536 -S 131072
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ib-host (172.29.9.46) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
262144 65536 65536 10.03 5401.83
# netperf -H ib-host TCP_STREAM -- -m 65536 -S 262144
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to ib-host (172.29.9.46) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
524288 65536 65536 10.01 5478.28
Dave Olson
dave.olson-h88ZbnxC6KDQT0dZR+AlfA@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Andrew Morton @ 2010-04-12 23:03 UTC (permalink / raw)
To: Eric B Munson; +Cc: linux-kernel, linux-rdma, rolandd, peterz, pavel, mingo
In-Reply-To: <1271053337-7121-1-git-send-email-ebmunson@us.ibm.com>
On Mon, 12 Apr 2010 07:22:17 +0100
Eric B Munson <ebmunson@us.ibm.com> wrote:
> Andrew,
>
> I am resubmitting this patch because I believe that the discussion
> has shown this to be an acceptable solution.
To whom? Some acked-by's would clarify.
> I have fixed the 32 bit
> build errors, but other than that change, the code is the same as
> Roland's V3 patch.
>
> From: Roland Dreier <rolandd@cisco.com>
>
> As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
> and follow-up messages, libraries using RDMA would like to track
> precisely when application code changes memory mapping via free(),
> munmap(), etc. Current pure-userspace solutions using malloc hooks
> and other tricks are not robust, and the feeling among experts is that
> the issue is unfixable without kernel help.
But this info could be reassembled by tracking syscall activity, yes?
Perhaps some discussion here explaining why the (possibly enhanced)
ptrace, audit, etc interfaces are unsuitable.
> We solve this not by implementing the full API proposed in the email
> linked above but rather with a simpler and more generic interface,
> which may be useful in other contexts. Specifically, we implement a
> new character device driver, ummunotify, that creates a /dev/ummunotify
> node. A userspace process can open this node read-only and use the fd
> as follows:
>
> 1. ioctl() to register/unregister an address range to watch in the
> kernel (cf struct ummunotify_register_ioctl in <linux/ummunotify.h>).
>
> 2. read() to retrieve events generated when a mapping in a watched
> address range is invalidated (cf struct ummunotify_event in
> <linux/ummunotify.h>). select()/poll()/epoll() and SIGIO are
> handled for this IO.
>
> 3. mmap() one page at offset 0 to map a kernel page that contains a
> generation counter that is incremented each time an event is
> generated. This allows userspace to have a fast path that checks
> that no events have occurred without a system call.
OK, what's missing from this whole description and from ummunotify.txt
is: how does one specify the target process? Does /dev/ummunotify
implicitly attach to current->mm? If so, why, and what are the
implications of this?
If instead it is possible to attach to some other process's mmu
activity (/proc/<pid>/ummunotity?) then how is that done and what are
the security/permissions implications?
Also, the whole thing is obviously racy: by the time userspace finds
out that something has happened, it might have changed. This
inevitably reduces the applicability/usefulness of the whole thing as
compared to some synchronous mechanism which halts the monitored thread
until the request has been processed and acked. All this should (IMO)
be explored, explained and justified.
Also, what prevents the obvious DoS which occurs when I register for
events and just let them queue up until the kernel runs out of memory?
presumably events get dropped - what are the reliability implications
of this and how is the max queue length managed?
Also, ioctls are unpopular. Were other intefaces considered?
> Thanks to Jason Gunthorpe <jgunthorpe <at> obsidianresearch.com> for
> suggestions on the interface design. Also thanks to Jeff Squyres
> <jsquyres <at> cisco.com> for prototyping support for this in Open MPI, which
> helped find several bugs during development.
>
> Signed-off-by: Roland Dreier <rolandd@cisco.com>
> Signed-off-by: Eric B Munson <ebmunson@us.ibm.com>
>
> ---
>
> Changes since v3:
> - Fixed replaced [get|put] user with copy_[from|to]_user to fix x86
> builds
> ---
> Documentation/Makefile | 3 +-
> Documentation/ummunotify/Makefile | 7 +
> Documentation/ummunotify/ummunotify.txt | 150 ++++++++
> Documentation/ummunotify/umn-test.c | 200 +++++++++++
> drivers/char/Kconfig | 12 +
> drivers/char/Makefile | 1 +
> drivers/char/ummunotify.c | 567 +++++++++++++++++++++++++++++++
> include/linux/ummunotify.h | 121 +++++++
> 8 files changed, 1060 insertions(+), 1 deletions(-)
> create mode 100644 Documentation/ummunotify/Makefile
> create mode 100644 Documentation/ummunotify/ummunotify.txt
> create mode 100644 Documentation/ummunotify/umn-test.c
> create mode 100644 drivers/char/ummunotify.c
> create mode 100644 include/linux/ummunotify.h
>
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 6fc7ea1..27ba76a 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -1,3 +1,4 @@
> obj-m := DocBook/ accounting/ auxdisplay/ connector/ \
> filesystems/ filesystems/configfs/ ia64/ laptops/ networking/ \
> - pcmcia/ spi/ timers/ video4linux/ vm/ watchdog/src/
> + pcmcia/ spi/ timers/ video4linux/ vm/ ummunotify/ \
> + watchdog/src/
> diff --git a/Documentation/ummunotify/Makefile b/Documentation/ummunotify/Makefile
> new file mode 100644
> index 0000000..89f31a0
> --- /dev/null
> +++ b/Documentation/ummunotify/Makefile
> @@ -0,0 +1,7 @@
> +# List of programs to build
> +hostprogs-y := umn-test
> +
> +# Tell kbuild to always build the programs
> +always := $(hostprogs-y)
> +
> +HOSTCFLAGS_umn-test.o += -I$(objtree)/usr/include
> diff --git a/Documentation/ummunotify/ummunotify.txt b/Documentation/ummunotify/ummunotify.txt
> new file mode 100644
> index 0000000..78a79c2
> --- /dev/null
> +++ b/Documentation/ummunotify/ummunotify.txt
> @@ -0,0 +1,150 @@
> +UMMUNOTIFY
> +
> + Ummunotify relays MMU notifier events to userspace. This is useful
> + for libraries that need to track the memory mapping of applications;
> + for example, MPI implementations using RDMA want to cache memory
> + registrations for performance, but tracking all possible crazy cases
> + such as when, say, the FORTRAN runtime frees memory is impossible
> + without kernel help.
> +
> +Basic Model
> +
> + A userspace process uses it by opening /dev/ummunotify, which
> + returns a file descriptor. Interest in address ranges is registered
> + using ioctl() and MMU notifier events are retrieved using read(), as
> + described in more detail below. Userspace can register multiple
> + address ranges to watch, and can unregister individual ranges.
> +
> + Userspace can also mmap() a single read-only page at offset 0 on
> + this file descriptor. This page contains (at offest 0) a single
> + 64-bit generation counter that the kernel increments each time an
> + MMU notifier event occurs. Userspace can use this to very quickly
> + check if there are any events to retrieve without needing to do a
> + system call.
> +
> +Control
> +
> + To start using ummunotify, a process opens /dev/ummunotify in
> + read-only mode. Control from userspace is done via ioctl(); the
> + defined ioctls are:
> +
> + UMMUNOTIFY_EXCHANGE_FEATURES: This ioctl takes a single 32-bit
> + word of feature flags as input, and the kernel updates the
> + features flags word to contain only features requested by
> + userspace and also supported by the kernel.
> +
> + This ioctl is only included for forward compatibility; no
> + feature flags are currently defined, and the kernel will simply
> + update any requested feature mask to 0. The kernel will always
> + default to a feature mask of 0 if this ioctl is not used, so
> + current userspace does not need to perform this ioctl.
> +
> + UMMUNOTIFY_REGISTER_REGION: Userspace uses this ioctl to tell the
> + kernel to start delivering events for an address range. The
> + range is described using struct ummunotify_register_ioctl:
> +
> + struct ummunotify_register_ioctl {
> + __u64 start;
> + __u64 end;
> + __u64 user_cookie;
> + __u32 flags;
> + __u32 reserved;
> + };
> +
> + start and end give the range of userspace virtual addresses;
> + start is included in the range and end is not, so an example of
> + a 4 KB range would be start=0x1000, end=0x2000.
> +
> + user_cookie is an opaque 64-bit quantity that is returned by the
> + kernel in events involving the range, and used by userspace to
> + stop watching the range. Each registered address range must
> + have a distinct user_cookie.
> +
> + It is fine with the kernel if userspace registers multiple
> + overlapping or even duplicate address ranges, as long as a
> + different cookie is used for each registration.
> +
> + flags and reserved are included for forward compatibility;
> + userspace should simply set them to 0 for the current interface.
> +
> + UMMUNOTIFY_UNREGISTER_REGION: Userspace passes in the 64-bit
> + user_cookie used to register a range to tell the kernel to stop
> + watching an address range. Once this ioctl completes, the
> + kernel will not deliver any further events for the range that is
> + unregistered.
What happens if I register two regions with the same cookie? Will
UMMUNOTIFY_UNREGISTER_REGION unregister both, or did I cause a kernel
leak?
If it's possible to register for events from a different process then
how does the lifetime management happen? What happens when the target
process exits?
What happens if I close() the fd when there are outstanding events?
What happens if I close() the fd when there are still-registered
regions?
> +Events
> +
> + When an event occurs that invalidates some of a process's memory
> + mapping in an address range being watched, ummunotify queues an
> + event report for that address range. If more than one event
> + invalidates parts of the same address range before userspace
> + retrieves the queued report, then further reports for the same range
> + will not be queued -- when userspace does read the queue, only a
> + single report for a given range will be returned.
So the implementation keeps track of queued events down the
operation/start-address/end-address level and, for each new event,
checks whether that event is wholly contained within one of the
preceding start-address/end-address ranges and if that queued event
"invalidated" the new event's range, the new event is suppressed?
Sounds complex and expensive. Why was this added? Can't competent
userspace handle this situation?
And what does "invalidate" mean? munmap?
Methinks this paragraph needs some fleshing out.
> + If multiple ranges being watched are invalidated by a single event
> + (which is especially likely if userspace registers overlapping
> + ranges), then an event report structure will be queued for each
> + address range registration.
> +
> + Userspace retrieves queued events via read() on the ummunotify file
> + descriptor; a buffer that is at least as big as struct
> + ummunotify_event should be used to retrieve event reports, and if a
> + larger buffer is passed to read(), multiple reports will be returned
> + (if available).
What happens if I try to read() three bytes from that fd?
> + If the ummunotify file descriptor is in blocking mode,
Done how? Omitting O_NONBLOCK at open() time?
> + a read() call
> + will wait for an event report to be available.
Under which circumstances will that read() terminate? signals, presumably?
> + Userspace may also
> + set the ummunotify file descriptor to non-blocking mode and use all
> + standard ways of waiting for data to be available on the ummunotify
> + file descriptor, including epoll/poll()/select() and SIGIO.
> +
> + The format of event reports is:
> +
> + struct ummunotify_event {
> + __u32 type;
> + __u32 flags;
> + __u64 hint_start;
> + __u64 hint_end;
> + __u64 user_cookie_counter;
> + };
> +
> + where the type field is either UMMUNOTIFY_EVENT_TYPE_INVAL or
> + UMMUNOTIFY_EVENT_TYPE_LAST. Events of type INVAL describe
> + invalidation events as follows:
As follows where? Confused.
> + user_cookie_counter contains the
> + cookie passed in when userspace registered the range that the event
> + is for.
Why does it have "_counter" in its name?
> + hint_start and hint_end contain the start address and end
> + address that were invalidated.
> +
> + The flags word contains bit flags, with only UMMUNOTIFY_EVENT_FLAG_HINT
> + defined at the moment. If HINT is set, then the invalidation event
> + invalidated less than the full address range and the kernel returns
> + the exact range invalidated;
So my registration now effectively covers a shorter address range? it
may end up being very holey?
I wish you'd told us what "invalidate" means. If it means munmap()
then presumably the monitored target can later start to fill those
holes in again.
"hint" seems a strange name to use here. I don't understand why it is
appropriate instead of, say, UMMUNOTIFY_EVENT_INVALIDATE.
> + if HINT is not sent then hint_start and
> + hint_end are set to the original range registered by userspace.
> + (HINT will not be set if, for example, multiple events invalidated
> + disjoint parts of the range and so a single start/end pair cannot
> + represent the parts of the range that were invalidated)
> +
> + If the event type is LAST, then the read operation has emptied the
> + list of invalidated regions, and the flags, hint_start and hint_end
> + fields are not used. user_cookie_counter holds the value of the
> + kernel's generation counter (see below of more details) when the
> + empty list occurred.
Ah. So user_cookie_counter is a union. C has support for unions - why
not use one??
> +Generation Count
> +
> + Userspace may mmap() a page on a ummunotify file descriptor via
> +
> + mmap(NULL, sizeof (__u64), PROT_READ, MAP_SHARED, ummunotify_fd, 0);
> +
> + to get a read-only mapping of the kernel's 64-bit generation
> + counter. The kernel will increment this generation counter each
> + time an event report is queued.
So we have one CPU writing a memory location and another CPU reading it
without any locking?
What are the weird-architecture memory-barrier requirements here and
how are they handled?
How does the code handle the non-atomicity of u64 writes on 32-bit CPUs?
> + Userspace can use the generation counter as a quick check to avoid
> + system calls; if the value read from the mapped kernel counter is
> + still equal to the value returned in user_cookie_counter for the
> + most recent LAST event retrieved, then no further events have been
> + queued and there is no need to try a read() on the ummunotify file
> + descriptor.
I _guess_ that works OK on 32-bit, as long as userspace _only_ compares
this value with some previous one.
umm, no, there's still a race I think. If the counter increases from
0x00000000ffffffff to 0x0000000100000000 then userspace could see this
as two events when using this scheme.
I didn't get around to reading the code yet ;)
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Jason Gunthorpe @ 2010-04-12 23:59 UTC (permalink / raw)
To: Andrew Morton
Cc: Eric B Munson, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI
In-Reply-To: <20100412160359.1d9074dc.akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
On Mon, Apr 12, 2010 at 04:03:59PM -0700, Andrew Morton wrote:
> > As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
> > and follow-up messages, libraries using RDMA would like to track
> > precisely when application code changes memory mapping via free(),
> > munmap(), etc. Current pure-userspace solutions using malloc hooks
> > and other tricks are not robust, and the feeling among experts is that
> > the issue is unfixable without kernel help.
>
> But this info could be reassembled by tracking syscall activity, yes?
> Perhaps some discussion here explaining why the (possibly enhanced)
> ptrace, audit, etc interfaces are unsuitable.
Just to summarize some of the key points of this thingy, as related to
your comments:
1) It is really very narrowly focused on a particular problem MPI and
RDMA have due to the way their APIs don't really match. Roland
tried to make the interface general.. Maybe that is a mistake ..
2) A 'self-tracing' scheme is used, again, because of an API
mistmatching between a MPI library and it's own
applications. Attempting to hook the appropriate calls has
proven unsatisfactory (missing cases, and slow).
3) Being intended for MPI applications, performance is a huge
concern. Synchronous operation is very undesirable. Tracing APIs
are lossy - and there is no recovery option if an event is lost.
4) Realistically the only thing MPI cares about is if a virtual page
is unmapped/remapped. Loosing events is unacceptable.
5) This isn't really tracing. There is no queue. There aren't really
events. This works more like the diry/access bit in a page table,
it doesn't matter how many times something has been modified, only
that it has at least once since last time you looked.
This means the memory used is proportional to the number of
page-ranges you watch, and the number of events against those
page-ranges doesn't matter. No other API has this property.
Basically, this entire scheme is designed to detect that when a == b,
the internal state held by some_mpi_call is no longer valid, in
this kind of situation:
a = mmap(ONE_PAGE);
some_mpi_call(a);
munmap(a);
b = mmap(ONE_PAGE); // Kernel picks b == a
some_mpi_call(b);
All the races you point out, just don't matter for the MPI use
case. Essentially, if the app hits those races, then it is using the
MPI library in a buggy way.
That said, this could be explained better in the documentation file. :)
I'm sure Eric can go through the rest of your questions in greater
detail..
> > + Userspace can use the generation counter as a quick check to avoid
> > + system calls; if the value read from the mapped kernel counter is
> > + still equal to the value returned in user_cookie_counter for the
> > + most recent LAST event retrieved, then no further events have been
> > + queued and there is no need to try a read() on the ummunotify file
> > + descriptor.
>
> I _guess_ that works OK on 32-bit, as long as userspace _only_ compares
> this value with some previous one.
>
> umm, no, there's still a race I think. If the counter increases from
> 0x00000000ffffffff to 0x0000000100000000 then userspace could see this
> as two events when using this scheme.
The only case that matters for the generation counter optimization is
a false negative. As long as user space does:
u64 val = *counter;
if (val != last_counter)
last_counter = val;
Then you can get false positives as you point out, but never a false
negative. A false positive results in an extra syscall and the kernel
just returns no data.
Regards,
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Håkon Bugge @ 2010-04-13 8:29 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Andrew Morton, Eric B Munson, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI
In-Reply-To: <20100412235937.GF15629-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
On Apr 13, 2010, at 1:59 , Jason Gunthorpe wrote:
> On Mon, Apr 12, 2010 at 04:03:59PM -0700, Andrew Morton wrote:
>
>>> As discussed in <http://article.gmane.org/gmane.linux.drivers.openib/61925>
>>> and follow-up messages, libraries using RDMA would like to track
>>> precisely when application code changes memory mapping via free(),
>>> munmap(), etc. Current pure-userspace solutions using malloc hooks
>>> and other tricks are not robust, and the feeling among experts is that
>>> the issue is unfixable without kernel help.
I am not sure I agree with the premises here. ptMalloc and malloc hooks are not related to the issue in my opinion. User space library calls do not change virtual to physical mapping, system calls do. The following sys calls might change virtual to physical mapping: munmap(), mremap(), sbrk(), madvice(). What we need is glibc to provide hooks for these 4 sys calls and the general syscall() when its argument is one of the four mentioned syscalls. To me, that is what is needed, and the ummunotify direction seems way too complicated to me.
It is further claimed that "… other tricks are not robust". I wrote the code used in Scali/Platform MPI handling the issue. I do not think its fair to claim that this MPI is not robust in this matter nor that is performance is bad.
Thanks, Håkon
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Weird board ID
From: Justin Clift @ 2010-04-13 8:37 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi all,
Just received an old Topspin 2 port PCIe card (MTLP25208-C), using
firmware 4.6.0 and thought I'd update it to the latest firmware before
trying to do anything with it.
However, the board id is being reported really strangely:
$ sudo mstflint -d mthca0 q
Image type: Failsafe
I.S. Version: 1
Chip Revision: A0
Description: Node Port1 Port2
Sys image
GUIDs: 0005ad00000467fc 0005ad00000467fd 0005ad00000467fe
0005ad00000467ff
Board ID: q��\x03
VSD: q��\x03
PSID:
$
Does anyone know what that's about?
Was considering the firmware on the board may be stuffed, but it seems
to be verifying ok:
$ sudo mstflint -d mthca0 v
Failsafe image:
Invariant /0x00000028-0x0000095f (0x000938)/ (BOOT2) - OK
Primary Image /0x00010000-0x00010107 (0x000108)/ (Pointer Sector)- OK
/0x00030028-0x000308af (0x000888)/ (BOOT2) - OK
/0x000308b0-0x000348df (0x004030)/ (BOOT2) - OK
/0x000348e0-0x00036bd3 (0x0022f4)/ (Configuration) - OK
/0x00036bd4-0x00036c07 (0x000034)/ (GUID) - OK
/0x00036c08-0x0004452f (0x00d928)/ (DDR) - OK
/0x00044530-0x0004c97b (0x00844c)/ (DDR) - OK
/0x0004c97c-0x000617a3 (0x014e28)/ (DDR) - OK
/0x000617a4-0x00074813 (0x013070)/ (DDR) - OK
/0x00074814-0x00077883 (0x003070)/ (DDR) - OK
/0x00077884-0x00078f37 (0x0016b4)/ (DDR) - OK
/0x00078f38-0x00078f4b (0x000014)/ (Configuration) - OK
/0x00078f4c-0x00078f8f (0x000044)/ (Jump addresses) - OK
/0x00078f90-0x00082bf3 (0x009c64)/ (EMT Service) - OK
Secondary Image /0x00020000-0x00020107 (0x000108)/ (Pointer Sector)- OK
/0x00090028-0x000908af (0x000888)/ (BOOT2) - OK
/0x000908b0-0x000948df (0x004030)/ (BOOT2) - OK
/0x000948e0-0x00096bd3 (0x0022f4)/ (Configuration) - OK
/0x00096bd4-0x00096c07 (0x000034)/ (GUID) - OK
/0x00096c08-0x000a452f (0x00d928)/ (DDR) - OK
/0x000a4530-0x000ac97b (0x00844c)/ (DDR) - OK
/0x000ac97c-0x000c17a3 (0x014e28)/ (DDR) - OK
/0x000c17a4-0x000d4813 (0x013070)/ (DDR) - OK
/0x000d4814-0x000d7883 (0x003070)/ (DDR) - OK
/0x000d7884-0x000d8f37 (0x0016b4)/ (DDR) - OK
/0x000d8f38-0x000d8f4b (0x000014)/ (Configuration) - OK
/0x000d8f4c-0x000d8f8f (0x000044)/ (Jump addresses) - OK
/0x000d8f90-0x000e2bf3 (0x009c64)/ (EMT Service) - OK
FW image verification succeeded. Image is bootable.
$
*********************
This is interesting too (hopefully the correct firmware image). Not
that it's not extracting a correct firmware version number?
$ sudo mstflint -d mthca0 -i fw-25208-4_8_200-MHEL-CF128-T.bin -nofs b
Current FW version on flash: N/A
New FW version: N/A
Burn process will not be failsafe. No checks will be performed.
ALL flash, including the Invariant Sector will be overwritten.
If this process fails, computer may remain in an inoperable state.
Do you want to continue ? (y/n) [n] : n
$
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* two questions about RDMA-WRITE
From: 丁定华 @ 2010-04-13 10:08 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi guys:
I'm doing a project using RDMA to transfer data between servers. I'm
using RDMA_PS_TCP type and do RDMA_WRITE operations from client node
to server node. While the program succeed in setting up connection and
transferring data, I 'm still very confused about two problems:
(1) After transfer the data using IB_WR_RDMA_WRITE , the complete
queue entry found that the opcode turned from IB_WR_RDMA_WRITE to
IB_WR_RDMA_WRITE_WITH_IMM, I don't know the meaning of this opcode and
the reason of the change.
(2) It takes a long time to finish rdma_disconnect work (about 10
seconds), so is it a reasonable time?
My develop environment is Centos5.3, kernel version: 2.6.18-128.el5. I
will very appreciated if you guys can give me some tips. Please CC to
me since I didn't subscribe this list.
Thanks.
--
丁定华
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] opensm/osm_subnet.h: cosmetics - fix description of event_plugin option
From: Yevgeny Kliteynik @ 2010-04-13 10:39 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_subnet.h | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h
index 3970e98..83705a5 100644
--- a/opensm/include/opensm/osm_subnet.h
+++ b/opensm/include/opensm/osm_subnet.h
@@ -456,8 +456,8 @@ typedef struct osm_subn_opt {
* event_db_dump_file
* File to dump the event database to
*
-* event_db_plugin
-* Specify the name of the event plugin
+* event_plugin_name
+* Specify the name(s) of the event plugin(s)
*
* qos_options
* Default set of QoS options
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH] opensm/osm_subnet.c: fixing some options to not "hot-swappable"
From: Yevgeny Kliteynik @ 2010-04-13 10:40 UTC (permalink / raw)
To: Sasha Khapyorsky, Linux RDMA
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Return-Path: kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
X-OriginalArrivalTime: 13 Apr 2010 10:40:45.0762 (UTC) FILETIME=[C6063A20:01CADAF5]
The following options are not hot-swappable:
transaction_timeout
transaction_retries
use_ucast_cache
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_subnet.c | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c
index 9132c82..3f5f0f3 100644
--- a/opensm/opensm/osm_subnet.c
+++ b/opensm/opensm/osm_subnet.c
@@ -299,8 +299,8 @@ static const opt_rec_t opt_tbl[] = {
{ "max_wire_smps", OPT_OFFSET(max_wire_smps), opts_parse_uint32, NULL, 1 },
{ "console", OPT_OFFSET(console), opts_parse_charp, NULL, 0 },
{ "console_port", OPT_OFFSET(console_port), opts_parse_uint16, NULL, 0 },
- { "transaction_timeout", OPT_OFFSET(transaction_timeout), opts_parse_uint32, NULL, 1 },
- { "transaction_retries", OPT_OFFSET(transaction_retries), opts_parse_uint32, NULL, 1 },
+ { "transaction_timeout", OPT_OFFSET(transaction_timeout), opts_parse_uint32, NULL, 0 },
+ { "transaction_retries", OPT_OFFSET(transaction_retries), opts_parse_uint32, NULL, 0 },
{ "max_msg_fifo_timeout", OPT_OFFSET(max_msg_fifo_timeout), opts_parse_uint32, NULL, 1 },
{ "sm_priority", OPT_OFFSET(sm_priority), opts_parse_uint8, opts_setup_sm_priority, 1 },
{ "lmc", OPT_OFFSET(lmc), opts_parse_uint8, NULL, 1 },
@@ -328,7 +328,7 @@ static const opt_rec_t opt_tbl[] = {
{ "sweep_on_trap", OPT_OFFSET(sweep_on_trap), opts_parse_boolean, NULL, 1 },
{ "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, NULL, 0 },
{ "connect_roots", OPT_OFFSET(connect_roots), opts_parse_boolean, NULL, 1 },
- { "use_ucast_cache", OPT_OFFSET(use_ucast_cache), opts_parse_boolean, NULL, 1 },
+ { "use_ucast_cache", OPT_OFFSET(use_ucast_cache), opts_parse_boolean, NULL, 0 },
{ "log_file", OPT_OFFSET(log_file), opts_parse_charp, NULL, 0 },
{ "log_max_size", OPT_OFFSET(log_max_size), opts_parse_uint32, opts_setup_log_max_size, 1 },
{ "log_flags", OPT_OFFSET(log_flags), opts_parse_uint8, opts_setup_log_flags, 1 },
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH] opensm/osm_subnet.{c,h}: passing options to the event plugins
From: Yevgeny Kliteynik @ 2010-04-13 10:43 UTC (permalink / raw)
To: Sasha Khapyorsky, Linux RDMA
Adding new option string: "event_plugin_options"
A user can use this option to pass any plugin options.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/include/opensm/osm_subnet.h | 4 ++++
opensm/opensm/osm_subnet.c | 12 ++++++++++--
2 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h
index 83705a5..c609723 100644
--- a/opensm/include/opensm/osm_subnet.h
+++ b/opensm/include/opensm/osm_subnet.h
@@ -223,6 +223,7 @@ typedef struct osm_subn_opt {
char *event_db_dump_file;
#endif /* ENABLE_OSM_PERF_MGR */
char *event_plugin_name;
+ char *event_plugin_options;
char *node_name_map_name;
char *prefix_routes_file;
char *log_prefix;
@@ -459,6 +460,9 @@ typedef struct osm_subn_opt {
* event_plugin_name
* Specify the name(s) of the event plugin(s)
*
+* event_plugin_options
+* Options string that would be passed to the plugin(s)
+*
* qos_options
* Default set of QoS options
*
diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c
index e4126bc..9132c82 100644
--- a/opensm/opensm/osm_subnet.c
+++ b/opensm/opensm/osm_subnet.c
@@ -364,6 +364,7 @@ static const opt_rec_t opt_tbl[] = {
{ "event_db_dump_file", OPT_OFFSET(event_db_dump_file), opts_parse_charp, NULL, 0 },
#endif /* ENABLE_OSM_PERF_MGR */
{ "event_plugin_name", OPT_OFFSET(event_plugin_name), opts_parse_charp, NULL, 0 },
+ { "event_plugin_options", OPT_OFFSET(event_plugin_options), opts_parse_charp, NULL, 0 },
{ "node_name_map_name", OPT_OFFSET(node_name_map_name), opts_parse_charp, NULL, 0 },
{ "qos_max_vls", OPT_OFFSET(qos_options.max_vls), opts_parse_uint32, NULL, 1 },
{ "qos_high_limit", OPT_OFFSET(qos_options.high_limit), opts_parse_int32, NULL, 1 },
@@ -727,6 +728,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt)
#endif /* ENABLE_OSM_PERF_MGR */
p_opt->event_plugin_name = NULL;
+ p_opt->event_plugin_options = NULL;
p_opt->node_name_map_name = NULL;
p_opt->dump_files_dir = getenv("OSM_TMP_DIR");
@@ -1544,8 +1546,14 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)
fprintf(out,
"#\n# Event Plugin Options\n#\n"
- "event_plugin_name %s\n\n", p_opts->event_plugin_name ?
- p_opts->event_plugin_name : null_str);
+ "# Event plugin name(s)\n"
+ "event_plugin_name %s\n\n"
+ "# Options string that would be passed to the plugin(s)\n"
+ "event_plugin_options %s\n\n",
+ p_opts->event_plugin_name ?
+ p_opts->event_plugin_name : null_str,
+ p_opts->event_plugin_options ?
+ p_opts->event_plugin_options : null_str);
fprintf(out,
"#\n# Node name map for mapping node's to more descriptive node descriptions\n"
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Sasha Khapyorsky @ 2010-04-13 10:46 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100218124933.c018a23d.weiny2-i2BcT+NCU+M@public.gmane.org>
On 12:49 Thu 18 Feb , Ira Weiny wrote:
>
> From: Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org>
> Date: Fri, 22 Jan 2010 17:33:30 -0800
> Subject: [PATCH] libibnetdisc: Convert to a multi-smp algorithm
>
> v3: change DEFAULT_MAX_SMP_ON_WIRE to 2
>
> Allow for multiple SMP's to be on the wire at a single time. This
> algorithm splits the processing of SMP's to a small smp engine which
> may be useful to split out in the future.
>
> Signed-off-by: Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org>
Applied. Thanks.
However see some comments and questions below.
> +static int recv_port_info(smp_engine_t *engine, ibnd_smp_t * smp,
> + uint8_t *mad, void *cb_data)
> +{
> + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> + ibnd_node_t *node = (ibnd_node_t *)cb_data;
> + ibnd_port_t *port;
> + uint8_t *port_info = mad + IB_SMP_DATA_OFFS;
> + uint8_t port_num, local_port;
> +
> + port_num = mad_get_field(mad, 0, IB_MAD_ATTRMOD_F);
> + local_port = mad_get_field(port_info, 0, IB_PORT_LOCAL_PORT_F);
> +
> + /* this may have been created before */
> + port = node->ports[port_num];
> + if (!port) {
> + port = node->ports[port_num] = calloc(1, sizeof(*port));
> + if (!port) {
> + IBND_ERROR("Failed to allocate port\n");
> + return -1;
> + }
> + }
> +
> + memcpy(port->info, port_info, sizeof(port->info));
> + port->node = node;
> + port->portnum = port_num;
> + port->ext_portnum = 0;
> + port->base_lid = (uint16_t) mad_get_field(port->info, 0, IB_PORT_LID_F);
> + port->lmc = (uint8_t) mad_get_field(port->info, 0, IB_PORT_LMC_F);
> +
> + if (port_num == 0) {
> + node->smalid = port->base_lid;
> + node->smalmc = port->lmc;
> + } else if (node->type == IB_NODE_SWITCH) {
> + port->base_lid = node->smalid;
> + port->lmc = node->smalmc;
> + }
> +
> + add_to_portguid_hash(port, fabric->portstbl);
> +
> + debug_port(&smp->path, port);
>
> - mad_dump_node_type(type, 64, &node->type, sizeof(int));
> - printf("%s -> %s %s {%016" PRIx64 "} portnum %d base lid %d-%d\"%s\"\n",
> - portid2str(path), prompt, type, node->guid,
> - node->type == IB_NODE_SWITCH ? 0 : port->portnum,
> - port->base_lid,
> - port->base_lid + (1 << port->lmc) - 1, node->nodedesc);
> + if (port_num &&
> + (mad_get_field(port->info, 0, IB_PORT_PHYS_STATE_F)
> + == IB_PORT_PHYS_STATE_LINKUP)
> + &&
> + (node->type == IB_NODE_SWITCH || node == fabric->from_node)) {
What will happen when ibnetdiscover runs from host connected by two
ports to different subnets? Wouldn't it run over both fabrics (following
PortInfo querying loop in recv_node_info() below)?
> +
> + ib_portid_t path = smp->path;
> + if (extend_dpath(engine, &path, port_num) != -1)
> + query_node_info(engine, &path, node);
> + }
> +
> + return 0;
> +}
[snip]
> +static int recv_node_info(smp_engine_t *engine, ibnd_smp_t * smp,
> + uint8_t *mad, void *cb_data)
> +{
> + ibnd_fabric_t *fabric = ((ibnd_scan_t *)engine->user_data)->fabric;
> + int i = 0;
> + uint8_t *node_info = mad + IB_SMP_DATA_OFFS;
> + ibnd_node_t * rem_node = (ibnd_node_t *)cb_data;
> ibnd_node_t *node;
> + int node_is_new = 0;
> + uint64_t node_guid = mad_get_field64(node_info, 0, IB_NODE_GUID_F);
> + uint64_t port_guid = mad_get_field64(node_info, 0, IB_NODE_PORT_GUID_F);
> + int port_num = mad_get_field(node_info, 0, IB_NODE_LOCAL_PORT_F);
> + ibnd_port_t *port = NULL;
>
> - for (node = fabric->nodestbl[hash]; node; node = node->htnext)
> - if (node->guid == new->guid)
> - return node;
> + node = ibnd_find_node_guid(fabric, node_guid);
> + if (!node) {
> + node = create_node(engine, &smp->path, node_info);
> + if (!node)
> + return -1;
> + node_is_new = 1;
> + }
> + IBND_DEBUG("Found %s node GUID %lx (%s)\n",
> + (node_is_new) ? "new": "old", node->guid,
> + portid2str(&smp->path));
>
> - return NULL;
> + port = node->ports[port_num];
> + if (!port) {
> + /* If we have not see this port before create a shell for it */
> + port = node->ports[port_num] = calloc(1, sizeof(*port));
> + port->node = node;
> + port->portnum = port_num;
> + }
> + port->guid = port_guid;
> +
> + if (rem_node == NULL) /* this is the start node */
> + fabric->from_node = node;
> + else {
> + /* link ports... */
> + int rem_port_num = get_last_port(&smp->path);
> +
> + if (!rem_node->ports[rem_port_num]) {
> + IBND_ERROR("Internal Error; "
> + "Node(%p) %lx Port %d no port created!?!?!?\n\n",
> + rem_node, rem_node->guid, rem_port_num);
> + return (-1);
> + }
> +
> + link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
> + }
> +
> + if (!node_is_new)
> + return 0;
> +
> + query_node_desc(engine, &smp->path, node);
> +
> + if (node->type == IB_NODE_SWITCH)
> + query_switch_info(engine, &smp->path, node);
> +
> + /* process all the ports on this node */
> + for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
> + i <= node->numports; i++) {
> + query_port_info(engine, &smp->path, node, i);
> + }
This one.
> +
> + return 0;
> +}
[snip]
> diff --git a/infiniband-diags/libibnetdisc/src/internal.h b/infiniband-diags/libibnetdisc/src/internal.h
> index 348bd0f..61b644d 100644
> --- a/infiniband-diags/libibnetdisc/src/internal.h
> +++ b/infiniband-diags/libibnetdisc/src/internal.h
> @@ -39,6 +39,7 @@
> #define _INTERNAL_H_
>
> #include <infiniband/ibnetdisc.h>
> +#include <complib/cl_qmap.h>
I'm not really happy with complib dependency introduction - we had an
issues with it in the past.
Also I think that we can implement simpler transaction tracker (likely
even more efficient) by storing sent MADs in cyclic array and encoding
its index as part of TRID.
>
> #define IBND_DEBUG(fmt, ...) \
> if (ibdebug) { \
> @@ -52,16 +53,44 @@
>
> #define MAXHOPS 63
>
> -typedef struct ibnd_node_scan {
> - ibnd_node_t *node;
> - struct ibnd_node_scan *dnext; /* nodesdist next */
> -} ibnd_node_scan_t;
> +#define DEFAULT_MAX_SMP_ON_WIRE 2
>
> typedef struct ibnd_scan {
> - ibnd_node_scan_t *nodesdist[MAXHOPS + 1];
> ib_portid_t selfportid;
> + ibnd_fabric_t *fabric;
> } ibnd_scan_t;
>
> +
> +typedef struct ibnd_smp ibnd_smp_t;
> +typedef struct smp_engine smp_engine_t;
> +typedef int (*smp_comp_cb_t)(smp_engine_t *engine, ibnd_smp_t * smp,
> + uint8_t *mad_resp, void *cb_data);
> +struct ibnd_smp {
> + cl_map_item_t on_wire;
> + struct ibnd_smp * qnext;
> + smp_comp_cb_t cb;
> + void * cb_data;
> + ib_portid_t path;
> + ib_rpc_t rpc;
> +};
> +struct smp_engine {
> + struct ibmad_port *ibmad_port;
> + ibnd_smp_t *smp_queue_head;
> + ibnd_smp_t *smp_queue_tail;
> + void * user_data;
> + cl_qmap_t smps_on_wire;
> + int num_smps_outstanding;
> + int max_smps_on_wire;
> +};
> +
> +void smp_engine_init(smp_engine_t * engine, struct ibmad_port *ibmad_port,
> + void * user_data, int max_smps_on_wire);
> +int issue_smp(smp_engine_t *engine, ib_portid_t * portid,
> + unsigned attrid, unsigned mod,
> + smp_comp_cb_t cb, void * cb_data);
> +int process_mads(smp_engine_t *engine);
> +void smp_engine_destroy(smp_engine_t *engine);
> +
> void add_to_nodeguid_hash(ibnd_node_t * node, ibnd_node_t * hash[]);
>
> void add_to_portguid_hash(ibnd_port_t * port, ibnd_port_t * hash[]);
> diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
> new file mode 100644
> index 0000000..5571314
> --- /dev/null
> +++ b/infiniband-diags/libibnetdisc/src/query_smp.c
> @@ -0,0 +1,249 @@
> +/*
> + * Copyright (c) 2010 Lawrence Livermore National Laboratory
> + *
> + * This software is available to you under a choice of one of two
> + * licenses. You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * - Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer.
> + *
> + * - Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + *
> + */
> +
> +#include <errno.h>
> +#include <infiniband/ibnetdisc.h>
> +#include <infiniband/umad.h>
> +#include "internal.h"
> +
> +void
> +queue_smp(smp_engine_t *engine, ibnd_smp_t *smp)
> +{
> + smp->qnext = NULL;
> + if (!engine->smp_queue_head) {
> + engine->smp_queue_head = smp;
> + engine->smp_queue_tail = smp;
> + } else {
> + engine->smp_queue_tail->qnext = smp;
> + engine->smp_queue_tail = smp;
> + }
> +}
> +
> +ibnd_smp_t *
> +get_smp(smp_engine_t *engine)
> +{
> + ibnd_smp_t *head = engine->smp_queue_head;
> + ibnd_smp_t *tail = engine->smp_queue_tail;
> + ibnd_smp_t *rc = head;
> + if (head) {
> + if (tail == head)
> + engine->smp_queue_tail = NULL;
> + engine->smp_queue_head = head->qnext;
> + }
> + return rc;
> +}
> +
> +int send_smp(ibnd_smp_t * smp, struct ibmad_port *srcport)
> +{
> + int rc = 0;
> + uint8_t umad[1024];
> + ib_rpc_t *rpc = &smp->rpc;
> +
> + memset(umad, 0, umad_size() + IB_MAD_SIZE);
> +
> + if ((rc = mad_build_pkt(umad, &smp->rpc, &smp->path, NULL, NULL))
> + < 0) {
> + IBND_ERROR("mad_build_pkt failed; %d", rc);
> + return rc;
> + }
> +
> + if ((rc = umad_send(mad_rpc_portid(srcport),
> + mad_rpc_class_agent(srcport, rpc->mgtclass),
> + umad, IB_MAD_SIZE,
> + mad_get_timeout(srcport, rpc->timeout),
> + mad_get_retries(srcport))) < 0) {
> + IBND_ERROR("send failed; %d", rc);
> + return rc;
> + }
> +
> + return 0;
> +}
> +
> +static int process_smp_queue(smp_engine_t *engine)
> +{
> + int rc = 0;
> + ibnd_smp_t *smp;
> + while (cl_qmap_count(&engine->smps_on_wire) < engine->max_smps_on_wire) {
> + smp = get_smp(engine);
> + if (!smp)
> + return 0;
> +
> + cl_qmap_insert(&engine->smps_on_wire, (uint32_t)smp->rpc.trid,
> + (cl_map_item_t *)smp);
> + if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
> + return rc;
> + }
> + return 0;
> +}
> +
> +int issue_smp(smp_engine_t *engine, ib_portid_t * portid,
> + unsigned attrid, unsigned mod,
> + smp_comp_cb_t cb, void * cb_data)
> +{
> + ibnd_smp_t *smp = calloc(1, sizeof *smp);
> + if (!smp) {
> + IBND_ERROR("OOM");
> + return -ENOMEM;
> + }
> +
> + smp->cb = cb;
> + smp->cb_data = cb_data;
> + smp->path = *portid;
> + smp->rpc.method = IB_MAD_METHOD_GET;
> + smp->rpc.attr.id = attrid;
> + smp->rpc.attr.mod = mod;
> + smp->rpc.timeout = mad_get_timeout(engine->ibmad_port, 0);
> + smp->rpc.datasz = IB_SMP_DATA_SIZE;
> + smp->rpc.dataoffs = IB_SMP_DATA_OFFS;
> + smp->rpc.trid = mad_trid();
BTW, by reviewing mad_trid() code in libibmad:
uint64_t mad_trid(void)
{
static uint64_t base;
static uint64_t trid;
uint64_t next;
if (!base) {
srandom((int)time(0) * getpid());
base = random();
trid = random();
}
next = ++trid | (base << 32);
return next;
}
For me it doesn't look as thread safe function.
> +
> + if ((portid->lid <= 0) ||
> + (portid->drpath.drslid == 0xffff) ||
> + (portid->drpath.drdlid == 0xffff))
> + smp->rpc.mgtclass = IB_SMI_DIRECT_CLASS; /* direct SMI */
> + else
> + smp->rpc.mgtclass = IB_SMI_CLASS; /* Lid routed SMI */
> +
> + portid->sl = 0;
> + portid->qp = 0;
> +
> + engine->num_smps_outstanding++;
> + queue_smp(engine, smp);
> + return (process_smp_queue(engine));
> +}
> +
> +int process_one_recv(smp_engine_t *engine)
> +{
> + int rc = 0;
> + int status = 0;
> + ibnd_smp_t *smp;
> + uint8_t *mad;
> + uint32_t trid;
> + uint8_t umad[umad_size() + IB_MAD_SIZE];
> + int length = umad_size() + IB_MAD_SIZE;
> +
> + memset(umad, 0, sizeof(umad));
> +
> + /* wait for the next message */
> + if ((rc = umad_recv(mad_rpc_portid(engine->ibmad_port), umad, &length,
> + 0)) < 0) {
> + if (rc == -EWOULDBLOCK)
> + return 0;
> + IBND_ERROR("umad_recv failed: %d\n", rc);
> + return -1;
> + }
> +
> + rc = process_smp_queue(engine);
> +
> + mad = umad_get_mad(umad);
> + trid = (uint32_t)mad_get_field64(mad, 0, IB_MAD_TRID_F);
> +
> + smp = (ibnd_smp_t *)cl_qmap_remove(&engine->smps_on_wire, trid);
> + if ((cl_map_item_t *)smp == cl_qmap_end(&engine->smps_on_wire)) {
> + IBND_ERROR("Failed to find matching smp for trid (%x)\n",
> + trid);
> + return -1;
> + }
> +
> + if (rc)
> + goto error;
> +
> + if ((status = umad_status(umad))) {
> + IBND_ERROR("umad (%s Attr 0x%x:%u) bad status %d; %s\n",
> + portid2str(&smp->path),
> + smp->rpc.attr.id, smp->rpc.attr.mod,
> + status, strerror(status));
> + } else if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F))) {
> + IBND_ERROR("mad (%s Attr 0x%x:%u) bad status 0x%x\n",
> + portid2str(&smp->path),
> + smp->rpc.attr.id, smp->rpc.attr.mod,
> + status);
> + } else
> + rc = smp->cb(engine, smp, mad, smp->cb_data);
> +
> +error:
> + free(smp);
> + engine->num_smps_outstanding--;
> + return (rc);
> +}
> +
> +void smp_engine_init(smp_engine_t * engine, struct ibmad_port *ibmad_port,
> + void * user_data, int max_smps_on_wire)
> +{
> + memset(engine, '\0', sizeof(*engine));
> + engine->ibmad_port = ibmad_port;
> + engine->user_data = user_data;
> + cl_qmap_init(&engine->smps_on_wire);
> + engine->num_smps_outstanding = 0;
> + engine->max_smps_on_wire = max_smps_on_wire;
> +}
> +
> +void smp_engine_destroy(smp_engine_t *engine)
> +{
> + cl_map_item_t *item;
> + ibnd_smp_t *smp;
> +
> + /* remove queued smps */
> + smp = get_smp(engine);
> + if (smp)
> + IBND_ERROR("outstanding SMP's\n");
> + for (/* */; smp; smp = get_smp(engine)) {
> + free(smp);
> + }
> +
> + /* remove smps from the wire queue */
> + item = cl_qmap_head(&engine->smps_on_wire);
> + if (item != cl_qmap_end(&engine->smps_on_wire))
> + IBND_ERROR("outstanding SMP's on wire\n");
> + for (/* */; item != cl_qmap_end(&engine->smps_on_wire);
> + item = cl_qmap_head(&engine->smps_on_wire)) {
> + cl_qmap_remove_item(&engine->smps_on_wire, item);
> + free(item);
> + }
> +
> + engine->num_smps_outstanding = 0;
> +}
> +
> +int process_mads(smp_engine_t *engine)
> +{
> + int rc = 0;
> + while (engine->num_smps_outstanding > 0) {
> + if ((rc = process_smp_queue(engine)) != 0)
> + return rc;
Is it really needed to run process_smp_queue() here (assuming that it is
already executed in issue_smp() and process_one_recv())? Or this is just
for handling process_one_recv() error case? If so wouldn't it better to
run it there?
Sasha
> + while (!cl_is_qmap_empty(&engine->smps_on_wire))
> + if ((rc = process_one_recv(engine)) != 0)
> + return rc;
> + }
> + return 0;
> +}
> +
> --
> 1.5.4.5
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] opensm/osm_subnet.h: cosmetics - fix description of event_plugin option
From: Sasha Khapyorsky @ 2010-04-13 10:56 UTC (permalink / raw)
To: Yevgeny Kliteynik; +Cc: Linux RDMA
In-Reply-To: <4BC449D2.3050500-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On 13:39 Tue 13 Apr , Yevgeny Kliteynik wrote:
>
> Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
> ---
Applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] opensm/osm_subnet.{c,h}: passing options to the event plugins
From: Sasha Khapyorsky @ 2010-04-13 10:58 UTC (permalink / raw)
To: Yevgeny Kliteynik; +Cc: Linux RDMA
In-Reply-To: <4BC44AC2.8010403-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On 13:43 Tue 13 Apr , Yevgeny Kliteynik wrote:
> Adding new option string: "event_plugin_options"
> A user can use this option to pass any plugin options.
>
> Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
Applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] opensm/osm_subnet.c: fixing some options to not "hot-swappable"
From: Sasha Khapyorsky @ 2010-04-13 11:03 UTC (permalink / raw)
To: Yevgeny Kliteynik; +Cc: Linux RDMA
In-Reply-To: <4BC44A23.8010509-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
On 13:40 Tue 13 Apr , Yevgeny Kliteynik wrote:
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: 7bit
> Return-Path: kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
> X-OriginalArrivalTime: 13 Apr 2010 10:40:45.0762 (UTC) FILETIME=[C6063A20:01CADAF5]
>
> The following options are not hot-swappable:
>
> transaction_timeout
> transaction_retries
> use_ucast_cache
>
> Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
Applied. Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] opensm/osm_sa_path_record.c: adding wrapper for pr_rcv_get_path_parms()
From: Yevgeny Kliteynik @ 2010-04-13 12:49 UTC (permalink / raw)
To: Sasha Khapyorsky; +Cc: Linux RDMA
Adding non-static wrapper function for pr_rcv_get_path_parms()
fuction to enable calling path record calculation function from
outside this file.
Signed-off-by: Yevgeny Kliteynik <kliteyn-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>
---
opensm/opensm/osm_sa_path_record.c | 12 ++++++++++++
1 files changed, 12 insertions(+), 0 deletions(-)
diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c
index c4c3f86..b168428 100644
--- a/opensm/opensm/osm_sa_path_record.c
+++ b/opensm/opensm/osm_sa_path_record.c
@@ -743,6 +743,18 @@ Exit:
return status;
}
+ib_api_status_t osm_get_path_params(IN osm_sa_t * sa,
+ IN const osm_port_t * p_src_port,
+ IN const osm_port_t * p_dest_port,
+ IN const uint16_t dlid_ho,
+ OUT osm_path_parms_t * p_parms)
+{
+ ib_path_rec_t pr;
+ memset(&pr, 0, sizeof(ib_path_rec_t));
+ return pr_rcv_get_path_parms(sa, &pr,
+ p_src_port, p_dest_port, dlid_ho, 0, p_parms);
+}
+
static void pr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t * p_src_port,
IN const osm_port_t * p_dest_port,
IN const ib_gid_t * p_dgid,
--
1.5.1.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [infiniband-diags] [1/3] support --diff in ibnetdiscover
From: Sasha Khapyorsky @ 2010-04-13 12:55 UTC (permalink / raw)
To: Al Chu; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1270659938.26381.39.camel-RLKWKRZIcZkVVsCFsIUZTRy+HRzXvqW9@public.gmane.org>
On 10:05 Wed 07 Apr , Al Chu wrote:
> Hi Sasha,
>
> This patch adds the default --diff support in ibnetdiscover.
>
> Al
>
> --
> Albert Chu
> chu11-i2BcT+NCU+M@public.gmane.org
> Computer Scientist
> High Performance Systems Division
> Lawrence Livermore National Laboratory
> Date: Wed, 31 Mar 2010 11:10:34 -0700
> From: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
> Subject: [PATCH] support --diff in ibnetdiscover
> Message-Id: <1270659341.26381.27.camel-RLKWKRZIcZkVVsCFsIUZTRy+HRzXvqW9@public.gmane.org>
> Mime-Version: 1.0
> Content-Transfer-Encoding: 7bit
>
>
> Signed-off-by: Albert Chu <chu11-i2BcT+NCU+M@public.gmane.org>
All three applied.Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [infiniband-diags] [0/3] support --diff and --diffcheck in ibnetdiscover
From: Sasha Khapyorsky @ 2010-04-13 13:08 UTC (permalink / raw)
To: Al Chu; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1270659929.26381.38.camel-RLKWKRZIcZkVVsCFsIUZTRy+HRzXvqW9@public.gmane.org>
Hi Al,
On 10:05 Wed 07 Apr , Al Chu wrote:
>
> Others on the list may wonder how this is different than just using the
> normal 'diff' tool. The differences I can think of are:
>
> 1) This checks differences in the network, not text. This is
> particularly important when lids, lmc, etc. are changed. Otherwise
> there are many differences in a normal diff output that aren't
> necessary.
>
> 2) This provides the appropriate "context" in the diff output, showing
> the appropriate system ids to allow a system administrator to identify
> ports on what switch have changed. Under normal diff output, you may
> not get that appropriate context of information. The system
> administrator can of course use options like --context in diff, but the
> goal is to make the diff output clear and concise, not outputting
> unnecessary junk.
>
> 3) As parallelization has been added into ibnetdisocver/libibnetdiscover
> this becomes more critical as output in ibnetdiscover/libibnetdiscover
> can be re-ordered. So a normal diff suddenly is non-functional.
I'm getting your arguments. And this remind me the question which was
already raised some time ago.
Would it be better to keep cache in a regular human readable
ibnetdiscover output format, so '--diff' will be usable not just against
cache, but also against a regular ibnetdiscover output files?
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [infiniband-diags] [0/3] support --diff and --diffcheck in ibnetdiscover
From: Sasha Khapyorsky @ 2010-04-13 13:10 UTC (permalink / raw)
To: Al Chu; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <20100413130803.GG10830@me>
On 16:08 Tue 13 Apr , Sasha Khapyorsky wrote:
>
> Would it be better to keep cache in a regular human readable
> ibnetdiscover output format, so '--diff' will be usable not just against
> cache, but also against a regular ibnetdiscover output files?
BTW, such implementation could be simplified by the fact that we have
already ibnetdiscover output parser in 'ibsim'.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox