* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Ira Weiny @ 2010-04-13 20:44 UTC (permalink / raw)
To: Ira Weiny
Cc: Sasha Khapyorsky,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413133826.00a8afc5.weiny2-i2BcT+NCU+M@public.gmane.org>
On Tue, 13 Apr 2010 13:38:26 -0700
Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
> This changes the logic. "num_smps_outstanding" is NOT the number on the wire, but it appears you have made it so. This is the number which will cause process_smp_queue to continue being called.
>
> If you are going to do this I think you need to change process_mads as well as process_one_recv. We discussed process_one_recv in the error case.
>
> What were you trying to fix?
Ok, I think I see. We should move cl_qmap_insert to after a successful umad_send and putting total_smps here is ok. But num_smps_outstanding should be put back I think.
Ira
>
> Ira
>
> On Tue, 13 Apr 2010 19:38:36 +0300
> Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
>
> >
> > send_smp() failure blocks whole discovery execution (it never returns).
> > Fix this by updating all outstanding SMPs related counters only after
> > successful MAD sending.
> >
> > Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
> > ---
> > infiniband-diags/libibnetdisc/src/query_smp.c | 8 ++++----
> > 1 files changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/infiniband-diags/libibnetdisc/src/query_smp.c b/infiniband-diags/libibnetdisc/src/query_smp.c
> > index b4322bc..08e3ef7 100644
> > --- a/infiniband-diags/libibnetdisc/src/query_smp.c
> > +++ b/infiniband-diags/libibnetdisc/src/query_smp.c
> > @@ -96,10 +96,12 @@ static int process_smp_queue(smp_engine_t * engine)
> > if (!smp)
> > return 0;
> >
> > - cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> > - (cl_map_item_t *) smp);
> > if ((rc = send_smp(smp, engine->ibmad_port)) != 0)
> > return rc;
> > + engine->num_smps_outstanding++;
> > + cl_qmap_insert(&engine->smps_on_wire, (uint32_t) smp->rpc.trid,
> > + (cl_map_item_t *) smp);
> > + engine->total_smps++;
> > }
> > return 0;
> > }
> > @@ -133,8 +135,6 @@ int issue_smp(smp_engine_t * engine, ib_portid_t * portid,
> > portid->sl = 0;
> > portid->qp = 0;
> >
> > - engine->total_smps++;
> > - engine->num_smps_outstanding++;
> > queue_smp(engine, smp);
> > return process_smp_queue(engine);
> > }
> > --
> > 1.7.0.4
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at http://*vger.kernel.org/majordomo-info.html
> >
>
>
> --
> Ira Weiny
> Math Programmer/Computer Scientist
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RDMA CM problems, ib_find_cached_gid() fails
From: Mike M @ 2010-04-13 22:26 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
I'm debugging an issue in trying to use the RDMA CM.
IPoIB is enabled, OpenSM running. I'm trying to use the "rping"
example program as a test. I get the RDMA_CM_EVENT_REJECTED on the
client.
Putting in lots of printk messages, I see that the server machine
indeed gets a connection request MAD, but that the
ib_find_cached_gid() call inside of cma_acquire_dev() fails.
Any ideas?
Thanks,
Mike
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: RDMA CM problems, ib_find_cached_gid() fails
From: Sean Hefty @ 2010-04-13 22:41 UTC (permalink / raw)
To: 'Mike M', linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <g2i3d2745531004131526gcfec6bc2pccbae15d55997e95-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
>Putting in lots of printk messages, I see that the server machine
>indeed gets a connection request MAD, but that the
>ib_find_cached_gid() call inside of cma_acquire_dev() fails.
>
>Any ideas?
What kernel version are you using, and are you trying to connect using IPv4 or
IPv6 addresses?
Can you look at the GID passed into ib_find_cached_gid() and compare it against
the GIDs available in the system?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: two questions about RDMA-WRITE
From: Ding Dinghua @ 2010-04-14 2:15 UTC (permalink / raw)
To: Steve Wise; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <4BC484AC.80804-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
在 2010年4月13日 下午10:50，Steve Wise <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> 写道：
> 丁定华 wrote:
>> Hi guys:
>> I'm doing a project using RDMA to transfer data between servers. I'm
>> using RDMA_PS_TCP type and do RDMA_WRITE operations from client node
>> to server node. While the program succeed in setting up connection and
>> transferring data, I 'm still very confused about two problems:
>> (1) After transfer the data using IB_WR_RDMA_WRITE , the complete
>> queue entry found that the opcode turned from IB_WR_RDMA_WRITE to
>> IB_WR_RDMA_WRITE_WITH_IMM, I don't know the meaning of this opcode and
>> the reason of the change.
>>
>
> The opcode in the completion is of type enum ib_wc_opcode, not enum
> ib_wr_opcode. You are mis-interpreting the opcode in the completion.
>
Oh, yes, I think I made a mistake, thanks.
>> (2) It takes a long time to finish rdma_disconnect work (about 10
>> seconds), so is it a reasonable time?
>>
>
> What device are you using?
I'm using "Mellanox Technologies MT25204"
>
>
> Steve.
>
--
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: two questions about RDMA-WRITE
From: Ding Dinghua @ 2010-04-14 3:56 UTC (permalink / raw)
To: Sean Hefty; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <77BFFFE3D9C34D64A03DA5A9DC22F42D-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org>
On Tue, Apr 13, 2010 at 08:48:42AM -0700, Sean Hefty wrote:
> >(2) It takes a long time to finish rdma_disconnect work (about 10
> >seconds), so is it a reasonable time?
>
> This is long. How is your disconnect code structured?
>
Thanks for your reply.The project is mirror fs-metadata to another node in
HA-system, and IB is selected for its low latency.Since I'm a 100% newbie in IB development,
I think the problem may be caused by something very detail, just like problem (1).
so I paste all the connection-related function of my IB-transfer sample code(in kernel space),
Please have a look if you guys have time.
BTW: is there any docs about ib-core in kernel space? It's rather hard for me to understand
the code.
====
#include <linux/delay.h>
#include <linux/wait.h>
#include <linux/bio.h>
#include <rdma/ib_verbs.h>
#include <rdma/rdma_cm.h>
#define TEST_PAGES 16
#define RECV_PAGES 4096
#define JM_MAGIC 0xa0b0c0d0
struct connect_header {
uint32_t connect_magic;
uint32_t mirror_size; /* in 4k */
uint64_t addr;
uint32_t rkey;
};
struct jm_ib_device {
struct list_head conn_list;
struct ib_device *dev;
struct ib_pd *pd;
struct ib_mr *mr;
int max_sge;
unsigned int max_wrs;
spinlock_t spinlock; /* protects all above */
};
struct jm_send_ctx {
struct ib_sge s_sge[64];
u64 s_offset;
int s_size;
int s_done;
wait_queue_head_t s_wait;
};
struct jm_rdma_conn {
int jc_incoming;
struct rdma_cm_id *jc_id;
struct ib_pd *jc_pd;
struct ib_mr *jc_mr;
struct ib_mr *jc_map_mr;
struct ib_cq *jc_cq;
struct ib_qp *jc_qp;
int jc_async_rc;
struct completion jc_done;
wait_queue_head_t jc_connect_wait;
int jc_connstate;
struct sockaddr_in jc_remoteaddr;
int jc_pb_nsegs;
int jc_pb_segsize;
u64 jc_pb_addrs[64];
u32 jc_pb_rkey;
struct page *jc_pages[RECV_PAGES];
unsigned long jc_mappings[RECV_PAGES];
int jc_page_count;
struct list_head list;
};
#define RDMA_RESOLVE_TIMEOUT (5000)
#define RDMA_CONNECT_RETRY_MAX (2)
#define JM_RDMA_MAX_DATA_SEGS (16)
#define JM_RDMA_PORT (18111) /* randomly chosen */
static int jm_disconnect(struct jm_rdma_conn *conn);
static void jm_conn_close(struct jm_rdma_conn *conn);
static int jm_handle_connect_req(struct rdma_cm_id *id,
struct jm_rdma_conn **rconn, uint32_t mirror_size);
static struct rdma_cm_id *jm_listen_id = NULL;
static struct jm_ib_device *jm_ibdev = NULL;
static struct ib_client jm_ib_client = {
.name = "jm_ib",
.add = jm_ib_add_one,
.remove = jm_ib_remove_one,
};
static void jm_cq_comp_handler(struct ib_cq *cq, void *context) {
struct jm_rdma_conn *conn = context;
struct ib_wc wc;
struct jm_send_ctx *send;
/* No idea why it should be called twice. */
printk("cq comp for id %p\n", conn->jc_id);
ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
while (ib_poll_cq(cq, 1, &wc) == 1) {
if (wc.opcode != IB_WC_RDMA_WRITE) {
printk("completed unknown opcode %d\n", wc.opcode);
/* continue; */
}
send = (struct jm_send_ctx *)wc.wr_id;
printk("got send=%p\n", send);
printk("completed RDMA_WRITE of IO(%Lu, %u)\n",
send->s_offset, send->s_size);
send->s_done = wc.status == IB_WC_SUCCESS ? 1 : -EIO;
wake_up_all(&send->s_wait);
}
ib_req_notify_cq(cq, IB_CQ_NEXT_COMP);
}
static void jm_cq_event_handler(struct ib_event *cause, void *context) {
printk(KERN_ERR "got cq event %d\n", cause->event);
}
static void jm_qp_event_handler(struct ib_event *event, void *data) {
switch (event->event) {
case IB_EVENT_PATH_MIG:
case IB_EVENT_COMM_EST:
case IB_EVENT_SQ_DRAINED:
case IB_EVENT_QP_LAST_WQE_REACHED:
printk("got QP event %d received for QP=%p\n",
event->event, event->element.qp);
break;
/* These are considered fatal events */
case IB_EVENT_PATH_MIG_ERR:
case IB_EVENT_QP_FATAL:
case IB_EVENT_QP_REQ_ERR:
case IB_EVENT_QP_ACCESS_ERR:
case IB_EVENT_DEVICE_FATAL:
default:
printk("got QP ERROR event %d for QP=%p\n",
event->event, event->element.qp);
break;
}
}
static int jm_setup_qp(struct jm_rdma_conn *conn, int outgoing) {
struct ib_qp_init_attr iattr;
int ret = 0;
/* protection domain and memory region */
conn->jc_mr = jm_ibdev->mr;
conn->jc_pd = jm_ibdev->pd;
/* create completion queue */
conn->jc_cq = ib_create_cq(conn->jc_id->device,
jm_cq_comp_handler,
jm_cq_event_handler,
conn, 16, 0);
if (IS_ERR(conn->jc_cq)) {
ret = PTR_ERR(conn->jc_cq);
conn->jc_cq = NULL;
printk("create cq failed: %d\n", ret);
return ret;
}
if (outgoing)
ret = ib_req_notify_cq(conn->jc_cq, IB_CQ_NEXT_COMP);
else
ret = ib_req_notify_cq(conn->jc_cq, IB_CQ_SOLICITED);
if (ret) {
printk("notify cq failed: %d\n", ret);
goto out_destroy_cq;
}
/* create queue pair */
memset(&iattr, 0, sizeof(iattr));
if (outgoing) {
iattr.cap.max_send_wr = 16;
iattr.cap.max_send_sge = 16;
iattr.cap.max_recv_wr = 0;
iattr.cap.max_recv_sge = 0;
} else {
iattr.cap.max_send_wr = 0;
iattr.cap.max_send_sge = 0;
iattr.cap.max_recv_wr = 16;
iattr.cap.max_recv_sge = 16;
}
iattr.send_cq = conn->jc_cq;
iattr.recv_cq = conn->jc_cq;
iattr.sq_sig_type = IB_SIGNAL_REQ_WR;
iattr.qp_type = IB_QPT_RC;
iattr.event_handler = jm_qp_event_handler;
iattr.qp_context = conn;
ret = rdma_create_qp(conn->jc_id, conn->jc_pd, &iattr);
if (ret) {
printk("create qp failed: %d\n", ret);
goto out_destroy_cq;
}
conn->jc_qp = conn->jc_id->qp;
printk("setup qp done\n");
return 0;
out_destroy_cq:
ib_destroy_cq(conn->jc_cq);
conn->jc_cq = NULL;
return ret;
}
static int jm_allocate_and_map_mr(struct jm_rdma_conn *conn,
uint32_t mirror_size, struct connect_header *ch) {
struct ib_phys_buf *ibp = NULL;
struct page **buf_pages = NULL;
u64 local_addr, addr;
int i = 0, ret = -ENOMEM;
buf_pages = kmalloc(sizeof(struct page *)*mirror_size, GFP_KERNEL);
if (!buf_pages)
goto out_free;
ibp = kmalloc(sizeof(struct ib_phys_buf)*mirror_size, GFP_KERNEL);
if (!ibp)
goto out_free;
for (i = 0; i < mirror_size; i++) {
buf_pages[i] = alloc_page(GFP_KERNEL);
if (!buf_pages[i])
goto out_free;
addr = ib_dma_map_page(conn->jc_id->device, buf_pages[i],
0, PAGE_SIZE, DMA_FROM_DEVICE);
if (ib_dma_mapping_error(conn->jc_id->device, addr)) {
__free_page(buf_pages[i]);
goto out_free;
}
ibp[i].addr = addr;
ibp[i].size = PAGE_SIZE;
}
/* call ib_dma_mapping_error to check for error */
local_addr = ibp[0].addr;
conn->jc_map_mr = ib_reg_phys_mr(conn->jc_pd, &ibp[0], mirror_size,
IB_ACCESS_REMOTE_WRITE |
IB_ACCESS_LOCAL_WRITE,
&local_addr);
if (IS_ERR(conn->jc_map_mr)) {
ret = PTR_ERR(conn->jc_map_mr);
conn->jc_map_mr = NULL;
printk("get DMA mr failed: %d\n", ret);
goto out_free;
}
conn->jc_pb_nsegs = 1;
conn->jc_pb_segsize = PAGE_SIZE * mirror_size;
conn->jc_pb_addrs[0] = local_addr;
conn->jc_pb_rkey = conn->jc_map_mr->rkey;
ch->addr = local_addr;
ch->rkey = conn->jc_map_mr->rkey;
for (i = 0; i < mirror_size; i++) {
conn->jc_pages[i] = buf_pages[i];
conn->jc_mappings[i] = ibp[i].addr;
conn->jc_page_count++;
}
return 0;
out_free:
printk("map mr failed at %d\n", i);
for (i--; i >= 0; i--) {
ib_dma_unmap_single(conn->jc_id->device, ibp[i].addr,
PAGE_SIZE, DMA_FROM_DEVICE);
__free_page(buf_pages[i]);
}
if (buf_pages)
kfree(buf_pages);
if (ibp)
kfree(ibp);
return ret;
}
static int jm_rdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) {
struct jm_rdma_conn *conn = id->context;
struct ib_qp_init_attr iattr;
struct ib_qp_attr attr;
int connstate = 0, ret = 0;
struct connect_header *ch = NULL;
printk("event %d comes in for id %p\n", event->event, id);
switch (event->event) {
case RDMA_CM_EVENT_CONNECT_REQUEST:
ch = (struct connect_header*)event->param.conn.private_data;
if (!ch || (ch->connect_magic != JM_MAGIC)) {
printk("Connect request error.\n");
ret = -EINVAL;
break;
}
/* in this case, it is a newly allocated cm_id */
printk("Connect request, event=%d, mirror_size=%d\n", event->event,
ch->mirror_size);
ret = jm_handle_connect_req(id, &conn, ch->mirror_size);
break;
case RDMA_CM_EVENT_ADDR_RESOLVED:
case RDMA_CM_EVENT_ROUTE_RESOLVED:
conn->jc_async_rc = 0;
complete(&conn->jc_done);
break;
case RDMA_CM_EVENT_ADDR_ERROR:
conn->jc_async_rc = -EHOSTUNREACH;
printk("CM address resolution error\n");
complete(&conn->jc_done);
break;
case RDMA_CM_EVENT_ROUTE_ERROR:
conn->jc_async_rc = -ENETUNREACH;
printk("CM route resolution error\n");
complete(&conn->jc_done);
break;
case RDMA_CM_EVENT_ESTABLISHED:
connstate = 1;
ib_query_qp(id->qp, &attr,
IB_QP_MAX_QP_RD_ATOMIC | IB_QP_MAX_DEST_RD_ATOMIC,
&iattr);
printk("%d responder resources (%d initiator)\n",
attr.max_dest_rd_atomic, attr.max_rd_atomic);
jm_add_conn_to_list(conn);
goto connected;
case RDMA_CM_EVENT_CONNECT_ERROR:
connstate = -ENOTCONN;
goto connected;
case RDMA_CM_EVENT_UNREACHABLE:
connstate = -ENETDOWN;
goto connected;
case RDMA_CM_EVENT_REJECTED:
connstate = -ECONNREFUSED;
goto connected;
case RDMA_CM_EVENT_DISCONNECTED:
connstate = -ECONNABORTED;
goto connected;
case RDMA_CM_EVENT_DEVICE_REMOVAL:
printk("Device removal id=%p\n", id);
connstate = -ENODEV;
connected:
printk("%pI4:%u (event 0x%x)\n",
&conn->jc_remoteaddr.sin_addr.s_addr,
ntohs(conn->jc_remoteaddr.sin_port),
event->event << 11);
conn->jc_connstate = connstate;
wake_up_all(&conn->jc_connect_wait);
break;
default:
printk("unexpected CM event %d on id %p\n", event->event, id);
break;
}
return ret;
}
static int jm_conn_open(struct jm_rdma_conn *conn, struct sockaddr *addr) {
struct rdma_cm_id *id;
int ret = 0;
init_completion(&conn->jc_done);
/* create interface device */
id = rdma_create_id(jm_rdma_cm_event_handler, conn, RDMA_PS_TCP);
if (IS_ERR(id)) {
ret = PTR_ERR(id);
printk("create RDMA id failed: %d\n", ret);
goto out;
}
conn->jc_id = id;
conn->jc_async_rc = -ETIMEDOUT;
ret = rdma_resolve_addr(id, NULL, addr, RDMA_RESOLVE_TIMEOUT);
if (ret) {
printk("RDMA resolve addr failed: %d\n", ret);
goto out_destroy_id;
}
wait_for_completion_interruptible_timeout(&conn->jc_done,
msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT) + 1);
ret = conn->jc_async_rc;
if (ret)
goto out_destroy_id;
conn->jc_remoteaddr = *(struct sockaddr_in *)addr;
conn->jc_async_rc = -ETIMEDOUT;
ret = rdma_resolve_route(id, RDMA_RESOLVE_TIMEOUT);
if (ret) {
printk("RDMA resolve route failed: %d\n", ret);
goto out_destroy_id;
}
wait_for_completion_interruptible_timeout(&conn->jc_done,
msecs_to_jiffies(RDMA_RESOLVE_TIMEOUT) + 1);
ret = conn->jc_async_rc;
if (ret)
goto out_destroy_id;
printk("open conn ok\n");
return 0;
out_destroy_id:
rdma_destroy_id(conn->jc_id);
conn->jc_id = NULL;
out:
return ret;
}
static void jm_conn_close(struct jm_rdma_conn *conn) {
if (conn->jc_qp) {
rdma_destroy_qp(conn->jc_id);
conn->jc_qp = NULL;
}
if (conn->jc_map_mr) {
int i = conn->jc_page_count - 1;
for (; i >= 0; i--) {
ib_dma_unmap_single(conn->jc_id->device,
conn->jc_mappings[i],
PAGE_SIZE, DMA_FROM_DEVICE);
conn->jc_mappings[i] = 0;
__free_page(conn->jc_pages[i]);
conn->jc_pages[i] = NULL;
conn->jc_page_count--;
}
ib_dereg_mr(conn->jc_map_mr);
conn->jc_map_mr = NULL;
}
if (conn->jc_cq) {
ib_destroy_cq(conn->jc_cq);
conn->jc_cq = NULL;
}
if (conn->jc_id) {
rdma_destroy_id(conn->jc_id);
conn->jc_id = NULL;
}
}
static int jm_connect(struct jm_rdma_conn *conn) {
struct rdma_conn_param conn_param;
struct connect_header header;
int ret;
ret = jm_setup_qp(conn, 1);
if (ret)
goto out;
/* connect server */
init_waitqueue_head(&conn->jc_connect_wait);
conn->jc_connstate = 0;
memset(&conn_param, 0, sizeof(conn_param));
memset(&header, 0, sizeof(header));
header.connect_magic = JM_MAGIC;
header.mirror_size = RECV_PAGES;
conn_param.private_data = &header;
conn_param.private_data_len = sizeof(header);
conn_param.initiator_depth = 1;
conn_param.retry_count = 7;
ret = rdma_connect(conn->jc_id, &conn_param);
if (ret) {
printk("RDMA connect failed: %d\n", ret);
goto out;
}
wait_event_interruptible(conn->jc_connect_wait,
conn->jc_connstate != 0);
if (conn->jc_connstate < 0)
ret = conn->jc_connstate;
out:
return ret;
}
static int jm_disconnect(struct jm_rdma_conn *conn) {
struct ib_wc wc;
int flush_count = 0;
int ret;
if (conn->jc_connstate <= 0)
return 0;
while (ib_poll_cq(conn->jc_cq, 1, &wc) == 1)
++flush_count;
printk("id(%p) is to disconnect, %d events flushed\n",
conn->jc_id, flush_count);
ret = rdma_disconnect(conn->jc_id);
if (ret) {
printk("unable to perform disconnect: %d\n", ret);
conn->jc_connstate = ret;
} else {
wait_event_interruptible(conn->jc_connect_wait,
conn->jc_connstate != 1);
printk("id(%p) after disconnect, connstate is %d\n",
conn->jc_id, conn->jc_connstate);
}
return ret;
}
/* receiver side */
static int jm_handle_connect_req(struct rdma_cm_id *id,
struct jm_rdma_conn **rconn, uint32_t mirror_size) {
struct rdma_conn_param conn_param;
struct jm_rdma_conn *conn;
int ret = 0, destroy = 1;
struct connect_header ch;
memset(&ch, 0, sizeof(ch));
if (mirror_size > RECV_PAGES) {
ret = -EINVAL;
goto out_reject;
}
conn = kzalloc(sizeof(*conn), GFP_KERNEL);
if (!conn) {
ret = -ENOMEM;
goto out_reject;
}
init_waitqueue_head(&conn->jc_connect_wait);
conn->jc_connstate = 0;
conn->jc_id = id;
id->context = conn;
ret = jm_setup_qp(conn, 0);
if (ret) {
kfree(conn);
conn = NULL;
goto out_reject;
}
memset(&conn_param, 0, sizeof(conn_param));
/* XXX tune these? */
conn_param.responder_resources = 1;
conn_param.initiator_depth = 1;
conn_param.retry_count = 7;
conn_param.rnr_retry_count = 7;
ch.connect_magic = JM_MAGIC;
ch.mirror_size = mirror_size;
conn_param.private_data = &ch;
conn_param.private_data_len = sizeof(ch);
printk("allocating memory and map it for receiver\n");
if ((ret = jm_allocate_and_map_mr(conn, mirror_size, &ch))) {
printk("failed to allocate and map mr: %d\n", ret);
goto out_reject;
}
/* rdma_accept() calls rdma_reject() internally if it fails */
ret = rdma_accept(id, &conn_param);
if (!ret)
conn->jc_connstate = 1;
conn->jc_incoming = 1;
*rconn = conn;
return 0;
out_reject:
rdma_reject(id, NULL, 0);
return destroy;
}
--
Ding Dinghua
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Håkon Bugge @ 2010-04-14 5:18 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Roland Dreier, Jason Gunthorpe, Andrew Morton, Eric B Munson,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
pavel-+ZI9xUNit7I, mingo-X9Un+BFzKDI,
jsquyres-FYB4Gu1CFyUAvxtiuMwx3w
In-Reply-To: <1271181774.4807.1868.camel@twins>
On Apr 13, 2010, at 20:02 , Peter Zijlstra wrote:
>
> Yeah, virtual<->physical maps can change through swapping, page
> migration, memory compaction, huge-page aggregation (the latter two not
> yet being upstream).
Assuming this holds true, RDMA will not work. And with no RDMA, we do not need ummunotify. Is that your argument?
Seriously, RDMA requires the virtual to physical mapping to remain constant for a period of time. And that time period is from memory registration to de-registration. If the virtual to physical mapping changes in that period for the registered memory area, I can't see how an HCA can handle that.
For MPI applications, the MPI API defines that the buffers used in communication cannot be changed or freed while a non-blocking transfer is in progress. So far, we are good. But memory registration (and in particular de-registration) is a costly process. Since MPI is about performance, the MPI library would like to re-use earlier memory registrations for other transfers. The MPI library records earlier memory mappings (VA+bound) and checks if the VA+bound of a new transfer is already contained in memory revisions registered earlier.
The problem with this approach is that _normal_ activity _between_ the transfers changes the virtual to physical mapping and ruins previous memory registrations. The problem is to catch these. I simply argue that they should be caught the simplest possible way; a call-back from the system calls affecting the virtual to physical mapping.
>
> Even mlock() doesn't pin virtual<->physical maps.
Can you elaborate on what you mean here?
Thanks, Håkon
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Andrea Gozzelino @ 2010-04-14 8:51 UTC (permalink / raw)
To: Tung, Chien Tin, Steve Wise
Cc: Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <2EFBCAEF10980645BBCFB605689E08E9047C7405E9-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
wrote:
> >>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
> >>> code
> >>> and it appears to do nothing in the map_phys_fmr functions.
> >>>
> >>
> >> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
> >>
> >No, but SDP uses FMRs. I don't think it will run without FMR support.
> >
>
>
> Good to know. Thanks.
>
> Chien
>
>
Hi Steve and Chien,
I understand that NE020 cards have problem with SDP connected with
map_phy_fmr (FMR stands for Fast Memory Region).
Is it possible to solve/fix this point?
If yes, have you an idea about the time that is necessary to code
development/build?
If no, can you suggest me a card that supports SDP protocol?
I work on NE020 cards from February 2010 for an INFN experimental
proposal, called REDIGO (Read out at 10 Gbits/s), about the data
acquisition and movement systems. The covergence of storage protocols
around 10 Gigabits/s Ethernet protocols shows that one way could be the
Remote Direct Memory Access (RDMA). The goals are the investigations of
latency time, the throughput, the buffer size schemes and finally the
global event building bandwidth.
Do you know if NE020 cards have problems with librdma (RDMA procedures,
in general) and / or with MPI versions?
Thank you very much,
Andrea
Andrea Gozzelino
INFN - Laboratori Nazionali di Legnaro (LNL)
Viale dell'Universita' 2
I-35020 - Legnaro (PD)- ITALIA
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Gleb Natapov @ 2010-04-14 8:52 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Roland Dreier, Håkon Bugge, Jason Gunthorpe, Andrew Morton,
Eric B Munson, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
pavel-+ZI9xUNit7I, mingo-X9Un+BFzKDI,
jsquyres-FYB4Gu1CFyUAvxtiuMwx3w
In-Reply-To: <1271181774.4807.1868.camel@twins>
On Tue, Apr 13, 2010 at 08:02:54PM +0200, Peter Zijlstra wrote:
> On Tue, 2010-04-13 at 10:57 -0700, Roland Dreier wrote:
> > Are those system calls the only possible way that virtual to physical
> > mappings can change? Can't page migration or something like that
> > potentially affect things? And even if you did have hooks into every
> > system call that mattered (keep in mind that relying on glibc is not
> > enough, since an MPI application may not use glibc) would decoding them
> > and figuring out what happened really be preferable to a single event
> > type that tells you exactly what address range was affected?
>
> Yeah, virtual<->physical maps can change through swapping, page
> migration, memory compaction, huge-page aggregation (the latter two not
> yet being upstream).
>
> Even mlock() doesn't pin virtual<->physical maps.
Pages registered for RDMA are GUPed so no method above should touch
them. Fork+cow or unmap/map on the other hand can change
virtual<->physical maps. GUPed pages are still GUPed, but they are no
longer mapped into process' virtual address space. MPI copes with
Fork+cow by marking registered memory as MADV_DONTFORK.
--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Gleb Natapov @ 2010-04-14 9:06 UTC (permalink / raw)
To: Roland Dreier
Cc: Håkon Bugge, Jason Gunthorpe, Andrew Morton, Eric B Munson,
linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI, jsquyres-FYB4Gu1CFyUAvxtiuMwx3w
In-Reply-To: <adahbnfme0j.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org>
On Tue, Apr 13, 2010 at 10:57:32AM -0700, Roland Dreier wrote:
> > It is further claimed that "… other tricks are not robust". I wrote
> > the code used in Scali/Platform MPI handling the issue. I do not
> > think its fair to claim that this MPI is not robust in this matter
> > nor that is performance is bad.
>
> The Open MPI developers have spent a lot of effort trying to handle this
> purely in userspace and still do not believe that a truly robust
> solution is possible without kernel help. Perhaps they can expand on
> what the obstacles are.
>
The problem is that glibc doesn't provide correct type of hooks for MPI
to use. You can hook into free(), but the hook is called when
application frees memory, not when memory is returned back to the kernel
and since MPI wants to cache registration across free()/malloc() if
possible those hooks are not good enough. To overcome this MPI tries to
provide its own memory management library (luckily glibc defines
most/all memory management functions as weak) with proper hooks present, but
that poses a whole lot of other problems and memory management is really
not MPI's job. Even if glibc will provide proper hooks some day HPC users
may want to use other, more performance oriented, memory management
libraries instead of using build in glibc one. Relying on glibc hooks
will prevent them from doing so.
--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [infiniband-diags] [0/3] support --diff and --diffcheck in ibnetdiscover
From: Sasha Khapyorsky @ 2010-04-14 9:54 UTC (permalink / raw)
To: Al Chu; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
In-Reply-To: <1271179075.17987.94.camel-X2zTWyBD0EhliZ7u+bvwcg@public.gmane.org>
On 10:17 Tue 13 Apr , Al Chu wrote:
>
> I had considered this at one point. There were several reasons I
> decided to go w/ the cache idea. Perhaps the major reason is that the
> current cache system has "all" the data (nodeinfo, portinfo, etc.)
> saved, whereas the normal ibnetdiscover output does not. So we would be
> limited in our diff output for what ibnetdiscover outputs (or perhaps
> limited in future extensions). Adding a --diff into iblinkinfo is also
> on my list to do, but that wouldn't be possible w/ a cached text output
> (the cached text file doesn't store portinfo data).
That makes sense.
> This isn't to say there are downsides. The major downside is that it's
> difficult to edit the cache when minor changes happen on the network
> (e.g. for our system administrators, when an HCA is replaced b/c a node
> dies). I have a tool for that (to be submitted soon too I hope :-).
It is difficult to edit and difficult to read. So I would really prefer
to have a text file (even with extended format). But well, let's see how
this will go.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [PATCH v3 1/2] libibnetdisc: Convert to a multi-smp algorithm
From: Sasha Khapyorsky @ 2010-04-14 9:58 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413133028.b55a0cb1.weiny2-i2BcT+NCU+M@public.gmane.org>
On 13:30 Tue 13 Apr , Ira Weiny wrote:
>
> If we are going to do something like this why not more like a context?
>
> Something like this?
>
> ibqueryerrors.c
> query_errors_ibmad_port = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 4);
>
> {
> ibnd_context_t ctx;
> ctx.ca_name = ibd_ca;
> ctx.ca_port = ibd_ca_port;
> ctx.portid = portid;
> ctx.hops = hops;
> fabric = ibnd_discover_fabric(&ctx); /* opens it's own ibmad_port */
> }
> ...
I didn't think about "context"-like, more as about few mandatory
(explicit) parameters + optional config.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] libibnetdisc: don't query CA ports not connected to a fabric
From: Sasha Khapyorsky @ 2010-04-14 9:59 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413112412.de66586d.weiny2-i2BcT+NCU+M@public.gmane.org>
We can save some amount of MADs by not querying CA/Router ports which
is not connected to our fabric. When discovery reaches CA or Router
node it will always get PortInfo for a port which was discovered and
not others.
Signed-off-by: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/w@public.gmane.org>
---
infiniband-diags/libibnetdisc/src/ibnetdisc.c | 21 ++++++++++-----------
1 files changed, 10 insertions(+), 11 deletions(-)
diff --git a/infiniband-diags/libibnetdisc/src/ibnetdisc.c b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
index 2ce15b7..51b36e5 100644
--- a/infiniband-diags/libibnetdisc/src/ibnetdisc.c
+++ b/infiniband-diags/libibnetdisc/src/ibnetdisc.c
@@ -339,19 +339,18 @@ static int recv_node_info(smp_engine_t * engine, ibnd_smp_t * smp,
link_ports(node, port, rem_node, rem_node->ports[rem_port_num]);
}
- if (!node_is_new)
- return 0;
-
- query_node_desc(engine, &smp->path, node);
+ if (node_is_new) {
+ query_node_desc(engine, &smp->path, node);
- if (node->type == IB_NODE_SWITCH)
- query_switch_info(engine, &smp->path, node);
-
- /* process all the ports on this node */
- for (i = (node->type == IB_NODE_SWITCH) ? 0 : 1;
- i <= node->numports; i++) {
- query_port_info(engine, &smp->path, node, i);
+ if (node->type == IB_NODE_SWITCH) {
+ query_switch_info(engine, &smp->path, node);
+ for (i = 0; i <= node->numports; i++)
+ query_port_info(engine, &smp->path, node, i);
+ }
}
+
+ if (node->type != IB_NODE_SWITCH)
+ query_port_info(engine, &smp->path, node, port_num);
return 0;
}
--
1.7.0.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: [PATCH] libibnetdisc: fix outstanding SMPs countung
From: Sasha Khapyorsky @ 2010-04-14 10:23 UTC (permalink / raw)
To: Ira Weiny
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
Hal Rosenstock
In-Reply-To: <20100413134446.72eb336a.weiny2-i2BcT+NCU+M@public.gmane.org>
On 13:44 Tue 13 Apr , Ira Weiny wrote:
>
> > This changes the logic. "num_smps_outstanding" is NOT the number on the wire, but it appears you have made it so.
Actually yes, it made it so.
> > This is the number which will cause process_smp_queue to continue being called.
> >
> > If you are going to do this I think you need to change process_mads as well as process_one_recv. We discussed process_one_recv in the error case.
process_one_recv() failure breaks the loop anyway.
> > What were you trying to fix?
>
> Ok, I think I see. We should move cl_qmap_insert to after a successful umad_send and putting total_smps here is ok. But num_smps_outstanding should be put back I think.
But then it blocks process_mads() to loop forever after single
send_smp() failure (with all empty queues and umad_recv() running
without timeout).
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH] amso1100: Add missing memset
From: Vladimir Sokolovsky @ 2010-04-14 14:20 UTC (permalink / raw)
To: rdreier-FYB4Gu1CFyUAvxtiuMwx3w; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Signed-off-by: Vladimir Sokolovsky <vlad-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
drivers/infiniband/hw/amso1100/c2_rnic.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/drivers/infiniband/hw/amso1100/c2_rnic.c b/drivers/infiniband/hw/amso1100/c2_rnic.c
index dd05c48..bac9856 100644
--- a/drivers/infiniband/hw/amso1100/c2_rnic.c
+++ b/drivers/infiniband/hw/amso1100/c2_rnic.c
@@ -155,6 +155,8 @@ static int c2_rnic_query(struct c2_dev *c2dev, struct ib_device_attr *props)
if (err)
goto bail2;
+ memset(props, 0, sizeof *props);
+
props->fw_ver =
((u64)be32_to_cpu(reply->fw_ver_major) << 32) |
((be32_to_cpu(reply->fw_ver_minor) & 0xFFFF) << 16) |
--
1.7.0.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH V4 0/2] Add support for enhanced atomic operations
From: Vladimir Sokolovsky @ 2010-04-14 14:22 UTC (permalink / raw)
To: rdreier-FYB4Gu1CFyUAvxtiuMwx3w; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Hi Roland,
This patchset adds support for the following enhanced atomic
operations:
- Masked atomic compare and swap
- Masked atomic fetch and add
These operations enable using a smaller amount of memory when using
multiple locks by using portions of a 64 bit value in an atomic
operation.
For some applications the memory savings are very significant. One
example is fine grain lock implementations for huge data sets. In
other cases, the benefit is the ability to update multiple fields with
a single io operation.
Vladimir Sokolovsky(2):
IB/core: Add support for enhanced atomic operations
mlx4/IB: Add support for enhanced atomic operations
changes from V3:
- patch #1:
Add a new capability masked_atomic_cap instead of using
IB_DEVICE_MASKED_ATOMIC
- patch #2:
Set props->masked_atomic_cap to IB_ATOMIC_HCA for mlx4
Note that all drivers clear props with memset except amso1100. So, no
need to define props->masked_atomic_cap to NONE for all drivers.
I will send a patch that fixes amso1100 issue.
Regards,
Vladimir
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH V4 1/2] IB/core: Add support for enhanced atomic operations
From: Vladimir Sokolovsky @ 2010-04-14 14:23 UTC (permalink / raw)
To: rdreier-FYB4Gu1CFyUAvxtiuMwx3w; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
- Add a new IB_WR_MASKED_ATOMIC_CMP_AND_SWP and IB_WR_MASKED_ATOMIC_FETCH_AND_ADD send
opcodes that can be used to mark a "masked atomic compare and swap" and
"masked atomic fetch and add" work request correspondingly.
- Add masked_atomic_cap capability.
- Add mask fields to atomic struct of ib_send_wr
- Add new opcodes to ib_wc_opcode
Masked Compare and Swap (MskCmpSwap)
The MskCmpSwap atomic operation is an extension to the CmpSwap operation
defined in the IB spec. MskCmpSwap allows the user to select a portion of the
64 bit target data for the “compare” check as well as to restrict the swap to a
(possibly different) portion. The pseudo code below describes the operation:
| atomic_response = *va
| if (!((compare_add ^ *va) & compare_add_mask)) then
| *va = (*va & ~(swap_mask)) | (swap & swap_mask)
|
| return atomic_response
The additional operands are carried in the Extended Transport Header. Atomic
response generation and packet format for MskCmpSwap is as for standard IB
Atomic operations.
Masked Fetch and Add (MFetchAdd)
The MFetchAdd Atomic operation extends the functionality of the standard IB
FetchAdd by allowing the user to split the target into multiple fields of
selectable length. The atomic add is done independently on each one of this
fields. A bit set in the field_boundary parameter specifies the field
boundaries. The pseudo code below describes the operation:
| bit_adder(ci, b1, b2, *co)
| {
| value = ci + b1 + b2
| *co = !!(value & 2)
|
| return value & 1
| }
|
| #define MASK_IS_SET(mask, attr) (!!((mask)&(attr)))
| bit_position = 1
| carry = 0
| atomic_response = 0
|
| for i = 0 to 63
| {
| if ( i != 0 )
| bit_position = bit_position << 1
|
| bit_add_res = bit_adder(carry, MASK_IS_SET(*va, bit_position), MASK_IS_SET(compare_add, bit_position), &new_carry)
| if (bit_add_res)
| atomic_response |= bit_position
|
| carry = ((new_carry) && (!MASK_IS_SET(compare_add_mask, bit_position)))
| }
|
| return atomic_response
Signed-off-by: Vladimir Sokolovsky <vlad-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
include/rdma/ib_verbs.h | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 09509ed..8ef6858 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -136,6 +136,7 @@ struct ib_device_attr {
int max_qp_init_rd_atom;
int max_ee_init_rd_atom;
enum ib_atomic_cap atomic_cap;
+ enum ib_atomic_cap masked_atomic_cap;
int max_ee;
int max_rdd;
int max_mw;
@@ -467,6 +468,8 @@ enum ib_wc_opcode {
IB_WC_LSO,
IB_WC_LOCAL_INV,
IB_WC_FAST_REG_MR,
+ IB_WC_MASKED_COMP_SWAP,
+ IB_WC_MASKED_FETCH_ADD,
/*
* Set value of IB_WC_RECV so consumers can test if a completion is a
* receive by testing (opcode & IB_WC_RECV).
@@ -689,6 +692,8 @@ enum ib_wr_opcode {
IB_WR_RDMA_READ_WITH_INV,
IB_WR_LOCAL_INV,
IB_WR_FAST_REG_MR,
+ IB_WR_MASKED_ATOMIC_CMP_AND_SWP,
+ IB_WR_MASKED_ATOMIC_FETCH_AND_ADD,
};
enum ib_send_flags {
@@ -731,6 +736,8 @@ struct ib_send_wr {
u64 remote_addr;
u64 compare_add;
u64 swap;
+ u64 compare_add_mask;
+ u64 swap_mask;
u32 rkey;
} atomic;
struct {
--
1.7.0.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* [PATCH V4 2/2] mlx4/IB: Add support for enhanced atomic operations
From: Vladimir Sokolovsky @ 2010-04-14 14:23 UTC (permalink / raw)
To: rdreier-FYB4Gu1CFyUAvxtiuMwx3w; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
Added support for masked atomic operations:
- Masked Compare and Swap
- Masked Fetch and Add
Signed-off-by: Vladimir Sokolovsky <vlad-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
---
drivers/infiniband/hw/mlx4/cq.c | 8 ++++++++
drivers/infiniband/hw/mlx4/main.c | 1 +
drivers/infiniband/hw/mlx4/qp.c | 27 +++++++++++++++++++++++++++
include/linux/mlx4/device.h | 4 ++--
include/linux/mlx4/qp.h | 7 +++++++
5 files changed, 45 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c
index de5263b..783eae7 100644
--- a/drivers/infiniband/hw/mlx4/cq.c
+++ b/drivers/infiniband/hw/mlx4/cq.c
@@ -660,6 +660,14 @@ repoll:
wc->opcode = IB_WC_FETCH_ADD;
wc->byte_len = 8;
break;
+ case MLX4_OPCODE_MASKED_ATOMIC_CS:
+ wc->opcode = IB_WC_MASKED_COMP_SWAP;
+ wc->byte_len = 8;
+ break;
+ case MLX4_OPCODE_MASKED_ATOMIC_FA:
+ wc->opcode = IB_WC_MASKED_FETCH_ADD;
+ wc->byte_len = 8;
+ break;
case MLX4_OPCODE_BIND_MW:
wc->opcode = IB_WC_BIND_MW;
break;
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index e596537..9b2a526 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -138,6 +138,7 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
props->local_ca_ack_delay = dev->dev->caps.local_ca_ack_delay;
props->atomic_cap = dev->dev->caps.flags & MLX4_DEV_CAP_FLAG_ATOMIC ?
IB_ATOMIC_HCA : IB_ATOMIC_NONE;
+ props->masked_atomic_cap = IB_ATOMIC_HCA;
props->max_pkeys = dev->dev->caps.pkey_table_len[1];
props->max_mcast_grp = dev->dev->caps.num_mgms + dev->dev->caps.num_amgms;
props->max_mcast_qp_attach = dev->dev->caps.num_qp_per_mgm;
diff --git a/drivers/infiniband/hw/mlx4/qp.c b/drivers/infiniband/hw/mlx4/qp.c
index 2a97c96..2ae8c81 100644
--- a/drivers/infiniband/hw/mlx4/qp.c
+++ b/drivers/infiniband/hw/mlx4/qp.c
@@ -84,6 +84,8 @@ static const __be32 mlx4_ib_opcode[] = {
[IB_WR_SEND_WITH_INV] = cpu_to_be32(MLX4_OPCODE_SEND_INVAL),
[IB_WR_LOCAL_INV] = cpu_to_be32(MLX4_OPCODE_LOCAL_INVAL),
[IB_WR_FAST_REG_MR] = cpu_to_be32(MLX4_OPCODE_FMR),
+ [IB_WR_MASKED_ATOMIC_CMP_AND_SWP] = cpu_to_be32(MLX4_OPCODE_MASKED_ATOMIC_CS),
+ [IB_WR_MASKED_ATOMIC_FETCH_AND_ADD] = cpu_to_be32(MLX4_OPCODE_MASKED_ATOMIC_FA),
};
static struct mlx4_ib_sqp *to_msqp(struct mlx4_ib_qp *mqp)
@@ -1406,6 +1408,9 @@ static void set_atomic_seg(struct mlx4_wqe_atomic_seg *aseg, struct ib_send_wr *
if (wr->opcode == IB_WR_ATOMIC_CMP_AND_SWP) {
aseg->swap_add = cpu_to_be64(wr->wr.atomic.swap);
aseg->compare = cpu_to_be64(wr->wr.atomic.compare_add);
+ } else if (wr->opcode == IB_WR_MASKED_ATOMIC_FETCH_AND_ADD) {
+ aseg->swap_add = cpu_to_be64(wr->wr.atomic.compare_add);
+ aseg->compare = cpu_to_be64(wr->wr.atomic.compare_add_mask);
} else {
aseg->swap_add = cpu_to_be64(wr->wr.atomic.compare_add);
aseg->compare = 0;
@@ -1413,6 +1418,14 @@ static void set_atomic_seg(struct mlx4_wqe_atomic_seg *aseg, struct ib_send_wr *
}
+static void set_mask_atomic_seg(struct mlx4_wqe_mask_atomic_seg *aseg, struct ib_send_wr *wr)
+{
+ aseg->swap_add = cpu_to_be64(wr->wr.atomic.swap);
+ aseg->swap_add_mask = cpu_to_be64(wr->wr.atomic.swap_mask);
+ aseg->compare = cpu_to_be64(wr->wr.atomic.compare_add);
+ aseg->compare_mask = cpu_to_be64(wr->wr.atomic.compare_add_mask);
+}
+
static void set_datagram_seg(struct mlx4_wqe_datagram_seg *dseg,
struct ib_send_wr *wr)
{
@@ -1566,6 +1579,7 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
switch (wr->opcode) {
case IB_WR_ATOMIC_CMP_AND_SWP:
case IB_WR_ATOMIC_FETCH_AND_ADD:
+ case IB_WR_MASKED_ATOMIC_FETCH_AND_ADD:
set_raddr_seg(wqe, wr->wr.atomic.remote_addr,
wr->wr.atomic.rkey);
wqe += sizeof (struct mlx4_wqe_raddr_seg);
@@ -1578,6 +1592,19 @@ int mlx4_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
break;
+ case IB_WR_MASKED_ATOMIC_CMP_AND_SWP:
+ set_raddr_seg(wqe, wr->wr.atomic.remote_addr,
+ wr->wr.atomic.rkey);
+ wqe += sizeof (struct mlx4_wqe_raddr_seg);
+
+ set_mask_atomic_seg(wqe, wr);
+ wqe += sizeof (struct mlx4_wqe_mask_atomic_seg);
+
+ size += (sizeof (struct mlx4_wqe_raddr_seg) +
+ sizeof (struct mlx4_wqe_mask_atomic_seg)) / 16;
+
+ break;
+
case IB_WR_RDMA_READ:
case IB_WR_RDMA_WRITE:
case IB_WR_RDMA_WRITE_WITH_IMM:
diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index e92d1bf..7a7f9c1 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -123,8 +123,8 @@ enum {
MLX4_OPCODE_RDMA_READ = 0x10,
MLX4_OPCODE_ATOMIC_CS = 0x11,
MLX4_OPCODE_ATOMIC_FA = 0x12,
- MLX4_OPCODE_ATOMIC_MASK_CS = 0x14,
- MLX4_OPCODE_ATOMIC_MASK_FA = 0x15,
+ MLX4_OPCODE_MASKED_ATOMIC_CS = 0x14,
+ MLX4_OPCODE_MASKED_ATOMIC_FA = 0x15,
MLX4_OPCODE_BIND_MW = 0x18,
MLX4_OPCODE_FMR = 0x19,
MLX4_OPCODE_LOCAL_INVAL = 0x1b,
diff --git a/include/linux/mlx4/qp.h b/include/linux/mlx4/qp.h
index 9f29d86..1a48413 100644
--- a/include/linux/mlx4/qp.h
+++ b/include/linux/mlx4/qp.h
@@ -285,6 +285,13 @@ struct mlx4_wqe_atomic_seg {
__be64 compare;
};
+struct mlx4_wqe_mask_atomic_seg {
+ __be64 swap_add;
+ __be64 compare;
+ __be64 swap_add_mask;
+ __be64 compare_mask;
+};
+
struct mlx4_wqe_data_seg {
__be32 byte_count;
__be32 lkey;
--
1.7.0.3
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-14 14:31 UTC (permalink / raw)
To: Andrea Gozzelino
Cc: Tung, Chien Tin, Steve Wise, linux-kernel@vger.kernel.org,
linux-rdma@vger.kernel.org, rolandd@cisco.com,
peterz@infradead.org, pavel@ucw.cz, mingo@elte.hu, Eric B Munson
In-Reply-To: <7037536.1271235105513.SLOX.WebMail.wwwrun@imap.lnl.infn.it>
Hi,
FMR are being used only in a special mode called ZCopy.
You could disable this mode by setting the module paramter
sdp_zcopy_thresh to 0, or by issuing:
# echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
This means that you won't get the benefits of Zero-copy.
- Amir
On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung@intel.com>
> wrote:
>
>
>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>> code
>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>
>>>>>
>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>
>>>>
>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>
>>>
>>
>> Good to know. Thanks.
>>
>> Chien
>>
>>
>>
> Hi Steve and Chien,
>
> I understand that NE020 cards have problem with SDP connected with
> map_phy_fmr (FMR stands for Fast Memory Region).
> Is it possible to solve/fix this point?
> If yes, have you an idea about the time that is necessary to code
> development/build?
> If no, can you suggest me a card that supports SDP protocol?
>
> I work on NE020 cards from February 2010 for an INFN experimental
> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
> acquisition and movement systems. The covergence of storage protocols
> around 10 Gigabits/s Ethernet protocols shows that one way could be the
> Remote Direct Memory Access (RDMA). The goals are the investigations of
> latency time, the throughput, the buffer size schemes and finally the
> global event building bandwidth.
>
> Do you know if NE020 cards have problems with librdma (RDMA procedures,
> in general) and / or with MPI versions?
>
> Thank you very much,
> Andrea
>
> Andrea Gozzelino
>
> INFN - Laboratori Nazionali di Legnaro (LNL)
> Viale dell'Universita' 2
> I-35020 - Legnaro (PD)- ITALIA
> Tel: +39 049 8068346
> Fax: +39 049 641925
> Mail: andrea.gozzelino@lnl.infn.it
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply
* Re: [PATCH] ummunotify: Userspace support for MMU notifications
From: Jeff Squyres @ 2010-04-14 14:36 UTC (permalink / raw)
To: Gleb Natapov
Cc: Roland Dreier (rdreier), Håkon Bugge, Jason Gunthorpe,
Andrew Morton, Eric B Munson, linux-kernel-u79uwXL29TY76Z2rM5mHXA,
linux-rdma-u79uwXL29TY76Z2rM5mHXA, rolandd-FYB4Gu1CFyUAvxtiuMwx3w,
peterz-wEGCiKHe2LqWVfeAwA7xHQ, pavel-+ZI9xUNit7I,
mingo-X9Un+BFzKDI
In-Reply-To: <20100414090623.GM23554-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
On Apr 14, 2010, at 5:06 AM, Gleb Natapov wrote:
> > The Open MPI developers have spent a lot of effort trying to handle this
> > purely in userspace and still do not believe that a truly robust
> > solution is possible without kernel help. Perhaps they can expand on
> > what the obstacles are.
By "truly robust" we mean that some other user-level code can't override the hooks installed by the MPI (user level) middleware. All current glibc hooks are overridable by other user-level code -- and sometimes real applications do this (for their own good reasons). Most of the time, apps blithely override our hooks because they either don't know or can't know that our hooks are installed. It can be dicey to know what you can and cannot override pre-main(), for example (e.g., via the __malloc_initialize_hook).
Opening up a direct channel to the kernel and saying "hey, tell me when something changes" is robust because no other entity can hijack your notifications. It also allows us to avoid using pre-main hooks, and makes it so that we don't have to hook into the memory subsystem (usually replacing it with our own). Both of these things are extremely distasteful -- fixing these two things alone make doing something like ummunotify worthwhile, IMHO.
--
Jeff Squyres
jsquyres-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-14 14:46 UTC (permalink / raw)
To: Andrea Gozzelino
Cc: Tung, Chien Tin, Steve Wise,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D1D3.3020004-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
One more thing - Please open a bug regarding the num_sge limitation at:
https://bugs.openfabrics.org/
Thanks,
Amir
On 04/14/2010 05:31 PM, Amir Vadai wrote:
> Hi,
>
> FMR are being used only in a special mode called ZCopy.
>
> You could disable this mode by setting the module paramter
> sdp_zcopy_thresh to 0, or by issuing:
> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>
> This means that you won't get the benefits of Zero-copy.
>
> - Amir
>
> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>
>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> wrote:
>>
>>
>>
>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>> code
>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>
>>>>>>
>>>>>>
>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>
>>>>>
>>>>>
>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>
>>>>
>>>>
>>> Good to know. Thanks.
>>>
>>> Chien
>>>
>>>
>>>
>>>
>> Hi Steve and Chien,
>>
>> I understand that NE020 cards have problem with SDP connected with
>> map_phy_fmr (FMR stands for Fast Memory Region).
>> Is it possible to solve/fix this point?
>> If yes, have you an idea about the time that is necessary to code
>> development/build?
>> If no, can you suggest me a card that supports SDP protocol?
>>
>> I work on NE020 cards from February 2010 for an INFN experimental
>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>> acquisition and movement systems. The covergence of storage protocols
>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>> latency time, the throughput, the buffer size schemes and finally the
>> global event building bandwidth.
>>
>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>> in general) and / or with MPI versions?
>>
>> Thank you very much,
>> Andrea
>>
>> Andrea Gozzelino
>>
>> INFN - Laboratori Nazionali di Legnaro (LNL)
>> Viale dell'Universita' 2
>> I-35020 - Legnaro (PD)- ITALIA
>> Tel: +39 049 8068346
>> Fax: +39 049 641925
>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-14 14:54 UTC (permalink / raw)
To: Amir Vadai
Cc: Andrea Gozzelino, Tung, Chien Tin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D1D3.3020004-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Hey Amir,
I don't think this helps because sdp_add_device() will not add rdma
devices that fail to create fmr pools.
So I guess you could key off of fmr pool failures and set
sdp_zcopy_thresh to 0 and allow the device to be used?
But what we really need is sdp support for fastreg_mrs as an
alternative to fmrs.
Steve.
Amir Vadai wrote:
> Hi,
>
> FMR are being used only in a special mode called ZCopy.
>
> You could disable this mode by setting the module paramter
> sdp_zcopy_thresh to 0, or by issuing:
> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>
> This means that you won't get the benefits of Zero-copy.
>
> - Amir
>
> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>
>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> wrote:
>>
>>
>>
>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>> code
>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>
>>>>>>
>>>>>>
>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>
>>>>>
>>>>>
>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>
>>>>
>>>>
>>> Good to know. Thanks.
>>>
>>> Chien
>>>
>>>
>>>
>>>
>> Hi Steve and Chien,
>>
>> I understand that NE020 cards have problem with SDP connected with
>> map_phy_fmr (FMR stands for Fast Memory Region).
>> Is it possible to solve/fix this point?
>> If yes, have you an idea about the time that is necessary to code
>> development/build?
>> If no, can you suggest me a card that supports SDP protocol?
>>
>> I work on NE020 cards from February 2010 for an INFN experimental
>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>> acquisition and movement systems. The covergence of storage protocols
>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>> latency time, the throughput, the buffer size schemes and finally the
>> global event building bandwidth.
>>
>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>> in general) and / or with MPI versions?
>>
>> Thank you very much,
>> Andrea
>>
>> Andrea Gozzelino
>>
>> INFN - Laboratori Nazionali di Legnaro (LNL)
>> Viale dell'Universita' 2
>> I-35020 - Legnaro (PD)- ITALIA
>> Tel: +39 049 8068346
>> Fax: +39 049 641925
>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* RE: Socket Direct Protocol: help (2)
From: Tung, Chien Tin @ 2010-04-14 15:01 UTC (permalink / raw)
To: Amir Vadai, Andrea Gozzelino
Cc: Steve Wise, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D529.1060701-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
>One more thing - Please open a bug regarding the num_sge limitation at:
>https://bugs.openfabrics.org/
Done, Bug 2027.
Chien
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Amir Vadai @ 2010-04-14 15:03 UTC (permalink / raw)
To: Steve Wise, Andrea Gozzelino
Cc: Tung, Chien Tin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D727.4090400-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
You are right - I missed it.
Andrea, Please open a bug at bugzilla (https://bugs.openfabrics.org) -
so that you will be notified as soon as I will fix SDP not use FMR if
not supported.
As to fastreg_mrs support - I don't know this mechanism. Do you mean FRWR?
Thanks,
Amir
On 04/14/2010 05:54 PM, Steve Wise wrote:
> Hey Amir,
>
> I don't think this helps because sdp_add_device() will not add rdma
> devices that fail to create fmr pools.
>
> So I guess you could key off of fmr pool failures and set
> sdp_zcopy_thresh to 0 and allow the device to be used?
>
> But what we really need is sdp support for fastreg_mrs as an
> alternative to fmrs.
>
>
> Steve.
>
>
> Amir Vadai wrote:
>
>> Hi,
>>
>> FMR are being used only in a special mode called ZCopy.
>>
>> You could disable this mode by setting the module paramter
>> sdp_zcopy_thresh to 0, or by issuing:
>> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>>
>> This means that you won't get the benefits of Zero-copy.
>>
>> - Amir
>>
>> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>>
>>
>>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>> wrote:
>>>
>>>
>>>
>>>
>>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>>> code
>>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Good to know. Thanks.
>>>>
>>>> Chien
>>>>
>>>>
>>>>
>>>>
>>>>
>>> Hi Steve and Chien,
>>>
>>> I understand that NE020 cards have problem with SDP connected with
>>> map_phy_fmr (FMR stands for Fast Memory Region).
>>> Is it possible to solve/fix this point?
>>> If yes, have you an idea about the time that is necessary to code
>>> development/build?
>>> If no, can you suggest me a card that supports SDP protocol?
>>>
>>> I work on NE020 cards from February 2010 for an INFN experimental
>>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>>> acquisition and movement systems. The covergence of storage protocols
>>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>>> latency time, the throughput, the buffer size schemes and finally the
>>> global event building bandwidth.
>>>
>>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>>> in general) and / or with MPI versions?
>>>
>>> Thank you very much,
>>> Andrea
>>>
>>> Andrea Gozzelino
>>>
>>> INFN - Laboratori Nazionali di Legnaro (LNL)
>>> Viale dell'Universita' 2
>>> I-35020 - Legnaro (PD)- ITALIA
>>> Tel: +39 049 8068346
>>> Fax: +39 049 641925
>>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.tux.org/lkml/
>>
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-14 15:05 UTC (permalink / raw)
To: Tung, Chien Tin
Cc: Amir Vadai, Andrea Gozzelino,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <2EFBCAEF10980645BBCFB605689E08E9047C79B918-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org>
Tung, Chien Tin wrote:
>> One more thing - Please open a bug regarding the num_sge limitation at:
>> https://bugs.openfabrics.org/
>>
>
> Done, Bug 2027.
>
> Chien
>
And 2028 opened to request fastreg support.
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Socket Direct Protocol: help (2)
From: Steve Wise @ 2010-04-14 15:08 UTC (permalink / raw)
To: Amir Vadai
Cc: Andrea Gozzelino, Tung, Chien Tin,
linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org,
peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org,
pavel-+ZI9xUNit7I@public.gmane.org,
mingo-X9Un+BFzKDI@public.gmane.org, Eric B Munson
In-Reply-To: <4BC5D95D.3010108-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>
Amir Vadai wrote:
> You are right - I missed it.
>
> Andrea, Please open a bug at bugzilla (https://bugs.openfabrics.org) -
> so that you will be notified as soon as I will fix SDP not use FMR if
> not supported.
>
> As to fastreg_mrs support - I don't know this mechanism. Do you mean FRWR?
>
>
ib_alloc_fast_reg_mr(), ib_alloc_fast_reg_page_list() and friends, plus
the IB_WR_FAST_REG_MR work request.
> Thanks,
> Amir
>
>
> On 04/14/2010 05:54 PM, Steve Wise wrote:
>
>> Hey Amir,
>>
>> I don't think this helps because sdp_add_device() will not add rdma
>> devices that fail to create fmr pools.
>>
>> So I guess you could key off of fmr pool failures and set
>> sdp_zcopy_thresh to 0 and allow the device to be used?
>>
>> But what we really need is sdp support for fastreg_mrs as an
>> alternative to fmrs.
>>
>>
>> Steve.
>>
>>
>> Amir Vadai wrote:
>>
>>
>>> Hi,
>>>
>>> FMR are being used only in a special mode called ZCopy.
>>>
>>> You could disable this mode by setting the module paramter
>>> sdp_zcopy_thresh to 0, or by issuing:
>>> # echo 0 > /sys/module/ib_sdp/parameters/sdp_zcopy_thresh
>>>
>>> This means that you won't get the benefits of Zero-copy.
>>>
>>> - Amir
>>>
>>> On 04/14/2010 11:51 AM, Andrea Gozzelino wrote:
>>>
>>>
>>>
>>>> On Apr 13, 2010 10:22 PM, "Tung, Chien Tin" <chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>>>> Chien, does the NE020 support FMRs? I looked at the nes ofed-1.5
>>>>>>>> code
>>>>>>>> and it appears to do nothing in the map_phys_fmr functions.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> We never implemented map_phys_fmr. Is it relevant to the # of SGEs?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> No, but SDP uses FMRs. I don't think it will run without FMR support.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Good to know. Thanks.
>>>>>
>>>>> Chien
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Hi Steve and Chien,
>>>>
>>>> I understand that NE020 cards have problem with SDP connected with
>>>> map_phy_fmr (FMR stands for Fast Memory Region).
>>>> Is it possible to solve/fix this point?
>>>> If yes, have you an idea about the time that is necessary to code
>>>> development/build?
>>>> If no, can you suggest me a card that supports SDP protocol?
>>>>
>>>> I work on NE020 cards from February 2010 for an INFN experimental
>>>> proposal, called REDIGO (Read out at 10 Gbits/s), about the data
>>>> acquisition and movement systems. The covergence of storage protocols
>>>> around 10 Gigabits/s Ethernet protocols shows that one way could be the
>>>> Remote Direct Memory Access (RDMA). The goals are the investigations of
>>>> latency time, the throughput, the buffer size schemes and finally the
>>>> global event building bandwidth.
>>>>
>>>> Do you know if NE020 cards have problems with librdma (RDMA procedures,
>>>> in general) and / or with MPI versions?
>>>>
>>>> Thank you very much,
>>>> Andrea
>>>>
>>>> Andrea Gozzelino
>>>>
>>>> INFN - Laboratori Nazionali di Legnaro (LNL)
>>>> Viale dell'Universita' 2
>>>> I-35020 - Legnaro (PD)- ITALIA
>>>> Tel: +39 049 8068346
>>>> Fax: +39 049 641925
>>>> Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at http://www.tux.org/lkml/
>>>
>>>
>>>
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox