* [PATCH 01/28] IB/srp: Use kstrtoull() instead of simple_strtoull()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 02/28] IB/srp: Make the path record query error message more informative Bart Van Assche
` (25 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Use kstrtoull() since simple_strtoull() is deprecated. This patch
improves error checking but otherwise does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srp/ib_srp.c | 33 +++++++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 62d88212c1b0..39b3e43efbbe 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -3111,6 +3111,7 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
char *options, *sep_opt;
char *p;
substring_t args[MAX_OPT_ARGS];
+ unsigned long long ull;
int opt_mask = 0;
int token;
int ret = -EINVAL;
@@ -3135,7 +3136,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
ret = -ENOMEM;
goto out;
}
- target->id_ext = cpu_to_be64(simple_strtoull(p, NULL, 16));
+ ret = kstrtoull(p, 16, &ull);
+ if (ret) {
+ pr_warn("invalid id_ext parameter '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->id_ext = cpu_to_be64(ull);
kfree(p);
break;
@@ -3145,7 +3152,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
ret = -ENOMEM;
goto out;
}
- target->ioc_guid = cpu_to_be64(simple_strtoull(p, NULL, 16));
+ ret = kstrtoull(p, 16, &ull);
+ if (ret) {
+ pr_warn("invalid ioc_guid parameter '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->ioc_guid = cpu_to_be64(ull);
kfree(p);
break;
@@ -3181,7 +3194,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
ret = -ENOMEM;
goto out;
}
- target->service_id = cpu_to_be64(simple_strtoull(p, NULL, 16));
+ ret = kstrtoull(p, 16, &ull);
+ if (ret) {
+ pr_warn("bad service_id parameter '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->service_id = cpu_to_be64(ull);
kfree(p);
break;
@@ -3235,7 +3254,13 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
ret = -ENOMEM;
goto out;
}
- target->initiator_ext = cpu_to_be64(simple_strtoull(p, NULL, 16));
+ ret = kstrtoull(p, 16, &ull);
+ if (ret) {
+ pr_warn("bad initiator_ext value '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->initiator_ext = cpu_to_be64(ull);
kfree(p);
break;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 02/28] IB/srp: Make the path record query error message more informative
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
2018-01-03 21:39 ` [PATCH 01/28] IB/srp: Use kstrtoull() instead of simple_strtoull() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 03/28] IB/srp: Refactor srp_send_req() Bart Van Assche
` (24 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Show all path record query parameters if a path record query fails.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srp/ib_srp.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 39b3e43efbbe..fec6e800adf4 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -702,7 +702,10 @@ static int srp_lookup_path(struct srp_rdma_ch *ch)
ret = ch->status;
if (ret < 0)
shost_printk(KERN_WARNING, target->scsi_host,
- PFX "Path record query failed\n");
+ PFX "Path record query failed: sgid %pI6, dgid %pI6, pkey %#04x, service_id %#16llx\n",
+ ch->path.sgid.raw, ch->path.dgid.raw,
+ be16_to_cpu(target->pkey),
+ be64_to_cpu(target->service_id));
put:
scsi_host_put(target->scsi_host);
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 03/28] IB/srp: Refactor srp_send_req()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
2018-01-03 21:39 ` [PATCH 01/28] IB/srp: Use kstrtoull() instead of simple_strtoull() Bart Van Assche
2018-01-03 21:39 ` [PATCH 02/28] IB/srp: Make the path record query error message more informative Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 04/28] IB/srp: Add RDMA/CM support Bart Van Assche
` (23 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
This patch does not change any functionality but makes the next
patch in this series easier to read.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srp/ib_srp.c | 65 +++++++++++++++++++------------------
1 file changed, 34 insertions(+), 31 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index fec6e800adf4..9472f5989d27 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -738,35 +738,21 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
struct ib_cm_req_param param;
struct srp_login_req priv;
} *req = NULL;
+ char *ipi, *tpi;
int status;
- u8 subnet_timeout;
-
- subnet_timeout = srp_get_subnet_timeout(target->srp_host);
req = kzalloc(sizeof *req, GFP_KERNEL);
if (!req)
return -ENOMEM;
- req->param.primary_path = &ch->path;
- req->param.alternate_path = NULL;
- req->param.service_id = target->service_id;
- req->param.qp_num = ch->qp->qp_num;
- req->param.qp_type = ch->qp->qp_type;
- req->param.private_data = &req->priv;
- req->param.private_data_len = sizeof req->priv;
req->param.flow_control = 1;
-
- get_random_bytes(&req->param.starting_psn, 4);
- req->param.starting_psn &= 0xffffff;
+ req->param.retry_count = target->tl_retry_count;
/*
* Pick some arbitrary defaults here; we could make these
* module parameters if anyone cared about setting them.
*/
req->param.responder_resources = 4;
- req->param.remote_cm_response_timeout = subnet_timeout + 2;
- req->param.local_cm_response_timeout = subnet_timeout + 2;
- req->param.retry_count = target->tl_retry_count;
req->param.rnr_retry_count = 7;
req->param.max_cm_retries = 15;
@@ -777,6 +763,28 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
SRP_BUF_FORMAT_INDIRECT);
req->priv.req_flags = (multich ? SRP_MULTICHAN_MULTI :
SRP_MULTICHAN_SINGLE);
+
+ {
+ u8 subnet_timeout;
+
+ subnet_timeout = srp_get_subnet_timeout(target->srp_host);
+
+ req->param.primary_path = &ch->path;
+ req->param.alternate_path = NULL;
+ req->param.service_id = target->service_id;
+ get_random_bytes(&req->param.starting_psn, 4);
+ req->param.starting_psn &= 0xffffff;
+ req->param.qp_num = ch->qp->qp_num;
+ req->param.qp_type = ch->qp->qp_type;
+ req->param.local_cm_response_timeout = subnet_timeout + 2;
+ req->param.remote_cm_response_timeout = subnet_timeout + 2;
+ req->param.private_data = &req->priv;
+ req->param.private_data_len = sizeof(req->priv);
+
+ ipi = req->priv.initiator_port_id;
+ tpi = req->priv.target_port_id;
+ }
+
/*
* In the published SRP specification (draft rev. 16a), the
* port identifier format is 8 bytes of ID extension followed
@@ -787,19 +795,15 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
* recognized by the I/O Class they report.
*/
if (target->io_class == SRP_REV10_IB_IO_CLASS) {
- memcpy(req->priv.initiator_port_id,
- &target->sgid.global.interface_id, 8);
- memcpy(req->priv.initiator_port_id + 8,
- &target->initiator_ext, 8);
- memcpy(req->priv.target_port_id, &target->ioc_guid, 8);
- memcpy(req->priv.target_port_id + 8, &target->id_ext, 8);
+ memcpy(ipi, &target->sgid.global.interface_id, 8);
+ memcpy(ipi + 8, &target->initiator_ext, 8);
+ memcpy(tpi, &target->ioc_guid, 8);
+ memcpy(tpi + 8, &target->id_ext, 8);
} else {
- memcpy(req->priv.initiator_port_id,
- &target->initiator_ext, 8);
- memcpy(req->priv.initiator_port_id + 8,
- &target->sgid.global.interface_id, 8);
- memcpy(req->priv.target_port_id, &target->id_ext, 8);
- memcpy(req->priv.target_port_id + 8, &target->ioc_guid, 8);
+ memcpy(ipi, &target->initiator_ext, 8);
+ memcpy(ipi + 8, &target->sgid.global.interface_id, 8);
+ memcpy(tpi, &target->id_ext, 8);
+ memcpy(tpi + 8, &target->ioc_guid, 8);
}
/*
@@ -812,9 +816,8 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
PFX "Topspin/Cisco initiator port ID workaround "
"activated for target GUID %016llx\n",
be64_to_cpu(target->ioc_guid));
- memset(req->priv.initiator_port_id, 0, 8);
- memcpy(req->priv.initiator_port_id + 8,
- &target->srp_host->srp_dev->dev->node_guid, 8);
+ memset(ipi, 0, 8);
+ memcpy(ipi + 8, &target->srp_host->srp_dev->dev->node_guid, 8);
}
status = ib_send_cm_req(ch->cm_id, &req->param);
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 04/28] IB/srp: Add RDMA/CM support
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (2 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 03/28] IB/srp: Refactor srp_send_req() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 07/28] IB/srpt: Remove an unused structure member Bart Van Assche
` (22 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Since the SRP_LOGIN_REQ defined in the SRP standard is larger than
what fits in the RDMA/CM login request private data, introduce a new
login request format for the RDMA/CM. Add a kernel module parameter
to the ib_srp kernel driver that allows to specify the port on which
to listen for RDMA/CM connections. The default value for this kernel
module parameter is 0, which means not to listen for RDMA/CM
connections.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srp/ib_srp.c | 683 ++++++++++++++++++++++++++++--------
drivers/infiniband/ulp/srp/ib_srp.h | 41 ++-
include/scsi/srp.h | 17 +
3 files changed, 590 insertions(+), 151 deletions(-)
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 9472f5989d27..6f59bf715a0d 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -41,6 +41,7 @@
#include <linux/random.h>
#include <linux/jiffies.h>
#include <linux/lockdep.h>
+#include <linux/inet.h>
#include <rdma/ib_cache.h>
#include <linux/atomic.h>
@@ -144,7 +145,9 @@ static void srp_remove_one(struct ib_device *device, void *client_data);
static void srp_recv_done(struct ib_cq *cq, struct ib_wc *wc);
static void srp_handle_qp_err(struct ib_cq *cq, struct ib_wc *wc,
const char *opname);
-static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event);
+static int srp_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event);
+static int srp_rdma_cm_handler(struct rdma_cm_id *cm_id,
+ struct rdma_cm_event *event);
static struct scsi_transport_template *ib_srp_transport_template;
static struct workqueue_struct *srp_remove_wq;
@@ -265,8 +268,8 @@ static void srp_qp_event(struct ib_event *event, void *context)
ib_event_msg(event->event), event->event);
}
-static int srp_init_qp(struct srp_target_port *target,
- struct ib_qp *qp)
+static int srp_init_ib_qp(struct srp_target_port *target,
+ struct ib_qp *qp)
{
struct ib_qp_attr *attr;
int ret;
@@ -277,7 +280,7 @@ static int srp_init_qp(struct srp_target_port *target,
ret = ib_find_cached_pkey(target->srp_host->srp_dev->dev,
target->srp_host->port,
- be16_to_cpu(target->pkey),
+ be16_to_cpu(target->ib_cm.pkey),
&attr->pkey_index);
if (ret)
goto out;
@@ -298,32 +301,110 @@ static int srp_init_qp(struct srp_target_port *target,
return ret;
}
-static int srp_new_cm_id(struct srp_rdma_ch *ch)
+static int srp_new_ib_cm_id(struct srp_rdma_ch *ch)
{
struct srp_target_port *target = ch->target;
struct ib_cm_id *new_cm_id;
new_cm_id = ib_create_cm_id(target->srp_host->srp_dev->dev,
- srp_cm_handler, ch);
+ srp_ib_cm_handler, ch);
if (IS_ERR(new_cm_id))
return PTR_ERR(new_cm_id);
- if (ch->cm_id)
- ib_destroy_cm_id(ch->cm_id);
- ch->cm_id = new_cm_id;
+ if (ch->ib_cm.cm_id)
+ ib_destroy_cm_id(ch->ib_cm.cm_id);
+ ch->ib_cm.cm_id = new_cm_id;
if (rdma_cap_opa_ah(target->srp_host->srp_dev->dev,
target->srp_host->port))
- ch->path.rec_type = SA_PATH_REC_TYPE_OPA;
+ ch->ib_cm.path.rec_type = SA_PATH_REC_TYPE_OPA;
else
- ch->path.rec_type = SA_PATH_REC_TYPE_IB;
- ch->path.sgid = target->sgid;
- ch->path.dgid = target->orig_dgid;
- ch->path.pkey = target->pkey;
- ch->path.service_id = target->service_id;
+ ch->ib_cm.path.rec_type = SA_PATH_REC_TYPE_IB;
+ ch->ib_cm.path.sgid = target->sgid;
+ ch->ib_cm.path.dgid = target->ib_cm.orig_dgid;
+ ch->ib_cm.path.pkey = target->ib_cm.pkey;
+ ch->ib_cm.path.service_id = target->ib_cm.service_id;
return 0;
}
+static const char *inet_ntop(const void *sa, char *dst, unsigned int size)
+{
+ switch (((struct sockaddr *)sa)->sa_family) {
+ case AF_INET:
+ snprintf(dst, size, "%pI4",
+ &((struct sockaddr_in *)sa)->sin_addr);
+ break;
+ case AF_INET6:
+ snprintf(dst, size, "%pI6",
+ &((struct sockaddr_in6 *)sa)->sin6_addr);
+ break;
+ default:
+ snprintf(dst, size, "???");
+ break;
+ }
+ return dst;
+}
+
+static int srp_new_rdma_cm_id(struct srp_rdma_ch *ch)
+{
+ struct srp_target_port *target = ch->target;
+ struct rdma_cm_id *new_cm_id;
+ char src_addr[64], dst_addr[64];
+ int ret;
+
+ new_cm_id = rdma_create_id(&init_net, srp_rdma_cm_handler, ch,
+ RDMA_PS_TCP, IB_QPT_RC);
+ if (IS_ERR(new_cm_id)) {
+ ret = PTR_ERR(new_cm_id);
+ new_cm_id = NULL;
+ goto out;
+ }
+
+ init_completion(&ch->done);
+ ret = rdma_resolve_addr(new_cm_id, target->rdma_cm.src_specified ?
+ (struct sockaddr *)&target->rdma_cm.src : NULL,
+ (struct sockaddr *)&target->rdma_cm.dst,
+ SRP_PATH_REC_TIMEOUT_MS);
+ if (ret) {
+ pr_err("No route available from %s to %s (%d)\n",
+ target->rdma_cm.src_specified ?
+ inet_ntop(&target->rdma_cm.src, src_addr,
+ sizeof(src_addr)) : "(any)",
+ inet_ntop(&target->rdma_cm.dst, dst_addr,
+ sizeof(dst_addr)),
+ ret);
+ goto out;
+ }
+ ret = wait_for_completion_interruptible(&ch->done);
+ if (ret < 0)
+ goto out;
+
+ ret = ch->status;
+ if (ret) {
+ pr_err("Resolving address %s failed (%d)\n",
+ inet_ntop(&target->rdma_cm.dst, dst_addr,
+ sizeof(dst_addr)),
+ ret);
+ goto out;
+ }
+
+ swap(ch->rdma_cm.cm_id, new_cm_id);
+
+out:
+ if (new_cm_id)
+ rdma_destroy_id(new_cm_id);
+
+ return ret;
+}
+
+static int srp_new_cm_id(struct srp_rdma_ch *ch)
+{
+ struct srp_target_port *target = ch->target;
+
+ return target->using_rdma_cm ? srp_new_rdma_cm_id(ch) :
+ srp_new_ib_cm_id(ch);
+}
+
static struct ib_fmr_pool *srp_alloc_fmr_pool(struct srp_target_port *target)
{
struct srp_device *dev = target->srp_host->srp_dev;
@@ -521,16 +602,25 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
init_attr->send_cq = send_cq;
init_attr->recv_cq = recv_cq;
- qp = ib_create_qp(dev->pd, init_attr);
- if (IS_ERR(qp)) {
- ret = PTR_ERR(qp);
+ if (target->using_rdma_cm) {
+ ret = rdma_create_qp(ch->rdma_cm.cm_id, dev->pd, init_attr);
+ qp = ch->rdma_cm.cm_id->qp;
+ } else {
+ qp = ib_create_qp(dev->pd, init_attr);
+ if (!IS_ERR(qp)) {
+ ret = srp_init_ib_qp(target, qp);
+ if (ret)
+ ib_destroy_qp(qp);
+ } else {
+ ret = PTR_ERR(qp);
+ }
+ }
+ if (ret) {
+ pr_err("QP creation failed for dev %s: %d\n",
+ dev_name(&dev->dev->dev), ret);
goto err_send_cq;
}
- ret = srp_init_qp(target, qp);
- if (ret)
- goto err_qp;
-
if (dev->use_fast_reg) {
fr_pool = srp_alloc_fr_pool(target);
if (IS_ERR(fr_pool)) {
@@ -574,7 +664,10 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch)
return 0;
err_qp:
- ib_destroy_qp(qp);
+ if (target->using_rdma_cm)
+ rdma_destroy_qp(ch->rdma_cm.cm_id);
+ else
+ ib_destroy_qp(qp);
err_send_cq:
ib_free_cq(send_cq);
@@ -600,9 +693,16 @@ static void srp_free_ch_ib(struct srp_target_port *target,
if (!ch->target)
return;
- if (ch->cm_id) {
- ib_destroy_cm_id(ch->cm_id);
- ch->cm_id = NULL;
+ if (target->using_rdma_cm) {
+ if (ch->rdma_cm.cm_id) {
+ rdma_destroy_id(ch->rdma_cm.cm_id);
+ ch->rdma_cm.cm_id = NULL;
+ }
+ } else {
+ if (ch->ib_cm.cm_id) {
+ ib_destroy_cm_id(ch->ib_cm.cm_id);
+ ch->ib_cm.cm_id = NULL;
+ }
}
/* If srp_new_cm_id() succeeded but srp_create_ch_ib() not, return. */
@@ -658,16 +758,16 @@ static void srp_path_rec_completion(int status,
shost_printk(KERN_ERR, target->scsi_host,
PFX "Got failed path rec status %d\n", status);
else
- ch->path = *pathrec;
+ ch->ib_cm.path = *pathrec;
complete(&ch->done);
}
-static int srp_lookup_path(struct srp_rdma_ch *ch)
+static int srp_ib_lookup_path(struct srp_rdma_ch *ch)
{
struct srp_target_port *target = ch->target;
int ret = -ENODEV;
- ch->path.numb_path = 1;
+ ch->ib_cm.path.numb_path = 1;
init_completion(&ch->done);
@@ -678,10 +778,10 @@ static int srp_lookup_path(struct srp_rdma_ch *ch)
if (!scsi_host_get(target->scsi_host))
goto out;
- ch->path_query_id = ib_sa_path_rec_get(&srp_sa_client,
+ ch->ib_cm.path_query_id = ib_sa_path_rec_get(&srp_sa_client,
target->srp_host->srp_dev->dev,
target->srp_host->port,
- &ch->path,
+ &ch->ib_cm.path,
IB_SA_PATH_REC_SERVICE_ID |
IB_SA_PATH_REC_DGID |
IB_SA_PATH_REC_SGID |
@@ -690,8 +790,8 @@ static int srp_lookup_path(struct srp_rdma_ch *ch)
SRP_PATH_REC_TIMEOUT_MS,
GFP_KERNEL,
srp_path_rec_completion,
- ch, &ch->path_query);
- ret = ch->path_query_id;
+ ch, &ch->ib_cm.path_query);
+ ret = ch->ib_cm.path_query_id;
if (ret < 0)
goto put;
@@ -703,9 +803,9 @@ static int srp_lookup_path(struct srp_rdma_ch *ch)
if (ret < 0)
shost_printk(KERN_WARNING, target->scsi_host,
PFX "Path record query failed: sgid %pI6, dgid %pI6, pkey %#04x, service_id %#16llx\n",
- ch->path.sgid.raw, ch->path.dgid.raw,
- be16_to_cpu(target->pkey),
- be64_to_cpu(target->service_id));
+ ch->ib_cm.path.sgid.raw, ch->ib_cm.path.dgid.raw,
+ be16_to_cpu(target->ib_cm.pkey),
+ be64_to_cpu(target->ib_cm.service_id));
put:
scsi_host_put(target->scsi_host);
@@ -714,6 +814,34 @@ static int srp_lookup_path(struct srp_rdma_ch *ch)
return ret;
}
+static int srp_rdma_lookup_path(struct srp_rdma_ch *ch)
+{
+ struct srp_target_port *target = ch->target;
+ int ret;
+
+ init_completion(&ch->done);
+
+ ret = rdma_resolve_route(ch->rdma_cm.cm_id, SRP_PATH_REC_TIMEOUT_MS);
+ if (ret)
+ return ret;
+
+ wait_for_completion_interruptible(&ch->done);
+
+ if (ch->status != 0)
+ shost_printk(KERN_WARNING, target->scsi_host,
+ PFX "Path resolution failed\n");
+
+ return ch->status;
+}
+
+static int srp_lookup_path(struct srp_rdma_ch *ch)
+{
+ struct srp_target_port *target = ch->target;
+
+ return target->using_rdma_cm ? srp_rdma_lookup_path(ch) :
+ srp_ib_lookup_path(ch);
+}
+
static u8 srp_get_subnet_timeout(struct srp_host *host)
{
struct ib_port_attr attr;
@@ -735,8 +863,10 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
{
struct srp_target_port *target = ch->target;
struct {
- struct ib_cm_req_param param;
- struct srp_login_req priv;
+ struct rdma_conn_param rdma_param;
+ struct srp_login_req_rdma rdma_req;
+ struct ib_cm_req_param ib_param;
+ struct srp_login_req ib_req;
} *req = NULL;
char *ipi, *tpi;
int status;
@@ -745,44 +875,62 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
if (!req)
return -ENOMEM;
- req->param.flow_control = 1;
- req->param.retry_count = target->tl_retry_count;
+ req->ib_param.flow_control = 1;
+ req->ib_param.retry_count = target->tl_retry_count;
/*
* Pick some arbitrary defaults here; we could make these
* module parameters if anyone cared about setting them.
*/
- req->param.responder_resources = 4;
- req->param.rnr_retry_count = 7;
- req->param.max_cm_retries = 15;
-
- req->priv.opcode = SRP_LOGIN_REQ;
- req->priv.tag = 0;
- req->priv.req_it_iu_len = cpu_to_be32(target->max_iu_len);
- req->priv.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
+ req->ib_param.responder_resources = 4;
+ req->ib_param.rnr_retry_count = 7;
+ req->ib_param.max_cm_retries = 15;
+
+ req->ib_req.opcode = SRP_LOGIN_REQ;
+ req->ib_req.tag = 0;
+ req->ib_req.req_it_iu_len = cpu_to_be32(target->max_iu_len);
+ req->ib_req.req_buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
SRP_BUF_FORMAT_INDIRECT);
- req->priv.req_flags = (multich ? SRP_MULTICHAN_MULTI :
- SRP_MULTICHAN_SINGLE);
-
- {
+ req->ib_req.req_flags = (multich ? SRP_MULTICHAN_MULTI :
+ SRP_MULTICHAN_SINGLE);
+
+ if (target->using_rdma_cm) {
+ req->rdma_param.flow_control = req->ib_param.flow_control;
+ req->rdma_param.responder_resources =
+ req->ib_param.responder_resources;
+ req->rdma_param.initiator_depth = req->ib_param.initiator_depth;
+ req->rdma_param.retry_count = req->ib_param.retry_count;
+ req->rdma_param.rnr_retry_count = req->ib_param.rnr_retry_count;
+ req->rdma_param.private_data = &req->rdma_req;
+ req->rdma_param.private_data_len = sizeof(req->rdma_req);
+
+ req->rdma_req.opcode = req->ib_req.opcode;
+ req->rdma_req.tag = req->ib_req.tag;
+ req->rdma_req.req_it_iu_len = req->ib_req.req_it_iu_len;
+ req->rdma_req.req_buf_fmt = req->ib_req.req_buf_fmt;
+ req->rdma_req.req_flags = req->ib_req.req_flags;
+
+ ipi = req->rdma_req.initiator_port_id;
+ tpi = req->rdma_req.target_port_id;
+ } else {
u8 subnet_timeout;
subnet_timeout = srp_get_subnet_timeout(target->srp_host);
- req->param.primary_path = &ch->path;
- req->param.alternate_path = NULL;
- req->param.service_id = target->service_id;
- get_random_bytes(&req->param.starting_psn, 4);
- req->param.starting_psn &= 0xffffff;
- req->param.qp_num = ch->qp->qp_num;
- req->param.qp_type = ch->qp->qp_type;
- req->param.local_cm_response_timeout = subnet_timeout + 2;
- req->param.remote_cm_response_timeout = subnet_timeout + 2;
- req->param.private_data = &req->priv;
- req->param.private_data_len = sizeof(req->priv);
-
- ipi = req->priv.initiator_port_id;
- tpi = req->priv.target_port_id;
+ req->ib_param.primary_path = &ch->ib_cm.path;
+ req->ib_param.alternate_path = NULL;
+ req->ib_param.service_id = target->ib_cm.service_id;
+ get_random_bytes(&req->ib_param.starting_psn, 4);
+ req->ib_param.starting_psn &= 0xffffff;
+ req->ib_param.qp_num = ch->qp->qp_num;
+ req->ib_param.qp_type = ch->qp->qp_type;
+ req->ib_param.local_cm_response_timeout = subnet_timeout + 2;
+ req->ib_param.remote_cm_response_timeout = subnet_timeout + 2;
+ req->ib_param.private_data = &req->ib_req;
+ req->ib_param.private_data_len = sizeof(req->ib_req);
+
+ ipi = req->ib_req.initiator_port_id;
+ tpi = req->ib_req.target_port_id;
}
/*
@@ -820,7 +968,10 @@ static int srp_send_req(struct srp_rdma_ch *ch, bool multich)
memcpy(ipi + 8, &target->srp_host->srp_dev->dev->node_guid, 8);
}
- status = ib_send_cm_req(ch->cm_id, &req->param);
+ if (target->using_rdma_cm)
+ status = rdma_connect(ch->rdma_cm.cm_id, &req->rdma_param);
+ else
+ status = ib_send_cm_req(ch->ib_cm.cm_id, &req->ib_param);
kfree(req);
@@ -847,14 +998,23 @@ static bool srp_queue_remove_work(struct srp_target_port *target)
static void srp_disconnect_target(struct srp_target_port *target)
{
struct srp_rdma_ch *ch;
- int i;
+ int i, ret;
/* XXX should send SRP_I_LOGOUT request */
for (i = 0; i < target->ch_count; i++) {
ch = &target->ch[i];
ch->connected = false;
- if (ch->cm_id && ib_send_cm_dreq(ch->cm_id, NULL, 0)) {
+ ret = 0;
+ if (target->using_rdma_cm) {
+ if (ch->rdma_cm.cm_id)
+ rdma_disconnect(ch->rdma_cm.cm_id);
+ } else {
+ if (ch->ib_cm.cm_id)
+ ret = ib_send_cm_dreq(ch->ib_cm.cm_id,
+ NULL, 0);
+ }
+ if (ret < 0) {
shost_printk(KERN_DEBUG, target->scsi_host,
PFX "Sending CM DREQ failed\n");
}
@@ -2355,7 +2515,7 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id,
struct srp_target_port *target = ch->target;
struct ib_qp_attr *qp_attr = NULL;
int attr_mask = 0;
- int ret;
+ int ret = 0;
int i;
if (lrsp->opcode == SRP_LOGIN_RSP) {
@@ -2385,40 +2545,42 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id,
goto error;
}
- ret = -ENOMEM;
- qp_attr = kmalloc(sizeof *qp_attr, GFP_KERNEL);
- if (!qp_attr)
- goto error;
-
- qp_attr->qp_state = IB_QPS_RTR;
- ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
- if (ret)
- goto error_free;
-
- ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
- if (ret)
- goto error_free;
-
for (i = 0; i < target->queue_size; i++) {
struct srp_iu *iu = ch->rx_ring[i];
ret = srp_post_recv(ch, iu);
if (ret)
- goto error_free;
+ goto error;
}
- qp_attr->qp_state = IB_QPS_RTS;
- ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
- if (ret)
- goto error_free;
+ if (!target->using_rdma_cm) {
+ ret = -ENOMEM;
+ qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL);
+ if (!qp_attr)
+ goto error;
- target->rq_tmo_jiffies = srp_compute_rq_tmo(qp_attr, attr_mask);
+ qp_attr->qp_state = IB_QPS_RTR;
+ ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
+ if (ret)
+ goto error_free;
- ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
- if (ret)
- goto error_free;
+ ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
+ if (ret)
+ goto error_free;
+
+ qp_attr->qp_state = IB_QPS_RTS;
+ ret = ib_cm_init_qp_attr(cm_id, qp_attr, &attr_mask);
+ if (ret)
+ goto error_free;
- ret = ib_send_cm_rtu(cm_id, NULL, 0);
+ target->rq_tmo_jiffies = srp_compute_rq_tmo(qp_attr, attr_mask);
+
+ ret = ib_modify_qp(ch->qp, qp_attr, attr_mask);
+ if (ret)
+ goto error_free;
+
+ ret = ib_send_cm_rtu(cm_id, NULL, 0);
+ }
error_free:
kfree(qp_attr);
@@ -2427,41 +2589,43 @@ static void srp_cm_rep_handler(struct ib_cm_id *cm_id,
ch->status = ret;
}
-static void srp_cm_rej_handler(struct ib_cm_id *cm_id,
- struct ib_cm_event *event,
- struct srp_rdma_ch *ch)
+static void srp_ib_cm_rej_handler(struct ib_cm_id *cm_id,
+ struct ib_cm_event *event,
+ struct srp_rdma_ch *ch)
{
struct srp_target_port *target = ch->target;
struct Scsi_Host *shost = target->scsi_host;
struct ib_class_port_info *cpi;
int opcode;
+ u16 dlid;
switch (event->param.rej_rcvd.reason) {
case IB_CM_REJ_PORT_CM_REDIRECT:
cpi = event->param.rej_rcvd.ari;
- sa_path_set_dlid(&ch->path, ntohs(cpi->redirect_lid));
- ch->path.pkey = cpi->redirect_pkey;
+ dlid = be16_to_cpu(cpi->redirect_lid);
+ sa_path_set_dlid(&ch->ib_cm.path, dlid);
+ ch->ib_cm.path.pkey = cpi->redirect_pkey;
cm_id->remote_cm_qpn = be32_to_cpu(cpi->redirect_qp) & 0x00ffffff;
- memcpy(ch->path.dgid.raw, cpi->redirect_gid, 16);
+ memcpy(ch->ib_cm.path.dgid.raw, cpi->redirect_gid, 16);
- ch->status = sa_path_get_dlid(&ch->path) ?
- SRP_DLID_REDIRECT : SRP_PORT_REDIRECT;
+ ch->status = dlid ? SRP_DLID_REDIRECT : SRP_PORT_REDIRECT;
break;
case IB_CM_REJ_PORT_REDIRECT:
if (srp_target_is_topspin(target)) {
+ union ib_gid *dgid = &ch->ib_cm.path.dgid;
+
/*
* Topspin/Cisco SRP gateways incorrectly send
* reject reason code 25 when they mean 24
* (port redirect).
*/
- memcpy(ch->path.dgid.raw,
- event->param.rej_rcvd.ari, 16);
+ memcpy(dgid->raw, event->param.rej_rcvd.ari, 16);
shost_printk(KERN_DEBUG, shost,
PFX "Topspin/Cisco redirect to target port GID %016llx%016llx\n",
- be64_to_cpu(ch->path.dgid.global.subnet_prefix),
- be64_to_cpu(ch->path.dgid.global.interface_id));
+ be64_to_cpu(dgid->global.subnet_prefix),
+ be64_to_cpu(dgid->global.interface_id));
ch->status = SRP_PORT_REDIRECT;
} else {
@@ -2490,7 +2654,8 @@ static void srp_cm_rej_handler(struct ib_cm_id *cm_id,
shost_printk(KERN_WARNING, shost, PFX
"SRP LOGIN from %pI6 to %pI6 REJECTED, reason 0x%08x\n",
target->sgid.raw,
- target->orig_dgid.raw, reason);
+ target->ib_cm.orig_dgid.raw,
+ reason);
} else
shost_printk(KERN_WARNING, shost,
" REJ reason: IB_CM_REJ_CONSUMER_DEFINED,"
@@ -2510,7 +2675,7 @@ static void srp_cm_rej_handler(struct ib_cm_id *cm_id,
}
}
-static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
+static int srp_ib_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
{
struct srp_rdma_ch *ch = cm_id->context;
struct srp_target_port *target = ch->target;
@@ -2533,7 +2698,7 @@ static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
shost_printk(KERN_DEBUG, target->scsi_host, PFX "REJ received\n");
comp = 1;
- srp_cm_rej_handler(cm_id, event, ch);
+ srp_ib_cm_rej_handler(cm_id, event, ch);
break;
case IB_CM_DREQ_RECEIVED:
@@ -2571,6 +2736,135 @@ static int srp_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
return 0;
}
+static void srp_rdma_cm_rej_handler(struct srp_rdma_ch *ch,
+ struct rdma_cm_event *event)
+{
+ struct srp_target_port *target = ch->target;
+ struct Scsi_Host *shost = target->scsi_host;
+ int opcode;
+
+ switch (event->status) {
+ case IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID:
+ shost_printk(KERN_WARNING, shost,
+ " REJ reason: IB_CM_REJ_DUPLICATE_LOCAL_COMM_ID\n");
+ ch->status = -ECONNRESET;
+ break;
+
+ case IB_CM_REJ_CONSUMER_DEFINED:
+ opcode = *(u8 *) event->param.conn.private_data;
+ if (opcode == SRP_LOGIN_REJ) {
+ struct srp_login_rej *rej =
+ (struct srp_login_rej *)
+ event->param.conn.private_data;
+ u32 reason = be32_to_cpu(rej->reason);
+
+ if (reason == SRP_LOGIN_REJ_REQ_IT_IU_LENGTH_TOO_LARGE)
+ shost_printk(KERN_WARNING, shost,
+ PFX "SRP_LOGIN_REJ: requested max_it_iu_len too large\n");
+ else
+ shost_printk(KERN_WARNING, shost,
+ PFX "SRP LOGIN REJECTED, reason 0x%08x\n", reason);
+ } else {
+ shost_printk(KERN_WARNING, shost,
+ " REJ reason: IB_CM_REJ_CONSUMER_DEFINED, opcode 0x%02x\n",
+ opcode);
+ }
+ ch->status = -ECONNRESET;
+ break;
+
+ case IB_CM_REJ_STALE_CONN:
+ shost_printk(KERN_WARNING, shost,
+ " REJ reason: stale connection\n");
+ ch->status = SRP_STALE_CONN;
+ break;
+
+ default:
+ shost_printk(KERN_WARNING, shost, " REJ reason 0x%x\n",
+ event->status);
+ ch->status = -ECONNRESET;
+ break;
+ }
+}
+
+static int srp_rdma_cm_handler(struct rdma_cm_id *cm_id,
+ struct rdma_cm_event *event)
+{
+ struct srp_rdma_ch *ch = cm_id->context;
+ struct srp_target_port *target = ch->target;
+ int comp = 0;
+
+ switch (event->event) {
+ case RDMA_CM_EVENT_ADDR_RESOLVED:
+ ch->status = 0;
+ comp = 1;
+ break;
+
+ case RDMA_CM_EVENT_ADDR_ERROR:
+ ch->status = -ENXIO;
+ comp = 1;
+ break;
+
+ case RDMA_CM_EVENT_ROUTE_RESOLVED:
+ ch->status = 0;
+ comp = 1;
+ break;
+
+ case RDMA_CM_EVENT_ROUTE_ERROR:
+ case RDMA_CM_EVENT_UNREACHABLE:
+ ch->status = -EHOSTUNREACH;
+ comp = 1;
+ break;
+
+ case RDMA_CM_EVENT_CONNECT_ERROR:
+ shost_printk(KERN_DEBUG, target->scsi_host,
+ PFX "Sending CM REQ failed\n");
+ comp = 1;
+ ch->status = -ECONNRESET;
+ break;
+
+ case RDMA_CM_EVENT_ESTABLISHED:
+ comp = 1;
+ srp_cm_rep_handler(NULL, event->param.conn.private_data, ch);
+ break;
+
+ case RDMA_CM_EVENT_REJECTED:
+ shost_printk(KERN_DEBUG, target->scsi_host, PFX "REJ received\n");
+ comp = 1;
+
+ srp_rdma_cm_rej_handler(ch, event);
+ break;
+
+ case RDMA_CM_EVENT_DISCONNECTED:
+ if (ch->connected) {
+ shost_printk(KERN_WARNING, target->scsi_host,
+ PFX "received DREQ\n");
+ rdma_disconnect(ch->rdma_cm.cm_id);
+ comp = 1;
+ ch->status = 0;
+ queue_work(system_long_wq, &target->tl_err_work);
+ }
+ break;
+
+ case RDMA_CM_EVENT_TIMEWAIT_EXIT:
+ shost_printk(KERN_ERR, target->scsi_host,
+ PFX "connection closed\n");
+
+ comp = 1;
+ ch->status = 0;
+ break;
+
+ default:
+ shost_printk(KERN_WARNING, target->scsi_host,
+ PFX "Unhandled CM event %d\n", event->event);
+ break;
+ }
+
+ if (comp)
+ complete(&ch->done);
+
+ return 0;
+}
+
/**
* srp_change_queue_depth - setting device queue depth
* @sdev: scsi device struct
@@ -2772,7 +3066,10 @@ static ssize_t show_service_id(struct device *dev,
{
struct srp_target_port *target = host_to_target(class_to_shost(dev));
- return sprintf(buf, "0x%016llx\n", be64_to_cpu(target->service_id));
+ if (target->using_rdma_cm)
+ return -ENOENT;
+ return sprintf(buf, "0x%016llx\n",
+ be64_to_cpu(target->ib_cm.service_id));
}
static ssize_t show_pkey(struct device *dev, struct device_attribute *attr,
@@ -2780,7 +3077,9 @@ static ssize_t show_pkey(struct device *dev, struct device_attribute *attr,
{
struct srp_target_port *target = host_to_target(class_to_shost(dev));
- return sprintf(buf, "0x%04x\n", be16_to_cpu(target->pkey));
+ if (target->using_rdma_cm)
+ return -ENOENT;
+ return sprintf(buf, "0x%04x\n", be16_to_cpu(target->ib_cm.pkey));
}
static ssize_t show_sgid(struct device *dev, struct device_attribute *attr,
@@ -2797,7 +3096,9 @@ static ssize_t show_dgid(struct device *dev, struct device_attribute *attr,
struct srp_target_port *target = host_to_target(class_to_shost(dev));
struct srp_rdma_ch *ch = &target->ch[0];
- return sprintf(buf, "%pI6\n", ch->path.dgid.raw);
+ if (target->using_rdma_cm)
+ return -ENOENT;
+ return sprintf(buf, "%pI6\n", ch->ib_cm.path.dgid.raw);
}
static ssize_t show_orig_dgid(struct device *dev,
@@ -2805,7 +3106,9 @@ static ssize_t show_orig_dgid(struct device *dev,
{
struct srp_target_port *target = host_to_target(class_to_shost(dev));
- return sprintf(buf, "%pI6\n", target->orig_dgid.raw);
+ if (target->using_rdma_cm)
+ return -ENOENT;
+ return sprintf(buf, "%pI6\n", target->ib_cm.orig_dgid.raw);
}
static ssize_t show_req_lim(struct device *dev,
@@ -3050,6 +3353,9 @@ static bool srp_conn_unique(struct srp_host *host,
if (t != target &&
target->id_ext == t->id_ext &&
target->ioc_guid == t->ioc_guid &&
+ (!target->using_rdma_cm ||
+ memcmp(&target->rdma_cm.dst, &t->rdma_cm.dst,
+ sizeof(target->rdma_cm.dst)) == 0) &&
target->initiator_ext == t->initiator_ext) {
ret = false;
break;
@@ -3066,6 +3372,9 @@ static bool srp_conn_unique(struct srp_host *host,
*
* id_ext=<SRP ID ext>,ioc_guid=<SRP IOC GUID>,dgid=<dest GID>,
* pkey=<P_Key>,service_id=<service ID>
+ * or
+ * id_ext=<SRP ID ext>,ioc_guid=<SRP IOC GUID>,
+ * [src=<IPv4 address>,]dest=<IPv4 address>:<port number>
*
* to the add_target sysfs attribute.
*/
@@ -3086,11 +3395,19 @@ enum {
SRP_OPT_COMP_VECTOR = 1 << 12,
SRP_OPT_TL_RETRY_COUNT = 1 << 13,
SRP_OPT_QUEUE_SIZE = 1 << 14,
- SRP_OPT_ALL = (SRP_OPT_ID_EXT |
- SRP_OPT_IOC_GUID |
- SRP_OPT_DGID |
- SRP_OPT_PKEY |
- SRP_OPT_SERVICE_ID),
+ SRP_OPT_IP_SRC = 1 << 15,
+ SRP_OPT_IP_DEST = 1 << 16,
+};
+
+static unsigned int srp_opt_mandatory[] = {
+ SRP_OPT_ID_EXT |
+ SRP_OPT_IOC_GUID |
+ SRP_OPT_DGID |
+ SRP_OPT_PKEY |
+ SRP_OPT_SERVICE_ID,
+ SRP_OPT_ID_EXT |
+ SRP_OPT_IOC_GUID |
+ SRP_OPT_IP_DEST,
};
static const match_table_t srp_opt_tokens = {
@@ -3109,9 +3426,25 @@ static const match_table_t srp_opt_tokens = {
{ SRP_OPT_COMP_VECTOR, "comp_vector=%u" },
{ SRP_OPT_TL_RETRY_COUNT, "tl_retry_count=%u" },
{ SRP_OPT_QUEUE_SIZE, "queue_size=%d" },
+ { SRP_OPT_IP_SRC, "src=%s" },
+ { SRP_OPT_IP_DEST, "dest=%s" },
{ SRP_OPT_ERR, NULL }
};
+static int srp_parse_in(struct sockaddr_storage *sa, const char *addr_port_str)
+{
+ char *addr = kstrdup(addr_port_str, GFP_KERNEL);
+ char *port_str = addr;
+ int ret;
+
+ if (!addr)
+ return -ENOMEM;
+ strsep(&port_str, ":");
+ ret = inet_pton_with_scope(&init_net, AF_UNSPEC, addr, port_str, sa);
+ kfree(addr);
+ return ret;
+}
+
static int srp_parse_options(const char *buf, struct srp_target_port *target)
{
char *options, *sep_opt;
@@ -3180,7 +3513,7 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
goto out;
}
- ret = hex2bin(target->orig_dgid.raw, p, 16);
+ ret = hex2bin(target->ib_cm.orig_dgid.raw, p, 16);
kfree(p);
if (ret < 0)
goto out;
@@ -3191,7 +3524,7 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
pr_warn("bad P_Key parameter '%s'\n", p);
goto out;
}
- target->pkey = cpu_to_be16(token);
+ target->ib_cm.pkey = cpu_to_be16(token);
break;
case SRP_OPT_SERVICE_ID:
@@ -3206,7 +3539,39 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
kfree(p);
goto out;
}
- target->service_id = cpu_to_be64(ull);
+ target->ib_cm.service_id = cpu_to_be64(ull);
+ kfree(p);
+ break;
+
+ case SRP_OPT_IP_SRC:
+ p = match_strdup(args);
+ if (!p) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = srp_parse_in(&target->rdma_cm.src.ss, p);
+ if (ret < 0) {
+ pr_warn("bad source parameter '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->rdma_cm.src_specified = true;
+ kfree(p);
+ break;
+
+ case SRP_OPT_IP_DEST:
+ p = match_strdup(args);
+ if (!p) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ ret = srp_parse_in(&target->rdma_cm.dst.ss, p);
+ if (ret < 0) {
+ pr_warn("bad dest parameter '%s'\n", p);
+ kfree(p);
+ goto out;
+ }
+ target->using_rdma_cm = true;
kfree(p);
break;
@@ -3321,14 +3686,14 @@ static int srp_parse_options(const char *buf, struct srp_target_port *target)
}
}
- if ((opt_mask & SRP_OPT_ALL) == SRP_OPT_ALL)
- ret = 0;
- else
- for (i = 0; i < ARRAY_SIZE(srp_opt_tokens); ++i)
- if ((srp_opt_tokens[i].token & SRP_OPT_ALL) &&
- !(srp_opt_tokens[i].token & opt_mask))
- pr_warn("target creation request is missing parameter '%s'\n",
- srp_opt_tokens[i].pattern);
+ for (i = 0; i < ARRAY_SIZE(srp_opt_mandatory); i++) {
+ if ((opt_mask & srp_opt_mandatory[i]) == srp_opt_mandatory[i]) {
+ ret = 0;
+ break;
+ }
+ }
+ if (ret)
+ pr_warn("target creation request is missing one or more parameters\n");
if (target->scsi_host->cmd_per_lun > target->scsi_host->can_queue
&& (opt_mask & SRP_OPT_MAX_CMD_PER_LUN))
@@ -3397,11 +3762,22 @@ static ssize_t srp_create_target(struct device *dev,
target->req_ring_size = target->queue_size - SRP_TSK_MGMT_SQ_SIZE;
if (!srp_conn_unique(target->srp_host, target)) {
- shost_printk(KERN_INFO, target->scsi_host,
- PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;initiator_ext=%016llx\n",
- be64_to_cpu(target->id_ext),
- be64_to_cpu(target->ioc_guid),
- be64_to_cpu(target->initiator_ext));
+ if (target->using_rdma_cm) {
+ char dst_addr[64];
+
+ shost_printk(KERN_INFO, target->scsi_host,
+ PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;dest=%s\n",
+ be64_to_cpu(target->id_ext),
+ be64_to_cpu(target->ioc_guid),
+ inet_ntop(&target->rdma_cm.dst, dst_addr,
+ sizeof(dst_addr)));
+ } else {
+ shost_printk(KERN_INFO, target->scsi_host,
+ PFX "Already connected to target port with id_ext=%016llx;ioc_guid=%016llx;initiator_ext=%016llx\n",
+ be64_to_cpu(target->id_ext),
+ be64_to_cpu(target->ioc_guid),
+ be64_to_cpu(target->initiator_ext));
+ }
ret = -EEXIST;
goto out;
}
@@ -3502,11 +3878,18 @@ static ssize_t srp_create_target(struct device *dev,
ret = srp_connect_ch(ch, multich);
if (ret) {
+ char dst[64];
+
+ if (target->using_rdma_cm)
+ inet_ntop(&target->rdma_cm.dst, dst,
+ sizeof(dst));
+ else
+ snprintf(dst, sizeof(dst), "%pI6",
+ target->ib_cm.orig_dgid.raw);
shost_printk(KERN_ERR, target->scsi_host,
- PFX "Connection %d/%d to %pI6 failed\n",
+ PFX "Connection %d/%d to %s failed\n",
ch_start + cpu_idx,
- target->ch_count,
- ch->target->orig_dgid.raw);
+ target->ch_count, dst);
if (node_idx == 0 && cpu_idx == 0) {
goto free_ch;
} else {
@@ -3531,13 +3914,25 @@ static ssize_t srp_create_target(struct device *dev,
goto err_disconnect;
if (target->state != SRP_TARGET_REMOVED) {
- shost_printk(KERN_DEBUG, target->scsi_host, PFX
- "new target: id_ext %016llx ioc_guid %016llx pkey %04x service_id %016llx sgid %pI6 dgid %pI6\n",
- be64_to_cpu(target->id_ext),
- be64_to_cpu(target->ioc_guid),
- be16_to_cpu(target->pkey),
- be64_to_cpu(target->service_id),
- target->sgid.raw, target->orig_dgid.raw);
+ if (target->using_rdma_cm) {
+ char dst[64];
+
+ inet_ntop(&target->rdma_cm.dst, dst, sizeof(dst));
+ shost_printk(KERN_DEBUG, target->scsi_host, PFX
+ "new target: id_ext %016llx ioc_guid %016llx sgid %pI6 dest %s\n",
+ be64_to_cpu(target->id_ext),
+ be64_to_cpu(target->ioc_guid),
+ target->sgid.raw, dst);
+ } else {
+ shost_printk(KERN_DEBUG, target->scsi_host, PFX
+ "new target: id_ext %016llx ioc_guid %016llx pkey %04x service_id %016llx sgid %pI6 dgid %pI6\n",
+ be64_to_cpu(target->id_ext),
+ be64_to_cpu(target->ioc_guid),
+ be16_to_cpu(target->ib_cm.pkey),
+ be64_to_cpu(target->ib_cm.service_id),
+ target->sgid.raw,
+ target->ib_cm.orig_dgid.raw);
+ }
}
ret = count;
diff --git a/drivers/infiniband/ulp/srp/ib_srp.h b/drivers/infiniband/ulp/srp/ib_srp.h
index a814f5ef16f9..ce9d8e0f84dc 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.h
+++ b/drivers/infiniband/ulp/srp/ib_srp.h
@@ -45,6 +45,7 @@
#include <rdma/ib_sa.h>
#include <rdma/ib_cm.h>
#include <rdma/ib_fmr_pool.h>
+#include <rdma/rdma_cm.h>
enum {
SRP_PATH_REC_TIMEOUT_MS = 1000,
@@ -153,11 +154,18 @@ struct srp_rdma_ch {
struct completion done;
int status;
- struct sa_path_rec path;
- struct ib_sa_query *path_query;
- int path_query_id;
+ union {
+ struct ib_cm {
+ struct sa_path_rec path;
+ struct ib_sa_query *path_query;
+ int path_query_id;
+ struct ib_cm_id *cm_id;
+ } ib_cm;
+ struct rdma_cm {
+ struct rdma_cm_id *cm_id;
+ } rdma_cm;
+ };
- struct ib_cm_id *cm_id;
struct srp_iu **tx_ring;
struct srp_iu **rx_ring;
struct srp_request *req_ring;
@@ -194,7 +202,6 @@ struct srp_target_port {
union ib_gid sgid;
__be64 id_ext;
__be64 ioc_guid;
- __be64 service_id;
__be64 initiator_ext;
u16 io_class;
struct srp_host *srp_host;
@@ -210,8 +217,28 @@ struct srp_target_port {
int comp_vector;
int tl_retry_count;
- union ib_gid orig_dgid;
- __be16 pkey;
+ bool using_rdma_cm;
+
+ union {
+ struct {
+ __be64 service_id;
+ union ib_gid orig_dgid;
+ __be16 pkey;
+ } ib_cm;
+ struct {
+ union {
+ struct sockaddr_in ip4;
+ struct sockaddr_in6 ip6;
+ struct sockaddr_storage ss;
+ } src;
+ union {
+ struct sockaddr_in ip4;
+ struct sockaddr_in6 ip6;
+ struct sockaddr_storage ss;
+ } dst;
+ bool src_specified;
+ } rdma_cm;
+ };
u32 rq_tmo_jiffies;
diff --git a/include/scsi/srp.h b/include/scsi/srp.h
index 5be834de491a..c16a3c9a4d9b 100644
--- a/include/scsi/srp.h
+++ b/include/scsi/srp.h
@@ -129,6 +129,23 @@ struct srp_login_req {
u8 target_port_id[16];
};
+/**
+ * struct srp_login_req_rdma - RDMA/CM login parameters.
+ *
+ * RDMA/CM over InfiniBand can only carry 92 - 36 = 56 bytes of private
+ * data. The %srp_login_req_rdma structure contains the same information as
+ * %srp_login_req but with the reserved data removed.
+ */
+struct srp_login_req_rdma {
+ u64 tag;
+ __be16 req_buf_fmt;
+ u8 req_flags;
+ u8 opcode;
+ __be32 req_it_iu_len;
+ u8 initiator_port_id[16];
+ u8 target_port_id[16];
+};
+
/*
* The SRP spec defines the size of the LOGIN_RSP structure to be 52
* bytes, so it needs to be packed to avoid having it padded to 56
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 07/28] IB/srpt: Remove an unused structure member
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (3 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 04/28] IB/srp: Add RDMA/CM support Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 08/28] IB/srpt: Fix kernel-doc warnings in ib_srpt.c Bart Van Assche
` (21 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Fixes: commit a42d985bd5b2 ("ib_srpt: Initial SRP Target merge for v3.3-rc1")
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 1 -
drivers/infiniband/ulp/srpt/ib_srpt.h | 1 -
2 files changed, 2 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index bfa576aa9f03..81e8085bc5bb 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1123,7 +1123,6 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
ioctx->state = SRPT_STATE_NEW;
ioctx->n_rdma = 0;
ioctx->n_rw_ctx = 0;
- init_completion(&ioctx->tx_done);
ioctx->queue_status_only = false;
/*
* transport_init_se_cmd() does not initialize all fields, so do it
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 673387d365a3..7fff73a5d1e6 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -207,7 +207,6 @@ struct srpt_send_ioctx {
spinlock_t spinlock;
enum srpt_command_state state;
struct se_cmd cmd;
- struct completion tx_done;
u8 n_rdma;
u8 n_rw_ctx;
bool queue_status_only;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 08/28] IB/srpt: Fix kernel-doc warnings in ib_srpt.c
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (4 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 07/28] IB/srpt: Remove an unused structure member Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 09/28] IB/srpt: Make it safe to use RCU for srpt_device.rch_list Bart Van Assche
` (20 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Avoid that warnings about missing parameter descriptions are reported
when building with W=1.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 90 ++++++++++++++++++-----------------
1 file changed, 46 insertions(+), 44 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 81e8085bc5bb..ebdd5a4f407b 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -119,7 +119,7 @@ static bool srpt_set_ch_state(struct srpt_rdma_ch *ch, enum rdma_ch_state new)
return changed;
}
-/**
+/*
* srpt_event_handler() - Asynchronous IB event callback function.
*
* Callback function called by the InfiniBand core when an asynchronous IB
@@ -168,7 +168,7 @@ static void srpt_event_handler(struct ib_event_handler *handler,
}
}
-/**
+/*
* srpt_srq_event() - SRQ event callback function.
*/
static void srpt_srq_event(struct ib_event *event, void *ctx)
@@ -193,7 +193,7 @@ static const char *get_ch_state_name(enum rdma_ch_state s)
return "???";
}
-/**
+/*
* srpt_qp_event() - QP event callback function.
*/
static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch)
@@ -218,7 +218,7 @@ static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch)
/**
* srpt_set_ioc() - Helper function for initializing an IOUnitInfo structure.
- *
+ * @c_list: I/O controller list.
* @slot: one-based slot number.
* @value: four-bit value.
*
@@ -240,7 +240,7 @@ static void srpt_set_ioc(u8 *c_list, u32 slot, u8 value)
}
}
-/**
+/*
* srpt_get_class_port_info() - Copy ClassPortInfo to a management datagram.
*
* See also section 16.3.3.1 ClassPortInfo in the InfiniBand Architecture
@@ -259,7 +259,7 @@ static void srpt_get_class_port_info(struct ib_dm_mad *mad)
mad->mad_hdr.status = 0;
}
-/**
+/*
* srpt_get_iou() - Write IOUnitInfo to a management datagram.
*
* See also section 16.3.3.3 IOUnitInfo in the InfiniBand Architecture
@@ -283,7 +283,7 @@ static void srpt_get_iou(struct ib_dm_mad *mad)
mad->mad_hdr.status = 0;
}
-/**
+/*
* srpt_get_ioc() - Write IOControllerprofile to a management datagram.
*
* See also section 16.3.3.4 IOControllerProfile in the InfiniBand
@@ -341,7 +341,7 @@ static void srpt_get_ioc(struct srpt_port *sport, u32 slot,
mad->mad_hdr.status = 0;
}
-/**
+/*
* srpt_get_svc_entries() - Write ServiceEntries to a management datagram.
*
* See also section 16.3.3.5 ServiceEntries in the InfiniBand Architecture
@@ -418,7 +418,7 @@ static void srpt_mgmt_method_get(struct srpt_port *sp, struct ib_mad *rq_mad,
}
}
-/**
+/*
* srpt_mad_send_handler() - Post MAD-send callback function.
*/
static void srpt_mad_send_handler(struct ib_mad_agent *mad_agent,
@@ -428,7 +428,7 @@ static void srpt_mad_send_handler(struct ib_mad_agent *mad_agent,
ib_free_send_mad(mad_wc->send_buf);
}
-/**
+/*
* srpt_mad_recv_handler() - MAD reception callback function.
*/
static void srpt_mad_recv_handler(struct ib_mad_agent *mad_agent,
@@ -493,7 +493,7 @@ static void srpt_mad_recv_handler(struct ib_mad_agent *mad_agent,
ib_free_recv_mad(mad_wc);
}
-/**
+/*
* srpt_refresh_port() - Configure a HCA port.
*
* Enable InfiniBand management datagram processing, update the cached sm_lid,
@@ -576,7 +576,7 @@ static int srpt_refresh_port(struct srpt_port *sport)
return ret;
}
-/**
+/*
* srpt_unregister_mad_agent() - Unregister MAD callback functions.
*
* Note: It is safe to call this function more than once for the same device.
@@ -601,7 +601,7 @@ static void srpt_unregister_mad_agent(struct srpt_device *sdev)
}
}
-/**
+/*
* srpt_alloc_ioctx() - Allocate an SRPT I/O context structure.
*/
static struct srpt_ioctx *srpt_alloc_ioctx(struct srpt_device *sdev,
@@ -632,7 +632,7 @@ static struct srpt_ioctx *srpt_alloc_ioctx(struct srpt_device *sdev,
return NULL;
}
-/**
+/*
* srpt_free_ioctx() - Free an SRPT I/O context structure.
*/
static void srpt_free_ioctx(struct srpt_device *sdev, struct srpt_ioctx *ioctx,
@@ -684,7 +684,7 @@ static struct srpt_ioctx **srpt_alloc_ioctx_ring(struct srpt_device *sdev,
return ring;
}
-/**
+/*
* srpt_free_ioctx_ring() - Free the ring of SRPT I/O context structures.
*/
static void srpt_free_ioctx_ring(struct srpt_ioctx **ioctx_ring,
@@ -701,7 +701,7 @@ static void srpt_free_ioctx_ring(struct srpt_ioctx **ioctx_ring,
kfree(ioctx_ring);
}
-/**
+/*
* srpt_get_cmd_state() - Get the state of a SCSI command.
*/
static enum srpt_command_state srpt_get_cmd_state(struct srpt_send_ioctx *ioctx)
@@ -717,7 +717,7 @@ static enum srpt_command_state srpt_get_cmd_state(struct srpt_send_ioctx *ioctx)
return state;
}
-/**
+/*
* srpt_set_cmd_state() - Set the state of a SCSI command.
*
* Does not modify the state of aborted commands. Returns the previous command
@@ -740,7 +740,7 @@ static enum srpt_command_state srpt_set_cmd_state(struct srpt_send_ioctx *ioctx,
return previous;
}
-/**
+/*
* srpt_test_and_set_cmd_state() - Test and set the state of a command.
*
* Returns true if and only if the previous command state was equal to 'old'.
@@ -764,7 +764,7 @@ static bool srpt_test_and_set_cmd_state(struct srpt_send_ioctx *ioctx,
return previous == old;
}
-/**
+/*
* srpt_post_recv() - Post an IB receive request.
*/
static int srpt_post_recv(struct srpt_device *sdev, struct srpt_rdma_ch *ch,
@@ -790,7 +790,7 @@ static int srpt_post_recv(struct srpt_device *sdev, struct srpt_rdma_ch *ch,
return ib_post_recv(ch->qp, &wr, &bad_wr);
}
-/**
+/*
* srpt_zerolength_write() - Perform a zero-length RDMA write.
*
* A quote from the InfiniBand specification: C9-88: For an HCA responder
@@ -933,6 +933,8 @@ static inline void *srpt_get_desc_buf(struct srp_cmd *srp_cmd)
* @srp_cmd: Pointer to the SRP_CMD request data.
* @dir: Pointer to the variable to which the transfer direction will be
* written.
+ * @sg: [out] Pointer to the sg-list extracted from the SRP command.
+ * @sg_cnt: [out] Number of entries in @sg.
* @data_len: Pointer to the variable to which the total data length of all
* descriptors in the SRP_CMD request will be written.
*
@@ -997,7 +999,7 @@ static int srpt_get_desc_tbl(struct srpt_send_ioctx *ioctx,
}
}
-/**
+/*
* srpt_init_ch_qp() - Initialize queue pair attributes.
*
* Initialized the attributes of queue pair 'qp' by allowing local write,
@@ -1085,7 +1087,7 @@ static int srpt_ch_qp_rts(struct srpt_rdma_ch *ch, struct ib_qp *qp)
return ret;
}
-/**
+/*
* srpt_ch_qp_err() - Set the channel queue pair state to 'error'.
*/
static int srpt_ch_qp_err(struct srpt_rdma_ch *ch)
@@ -1096,7 +1098,7 @@ static int srpt_ch_qp_err(struct srpt_rdma_ch *ch)
return ib_modify_qp(ch->qp, &qp_attr, IB_QP_STATE);
}
-/**
+/*
* srpt_get_send_ioctx() - Obtain an I/O context for sending to the initiator.
*/
static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
@@ -1137,7 +1139,6 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
/**
* srpt_abort_cmd() - Abort a SCSI command.
* @ioctx: I/O context associated with the SCSI command.
- * @context: Preferred execution context.
*/
static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
{
@@ -1204,7 +1205,7 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
return state;
}
-/**
+/*
* XXX: what is now target_execute_cmd used to be asynchronous, and unmapping
* the data that has been transferred via IB RDMA had to be postponed until the
* check_stop_free() callback. None of this is necessary anymore and needs to
@@ -1343,7 +1344,7 @@ static int srpt_check_stop_free(struct se_cmd *cmd)
return target_put_sess_cmd(&ioctx->cmd);
}
-/**
+/*
* srpt_handle_cmd() - Process SRP_CMD.
*/
static void srpt_handle_cmd(struct srpt_rdma_ch *ch,
@@ -1425,7 +1426,7 @@ static int srp_tmr_to_tcm(int fn)
}
}
-/**
+/*
* srpt_handle_tsk_mgmt() - Process an SRP_TSK_MGMT information unit.
*
* Returns 0 if and only if the request will be processed by the target core.
@@ -1471,7 +1472,8 @@ static void srpt_handle_tsk_mgmt(struct srpt_rdma_ch *ch,
/**
* srpt_handle_new_iu() - Process a newly received information unit.
* @ch: RDMA channel through which the information unit has been received.
- * @ioctx: SRPT I/O context associated with the information unit.
+ * @recv_ioctx: Receive I/O context associated with the information unit.
+ * @send_ioctx: Send I/O context associated with the information unit.
*/
static void srpt_handle_new_iu(struct srpt_rdma_ch *ch,
struct srpt_recv_ioctx *recv_ioctx,
@@ -1576,7 +1578,7 @@ static void srpt_process_wait_list(struct srpt_rdma_ch *ch)
}
}
-/**
+/*
* Note: Although this has not yet been observed during tests, at least in
* theory it is possible that the srpt_get_send_ioctx() call invoked by
* srpt_handle_new_iu() fails. This is possible because the req_lim_delta
@@ -1617,7 +1619,7 @@ static void srpt_send_done(struct ib_cq *cq, struct ib_wc *wc)
srpt_process_wait_list(ch);
}
-/**
+/*
* srpt_create_ch_ib() - Create receive and send completion queues.
*/
static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
@@ -1716,7 +1718,7 @@ static void srpt_destroy_ch_ib(struct srpt_rdma_ch *ch)
ib_free_cq(ch->cq);
}
-/**
+/*
* srpt_close_ch() - Close an RDMA channel.
*
* Make sure all resources associated with the channel will be deallocated at
@@ -1899,7 +1901,7 @@ static void srpt_release_channel_work(struct work_struct *w)
kref_put(&ch->kref, srpt_free_ch);
}
-/**
+/*
* srpt_cm_req_recv() - Process the event IB_CM_REQ_RECEIVED.
*
* Ownership of the cm_id is transferred to the target session if this
@@ -2204,7 +2206,7 @@ static void srpt_cm_rej_recv(struct srpt_rdma_ch *ch,
kfree(priv);
}
-/**
+/*
* srpt_cm_rtu_recv() - Process an IB_CM_RTU_RECEIVED or USER_ESTABLISHED event.
*
* An IB_CM_RTU_RECEIVED message indicates that the connection is established
@@ -2227,7 +2229,7 @@ static void srpt_cm_rtu_recv(struct srpt_rdma_ch *ch)
}
}
-/**
+/*
* srpt_cm_handler() - IB connection manager callback function.
*
* A non-zero return value will cause the caller destroy the CM ID.
@@ -2353,7 +2355,7 @@ static u8 tcm_to_srp_tsk_mgmt_status(const int tcm_mgmt_status)
return SRP_TSK_MGMT_FAILED;
}
-/**
+/*
* srpt_queue_response() - Transmits the response to a SCSI command.
*
* Callback function called by the TCM core. Must not block since it can be
@@ -2493,7 +2495,7 @@ static void srpt_refresh_port_work(struct work_struct *work)
srpt_refresh_port(sport);
}
-/**
+/*
* srpt_release_sdev() - Free the channel resources associated with a target.
*/
static int srpt_release_sdev(struct srpt_device *sdev)
@@ -2621,7 +2623,7 @@ static int srpt_use_srq(struct srpt_device *sdev, bool use_srq)
return ret;
}
-/**
+/*
* srpt_add_one() - Infiniband device addition callback function.
*/
static void srpt_add_one(struct ib_device *device)
@@ -2719,7 +2721,7 @@ static void srpt_add_one(struct ib_device *device)
goto out;
}
-/**
+/*
* srpt_remove_one() - InfiniBand device removal callback function.
*/
static void srpt_remove_one(struct ib_device *device, void *client_data)
@@ -2825,7 +2827,7 @@ static void srpt_release_cmd(struct se_cmd *se_cmd)
spin_unlock_irqrestore(&ch->spinlock, flags);
}
-/**
+/*
* srpt_close_session() - Forcibly close a session.
*
* Callback function invoked by the TCM core to clean up sessions associated
@@ -2842,7 +2844,7 @@ static void srpt_close_session(struct se_session *se_sess)
mutex_unlock(&sdev->mutex);
}
-/**
+/*
* srpt_sess_get_index() - Return the value of scsiAttIntrPortIndex (SCSI-MIB).
*
* A quote from RFC 4455 (SCSI-MIB) about this MIB object:
@@ -3132,7 +3134,7 @@ static struct configfs_attribute *srpt_tpg_attrs[] = {
NULL,
};
-/**
+/*
* configfs callback invoked for
* mkdir /sys/kernel/config/target/$driver/$port/$tpg
*/
@@ -3155,7 +3157,7 @@ static struct se_portal_group *srpt_make_tpg(struct se_wwn *wwn,
return tpg;
}
-/**
+/*
* configfs callback invoked for
* rmdir /sys/kernel/config/target/$driver/$port/$tpg
*/
@@ -3167,7 +3169,7 @@ static void srpt_drop_tpg(struct se_portal_group *tpg)
core_tpg_deregister(tpg);
}
-/**
+/*
* configfs callback invoked for
* mkdir /sys/kernel/config/target/$driver/$port
*/
@@ -3178,7 +3180,7 @@ static struct se_wwn *srpt_make_tport(struct target_fabric_configfs *tf,
return srpt_lookup_wwn(name) ? : ERR_PTR(-EINVAL);
}
-/**
+/*
* configfs callback invoked for
* rmdir /sys/kernel/config/target/$driver/$port
*/
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 09/28] IB/srpt: Make it safe to use RCU for srpt_device.rch_list
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (5 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 08/28] IB/srpt: Fix kernel-doc warnings in ib_srpt.c Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 10/28] IB/srpt: Rework srpt_disconnect_ch_sync() Bart Van Assche
` (19 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
The next patch will iterate over rch_list from a context from which
it is not allowed to block. Hence make rch_list RCU-safe.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 5 +++--
drivers/infiniband/ulp/srpt/ib_srpt.h | 2 ++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index ebdd5a4f407b..d7143cf6a882 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1852,7 +1852,7 @@ static void srpt_free_ch(struct kref *kref)
{
struct srpt_rdma_ch *ch = container_of(kref, struct srpt_rdma_ch, kref);
- kfree(ch);
+ kfree_rcu(ch, rcu);
}
static void srpt_release_channel_work(struct work_struct *w)
@@ -2016,6 +2016,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto reject;
}
+ init_rcu_head(&ch->rcu);
kref_init(&ch->kref);
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
@@ -2144,7 +2145,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
}
mutex_lock(&sdev->mutex);
- list_add_tail(&ch->list, &sdev->rch_list);
+ list_add_tail_rcu(&ch->list, &sdev->rch_list);
mutex_unlock(&sdev->mutex);
goto out;
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 7fff73a5d1e6..d2c08ca2bd74 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -236,6 +236,7 @@ enum rdma_ch_state {
* @cm_id: IB CM ID associated with the channel.
* @qp: IB queue pair used for communicating over this channel.
* @cq: IB completion queue for this channel.
+ * @rcu: RCU head.
* @rq_size: IB receive queue size.
* @rsp_size IB response message size in bytes.
* @sq_wr_avail: number of work requests available in the send queue.
@@ -267,6 +268,7 @@ struct srpt_rdma_ch {
struct ib_qp *qp;
struct ib_cq *cq;
struct ib_cqe zw_cqe;
+ struct rcu_head rcu;
struct kref kref;
int rq_size;
u32 rsp_size;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 10/28] IB/srpt: Rework srpt_disconnect_ch_sync()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (6 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 09/28] IB/srpt: Make it safe to use RCU for srpt_device.rch_list Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 11/28] IB/srpt: Document all structure members in ib_srpt.h Bart Van Assche
` (18 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
This patch fixes a use-after-free issue for ch->release_done when
running the SRP protocol on top of the rdma_rxe driver.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 40 +++++++++++++++++++++--------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 2 --
2 files changed, 24 insertions(+), 18 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index d7143cf6a882..943fec0d0548 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1787,6 +1787,24 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
return ret;
}
+static bool srpt_ch_closed(struct srpt_device *sdev, struct srpt_rdma_ch *ch)
+{
+ struct srpt_rdma_ch *ch2;
+ bool res = true;
+
+ rcu_read_lock();
+ list_for_each_entry(ch2, &sdev->rch_list, list) {
+ if (ch2 == ch) {
+ res = false;
+ goto done;
+ }
+ }
+done:
+ rcu_read_unlock();
+
+ return res;
+}
+
/*
* Send DREQ and wait for DREP. Return true if and only if this function
* changed the state of @ch.
@@ -1794,31 +1812,24 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
static bool srpt_disconnect_ch_sync(struct srpt_rdma_ch *ch)
__must_hold(&sdev->mutex)
{
- DECLARE_COMPLETION_ONSTACK(release_done);
struct srpt_device *sdev = ch->sport->sdev;
- bool wait;
+ int ret;
lockdep_assert_held(&sdev->mutex);
pr_debug("ch %s-%d state %d\n", ch->sess_name, ch->qp->qp_num,
ch->state);
- WARN_ON(ch->release_done);
- ch->release_done = &release_done;
- wait = !list_empty(&ch->list);
- srpt_disconnect_ch(ch);
+ ret = srpt_disconnect_ch(ch);
mutex_unlock(&sdev->mutex);
- if (!wait)
- goto out;
-
- while (wait_for_completion_timeout(&release_done, 180 * HZ) == 0)
+ while (wait_event_timeout(sdev->ch_releaseQ, srpt_ch_closed(sdev, ch),
+ 5 * HZ) == 0)
pr_info("%s(%s-%d state %d): still waiting ...\n", __func__,
ch->sess_name, ch->qp->qp_num, ch->state);
-out:
mutex_lock(&sdev->mutex);
- return wait;
+ return ret == 0;
}
static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
@@ -1862,8 +1873,7 @@ static void srpt_release_channel_work(struct work_struct *w)
struct se_session *se_sess;
ch = container_of(w, struct srpt_rdma_ch, release_work);
- pr_debug("%s: %s-%d; release_done = %p\n", __func__, ch->sess_name,
- ch->qp->qp_num, ch->release_done);
+ pr_debug("%s: %s-%d\n", __func__, ch->sess_name, ch->qp->qp_num);
sdev = ch->sport->sdev;
BUG_ON(!sdev);
@@ -1892,8 +1902,6 @@ static void srpt_release_channel_work(struct work_struct *w)
mutex_lock(&sdev->mutex);
list_del_init(&ch->list);
- if (ch->release_done)
- complete(ch->release_done);
mutex_unlock(&sdev->mutex);
wake_up(&sdev->ch_releaseQ);
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index d2c08ca2bd74..3f4142edc53b 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -261,7 +261,6 @@ enum rdma_ch_state {
* @sess_name: Session name.
* @ini_guid: Initiator port GUID.
* @release_work: Allows scheduling of srpt_release_channel().
- * @release_done: Enables waiting for srpt_release_channel() completion.
*/
struct srpt_rdma_ch {
struct ib_cm_id *cm_id;
@@ -290,7 +289,6 @@ struct srpt_rdma_ch {
u8 sess_name[36];
u8 ini_guid[24];
struct work_struct release_work;
- struct completion *release_done;
};
/**
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 11/28] IB/srpt: Document all structure members in ib_srpt.h
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (7 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 10/28] IB/srpt: Rework srpt_disconnect_ch_sync() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 12/28] IB/srpt: Rename a local variable, a member variable and a constant Bart Van Assche
` (17 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
This patch avoids that the following command reports any warnings:
scripts/kernel-doc -none drivers/infiniband/ulp/srpt/ib_srpt.h
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.h | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 3f4142edc53b..3cf917720b35 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -159,6 +159,7 @@ enum srpt_command_state {
/**
* struct srpt_ioctx - Shared SRPT I/O context information.
+ * @cqe: Completion queue element.
* @buf: Pointer to the buffer.
* @dma: DMA address of the buffer.
* @index: Index of the I/O context in its ioctx_ring array.
@@ -190,10 +191,18 @@ struct srpt_rw_ctx {
* struct srpt_send_ioctx - SRPT send I/O context.
* @ioctx: See above.
* @ch: Channel pointer.
+ * @s_rw_ctx: @rw_ctxs points here if only a single rw_ctx is needed.
+ * @rw_ctxs: RDMA read/write contexts.
+ * @rdma_cqe: RDMA completion queue element.
+ * @free_list: Node in srpt_rdma_ch.free_list.
* @spinlock: Protects 'state'.
* @state: I/O context state.
* @cmd: Target core command data structure.
* @sense_data: SCSI sense data.
+ * @n_rdma: Number of work requests needed to transfer this ioctx.
+ * @n_rw_ctx: Size of rw_ctxs array.
+ * @queue_status_only: Send a SCSI status back to the initiator but no data.
+ * @sense_data: Sense data to be sent to the initiator.
*/
struct srpt_send_ioctx {
struct srpt_ioctx ioctx;
@@ -236,9 +245,11 @@ enum rdma_ch_state {
* @cm_id: IB CM ID associated with the channel.
* @qp: IB queue pair used for communicating over this channel.
* @cq: IB completion queue for this channel.
+ * @zw_cqe: Zero-length write CQE.
* @rcu: RCU head.
+ * @kref: kref for this channel.
* @rq_size: IB receive queue size.
- * @rsp_size IB response message size in bytes.
+ * @rsp_size: IB response message size in bytes.
* @sq_wr_avail: number of work requests available in the send queue.
* @sport: pointer to the information of the HCA port used by this
* channel.
@@ -322,7 +333,7 @@ struct srpt_port_attrib {
* @port_guid_wwn: WWN associated with target port GUID.
* @port_gid_tpg: TPG associated with target port GID.
* @port_gid_wwn: WWN associated with target port GID.
- * @port_acl_list: Head of the list with all node ACLs for this port.
+ * @port_attrib: Port attributes that can be accessed through configfs.
*/
struct srpt_port {
struct srpt_device *sdev;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 12/28] IB/srpt: Rename a local variable, a member variable and a constant
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (8 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 11/28] IB/srpt: Document all structure members in ib_srpt.h Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 13/28] IB/srpt: Reduce the severity level of a log message Bart Van Assche
` (16 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Rename rsp_size into max_rsp_size and SRPT_RQ_SIZE into MAX_SRPT_RQ_SIZE.
The new names better reflect the role of this member variable and constant.
Since the prefix "srp_" is superfluous in the context of the function
that creates an RDMA channel, rename srp_sq_size into sq_size.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 26 +++++++++++++-------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 6 +++---
2 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 943fec0d0548..6cff7a9b4cbb 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -314,7 +314,7 @@ static void srpt_get_ioc(struct srpt_port *sport, u32 slot,
if (sdev->use_srq)
send_queue_depth = sdev->srq_size;
else
- send_queue_depth = min(SRPT_RQ_SIZE,
+ send_queue_depth = min(MAX_SRPT_RQ_SIZE,
sdev->device->attrs.max_qp_wr);
memset(iocp, 0, sizeof(*iocp));
@@ -1628,7 +1628,7 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
struct srpt_port *sport = ch->sport;
struct srpt_device *sdev = sport->sdev;
const struct ib_device_attr *attrs = &sdev->device->attrs;
- u32 srp_sq_size = sport->port_attrib.srp_sq_size;
+ int sq_size = sport->port_attrib.srp_sq_size;
int i, ret;
WARN_ON(ch->rq_size < 1);
@@ -1639,12 +1639,12 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
goto out;
retry:
- ch->cq = ib_alloc_cq(sdev->device, ch, ch->rq_size + srp_sq_size,
+ ch->cq = ib_alloc_cq(sdev->device, ch, ch->rq_size + sq_size,
0 /* XXX: spread CQs */, IB_POLL_WORKQUEUE);
if (IS_ERR(ch->cq)) {
ret = PTR_ERR(ch->cq);
pr_err("failed to create CQ cqe= %d ret= %d\n",
- ch->rq_size + srp_sq_size, ret);
+ ch->rq_size + sq_size, ret);
goto out;
}
@@ -1662,8 +1662,8 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
* both both, as RDMA contexts will also post completions for the
* RDMA READ case.
*/
- qp_init->cap.max_send_wr = min(srp_sq_size / 2, attrs->max_qp_wr + 0U);
- qp_init->cap.max_rdma_ctxs = srp_sq_size / 2;
+ qp_init->cap.max_send_wr = min(sq_size / 2, attrs->max_qp_wr);
+ qp_init->cap.max_rdma_ctxs = sq_size / 2;
qp_init->cap.max_send_sge = min(attrs->max_sge, SRPT_MAX_SG_PER_WQE);
qp_init->port_num = ch->sport->port;
if (sdev->use_srq) {
@@ -1677,8 +1677,8 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
if (IS_ERR(ch->qp)) {
ret = PTR_ERR(ch->qp);
if (ret == -ENOMEM) {
- srp_sq_size /= 2;
- if (srp_sq_size >= MIN_SRPT_SQ_SIZE) {
+ sq_size /= 2;
+ if (sq_size >= MIN_SRPT_SQ_SIZE) {
ib_destroy_cq(ch->cq);
goto retry;
}
@@ -1894,7 +1894,7 @@ static void srpt_release_channel_work(struct work_struct *w)
srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_ring,
ch->sport->sdev, ch->rq_size,
- ch->rsp_size, DMA_TO_DEVICE);
+ ch->max_rsp_size, DMA_TO_DEVICE);
srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_recv_ring,
sdev, ch->rq_size,
@@ -2038,16 +2038,16 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
* depth to avoid that the initiator driver has to report QUEUE_FULL
* to the SCSI mid-layer.
*/
- ch->rq_size = min(SRPT_RQ_SIZE, sdev->device->attrs.max_qp_wr);
+ ch->rq_size = min(MAX_SRPT_RQ_SIZE, sdev->device->attrs.max_qp_wr);
spin_lock_init(&ch->spinlock);
ch->state = CH_CONNECTING;
INIT_LIST_HEAD(&ch->cmd_wait_list);
- ch->rsp_size = ch->sport->port_attrib.srp_max_rsp_size;
+ ch->max_rsp_size = ch->sport->port_attrib.srp_max_rsp_size;
ch->ioctx_ring = (struct srpt_send_ioctx **)
srpt_alloc_ioctx_ring(ch->sport->sdev, ch->rq_size,
sizeof(*ch->ioctx_ring[0]),
- ch->rsp_size, DMA_TO_DEVICE);
+ ch->max_rsp_size, DMA_TO_DEVICE);
if (!ch->ioctx_ring)
goto free_ch;
@@ -2175,7 +2175,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
free_ring:
srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_ring,
ch->sport->sdev, ch->rq_size,
- ch->rsp_size, DMA_TO_DEVICE);
+ ch->max_rsp_size, DMA_TO_DEVICE);
free_ch:
kfree(ch);
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 3cf917720b35..ac76825152f1 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -114,7 +114,7 @@ enum {
MIN_SRPT_SQ_SIZE = 16,
DEF_SRPT_SQ_SIZE = 4096,
- SRPT_RQ_SIZE = 128,
+ MAX_SRPT_RQ_SIZE = 128,
MIN_SRPT_SRQ_SIZE = 4,
DEFAULT_SRPT_SRQ_SIZE = 4095,
MAX_SRPT_SRQ_SIZE = 65535,
@@ -249,7 +249,7 @@ enum rdma_ch_state {
* @rcu: RCU head.
* @kref: kref for this channel.
* @rq_size: IB receive queue size.
- * @rsp_size: IB response message size in bytes.
+ * @max_rsp_size: Maximum size of an RSP response message in bytes.
* @sq_wr_avail: number of work requests available in the send queue.
* @sport: pointer to the information of the HCA port used by this
* channel.
@@ -281,7 +281,7 @@ struct srpt_rdma_ch {
struct rcu_head rcu;
struct kref kref;
int rq_size;
- u32 rsp_size;
+ u32 max_rsp_size;
atomic_t sq_wr_avail;
struct srpt_port *sport;
u8 i_port_id[16];
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 13/28] IB/srpt: Reduce the severity level of a log message
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (9 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 12/28] IB/srpt: Rename a local variable, a member variable and a constant Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 14/28] IB/srpt: Verify port numbers in srpt_event_handler() Bart Van Assche
` (15 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Since the SRQ event message is only useful for debugging purposes,
reduce its severity from "informational" to "debug".
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 6cff7a9b4cbb..7a36d0433706 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -173,7 +173,7 @@ static void srpt_event_handler(struct ib_event_handler *handler,
*/
static void srpt_srq_event(struct ib_event *event, void *ctx)
{
- pr_info("SRQ event %d\n", event->event);
+ pr_debug("SRQ event %d\n", event->event);
}
static const char *get_ch_state_name(enum rdma_ch_state s)
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 14/28] IB/srpt: Verify port numbers in srpt_event_handler()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (10 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 13/28] IB/srpt: Reduce the severity level of a log message Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 15/28] IB/srpt: Use the IPv6 format for GIDs in log messages Bart Van Assche
` (14 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Verify whether port numbers are in the expected range before using
these as an array index. Complain if a port number is out of range.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 22 ++++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 7a36d0433706..44096b93d783 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -132,6 +132,7 @@ static void srpt_event_handler(struct ib_event_handler *handler,
{
struct srpt_device *sdev;
struct srpt_port *sport;
+ u8 port_num;
sdev = ib_get_client_data(event->device, &srpt_client);
if (!sdev || sdev->device != event->device)
@@ -142,10 +143,15 @@ static void srpt_event_handler(struct ib_event_handler *handler,
switch (event->event) {
case IB_EVENT_PORT_ERR:
- if (event->element.port_num <= sdev->device->phys_port_cnt) {
- sport = &sdev->port[event->element.port_num - 1];
+ port_num = event->element.port_num - 1;
+ if (port_num < sdev->device->phys_port_cnt) {
+ sport = &sdev->port[port_num];
sport->lid = 0;
sport->sm_lid = 0;
+ } else {
+ WARN(true, "event %d: port_num %d out of range 1..%d\n",
+ event->event, port_num + 1,
+ sdev->device->phys_port_cnt);
}
break;
case IB_EVENT_PORT_ACTIVE:
@@ -155,15 +161,19 @@ static void srpt_event_handler(struct ib_event_handler *handler,
case IB_EVENT_CLIENT_REREGISTER:
case IB_EVENT_GID_CHANGE:
/* Refresh port data asynchronously. */
- if (event->element.port_num <= sdev->device->phys_port_cnt) {
- sport = &sdev->port[event->element.port_num - 1];
+ port_num = event->element.port_num - 1;
+ if (port_num < sdev->device->phys_port_cnt) {
+ sport = &sdev->port[port_num];
if (!sport->lid && !sport->sm_lid)
schedule_work(&sport->work);
+ } else {
+ WARN(true, "event %d: port_num %d out of range 1..%d\n",
+ event->event, port_num + 1,
+ sdev->device->phys_port_cnt);
}
break;
default:
- pr_err("received unrecognized IB event %d\n",
- event->event);
+ pr_err("received unrecognized IB event %d\n", event->event);
break;
}
}
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 15/28] IB/srpt: Use the IPv6 format for GIDs in log messages
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (11 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 14/28] IB/srpt: Verify port numbers in srpt_event_handler() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 16/28] IB/srpt: Reduce frequency of receive failure messages Bart Van Assche
` (13 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Make the ib_srpt driver use the IPv6 format for GIDs in log messages
to improve consistency of this driver with other RDMA kernel drivers.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 44096b93d783..ad7bdbbddf45 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1949,17 +1949,9 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
it_iu_len = be32_to_cpu(req->req_it_iu_len);
- pr_info("Received SRP_LOGIN_REQ with i_port_id 0x%llx:0x%llx,"
- " t_port_id 0x%llx:0x%llx and it_iu_len %d on port %d"
- " (guid=0x%llx:0x%llx)\n",
- be64_to_cpu(*(__be64 *)&req->initiator_port_id[0]),
- be64_to_cpu(*(__be64 *)&req->initiator_port_id[8]),
- be64_to_cpu(*(__be64 *)&req->target_port_id[0]),
- be64_to_cpu(*(__be64 *)&req->target_port_id[8]),
- it_iu_len,
- param->port,
- be64_to_cpu(*(__be64 *)&sdev->port[param->port - 1].gid.raw[0]),
- be64_to_cpu(*(__be64 *)&sdev->port[param->port - 1].gid.raw[8]));
+ pr_info("Received SRP_LOGIN_REQ with i_port_id %pI6, t_port_id %pI6 and it_iu_len %d on port %d (guid=%pI6)\n",
+ req->initiator_port_id, req->target_port_id, it_iu_len,
+ param->port, &sport->gid);
rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
rej = kzalloc(sizeof(*rej), GFP_KERNEL);
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 16/28] IB/srpt: Reduce frequency of receive failure messages
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (12 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 15/28] IB/srpt: Use the IPv6 format for GIDs in log messages Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 17/28] IB/srpt: Introduce srpt_format_guid() Bart Van Assche
` (12 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Disabling an SRP target port causes the state of all QPs associated
with a port to be changed into IB_QPS_ERR. Avoid that this causes
one error message per I/O context to be reported.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index ad7bdbbddf45..99d30c35f536 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1561,8 +1561,8 @@ static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc)
pr_err("req_lim = %d < 0\n", req_lim);
srpt_handle_new_iu(ch, ioctx, NULL);
} else {
- pr_info("receiving failed for ioctx %p with status %d\n",
- ioctx, wc->status);
+ pr_info_ratelimited("receiving failed for ioctx %p with status %d\n",
+ ioctx, wc->status);
}
}
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 17/28] IB/srpt: Introduce srpt_format_guid()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (13 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 16/28] IB/srpt: Reduce frequency of receive failure messages Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 18/28] IB/srpt: Inline srpt_get_cmd_state() Bart Van Assche
` (11 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Introduce a function for converting a GUID into an ASCII string. This
patch does not change any functionality.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 24 +++++++++++++-----------
1 file changed, 13 insertions(+), 11 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 99d30c35f536..d3cafeceea11 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -503,6 +503,15 @@ static void srpt_mad_recv_handler(struct ib_mad_agent *mad_agent,
ib_free_recv_mad(mad_wc);
}
+static int srpt_format_guid(char *buf, unsigned int size, const __be64 *guid)
+{
+ const __be16 *g = (const __be16 *)guid;
+
+ return snprintf(buf, size, "%04x:%04x:%04x:%04x",
+ be16_to_cpu(g[0]), be16_to_cpu(g[1]),
+ be16_to_cpu(g[2]), be16_to_cpu(g[3]));
+}
+
/*
* srpt_refresh_port() - Configure a HCA port.
*
@@ -517,7 +526,6 @@ static int srpt_refresh_port(struct srpt_port *sport)
struct ib_mad_reg_req reg_req;
struct ib_port_modify port_modify;
struct ib_port_attr port_attr;
- __be16 *guid;
int ret;
memset(&port_modify, 0, sizeof(port_modify));
@@ -541,11 +549,8 @@ static int srpt_refresh_port(struct srpt_port *sport)
goto err_query_port;
sport->port_guid_wwn.priv = sport;
- guid = (__be16 *)&sport->gid.global.interface_id;
- snprintf(sport->port_guid, sizeof(sport->port_guid),
- "%04x:%04x:%04x:%04x",
- be16_to_cpu(guid[0]), be16_to_cpu(guid[1]),
- be16_to_cpu(guid[2]), be16_to_cpu(guid[3]));
+ srpt_format_guid(sport->port_guid, sizeof(sport->port_guid),
+ &sport->gid.global.interface_id);
sport->port_gid_wwn.priv = sport;
snprintf(sport->port_gid, sizeof(sport->port_gid),
"0x%016llx%016llx",
@@ -1936,7 +1941,6 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
struct srp_login_rej *rej;
struct ib_cm_rep_param *rep_param;
struct srpt_rdma_ch *ch, *tmp_ch;
- __be16 *guid;
u32 it_iu_len;
int i, ret = 0;
@@ -2089,10 +2093,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto destroy_ib;
}
- guid = (__be16 *)¶m->primary_path->dgid.global.interface_id;
- snprintf(ch->ini_guid, sizeof(ch->ini_guid), "%04x:%04x:%04x:%04x",
- be16_to_cpu(guid[0]), be16_to_cpu(guid[1]),
- be16_to_cpu(guid[2]), be16_to_cpu(guid[3]));
+ srpt_format_guid(ch->ini_guid, sizeof(ch->ini_guid),
+ ¶m->primary_path->dgid.global.interface_id);
snprintf(ch->sess_name, sizeof(ch->sess_name), "0x%016llx%016llx",
be64_to_cpu(*(__be64 *)ch->i_port_id),
be64_to_cpu(*(__be64 *)(ch->i_port_id + 8)));
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 18/28] IB/srpt: Inline srpt_get_cmd_state()
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (14 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 17/28] IB/srpt: Introduce srpt_format_guid() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 19/28] IB/srpt: Micro-optimize I/O context state manipulation Bart Van Assche
` (10 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
It is not necessary to obtain ioctx->spinlock when reading the ioctx
state. Since after removal of this locking only a single line remains,
inline the srpt_get_cmd_state() function.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 22 +++-------------------
1 file changed, 3 insertions(+), 19 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index d3cafeceea11..9a1d3d1e614b 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -716,22 +716,6 @@ static void srpt_free_ioctx_ring(struct srpt_ioctx **ioctx_ring,
kfree(ioctx_ring);
}
-/*
- * srpt_get_cmd_state() - Get the state of a SCSI command.
- */
-static enum srpt_command_state srpt_get_cmd_state(struct srpt_send_ioctx *ioctx)
-{
- enum srpt_command_state state;
- unsigned long flags;
-
- BUG_ON(!ioctx);
-
- spin_lock_irqsave(&ioctx->spinlock, flags);
- state = ioctx->state;
- spin_unlock_irqrestore(&ioctx->spinlock, flags);
- return state;
-}
-
/*
* srpt_set_cmd_state() - Set the state of a SCSI command.
*
@@ -1248,7 +1232,7 @@ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc)
target_execute_cmd(&ioctx->cmd);
else
pr_err("%s[%d]: wrong state = %d\n", __func__,
- __LINE__, srpt_get_cmd_state(ioctx));
+ __LINE__, ioctx->state);
}
/**
@@ -2308,7 +2292,7 @@ static int srpt_write_pending_status(struct se_cmd *se_cmd)
struct srpt_send_ioctx *ioctx;
ioctx = container_of(se_cmd, struct srpt_send_ioctx, cmd);
- return srpt_get_cmd_state(ioctx) == SRPT_STATE_NEED_DATA;
+ return ioctx->state == SRPT_STATE_NEED_DATA;
}
/*
@@ -2880,7 +2864,7 @@ static int srpt_get_tcm_cmd_state(struct se_cmd *se_cmd)
struct srpt_send_ioctx *ioctx;
ioctx = container_of(se_cmd, struct srpt_send_ioctx, cmd);
- return srpt_get_cmd_state(ioctx);
+ return ioctx->state;
}
static int srpt_parse_guid(u64 *guid, const char *name)
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 19/28] IB/srpt: Micro-optimize I/O context state manipulation
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (15 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 18/28] IB/srpt: Inline srpt_get_cmd_state() Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 20/28] IB/srpt: Add P_Key support Bart Van Assche
` (9 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Since all I/O context state changes are already serialized, it is
not necessary to protect I/O context state changes with the I/O
context spinlock. Hence remove that spinlock.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 16 +---------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 2 --
2 files changed, 1 insertion(+), 17 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 9a1d3d1e614b..323d55e6daf1 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -726,15 +726,10 @@ static enum srpt_command_state srpt_set_cmd_state(struct srpt_send_ioctx *ioctx,
enum srpt_command_state new)
{
enum srpt_command_state previous;
- unsigned long flags;
-
- BUG_ON(!ioctx);
- spin_lock_irqsave(&ioctx->spinlock, flags);
previous = ioctx->state;
if (previous != SRPT_STATE_DONE)
ioctx->state = new;
- spin_unlock_irqrestore(&ioctx->spinlock, flags);
return previous;
}
@@ -749,17 +744,15 @@ static bool srpt_test_and_set_cmd_state(struct srpt_send_ioctx *ioctx,
enum srpt_command_state new)
{
enum srpt_command_state previous;
- unsigned long flags;
WARN_ON(!ioctx);
WARN_ON(old == SRPT_STATE_DONE);
WARN_ON(new == SRPT_STATE_NEW);
- spin_lock_irqsave(&ioctx->spinlock, flags);
previous = ioctx->state;
if (previous == old)
ioctx->state = new;
- spin_unlock_irqrestore(&ioctx->spinlock, flags);
+
return previous == old;
}
@@ -1120,7 +1113,6 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
return ioctx;
BUG_ON(ioctx->ch != ch);
- spin_lock_init(&ioctx->spinlock);
ioctx->state = SRPT_STATE_NEW;
ioctx->n_rdma = 0;
ioctx->n_rw_ctx = 0;
@@ -1142,7 +1134,6 @@ static struct srpt_send_ioctx *srpt_get_send_ioctx(struct srpt_rdma_ch *ch)
static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
{
enum srpt_command_state state;
- unsigned long flags;
BUG_ON(!ioctx);
@@ -1151,7 +1142,6 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
* the ib_srpt driver, change the state to the next state.
*/
- spin_lock_irqsave(&ioctx->spinlock, flags);
state = ioctx->state;
switch (state) {
case SRPT_STATE_NEED_DATA:
@@ -1166,7 +1156,6 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx)
__func__, state);
break;
}
- spin_unlock_irqrestore(&ioctx->spinlock, flags);
pr_debug("Aborting cmd with state %d -> %d and tag %lld\n", state,
ioctx->state, ioctx->cmd.tag);
@@ -2367,13 +2356,11 @@ static void srpt_queue_response(struct se_cmd *cmd)
struct ib_send_wr send_wr, *first_wr = &send_wr, *bad_wr;
struct ib_sge sge;
enum srpt_command_state state;
- unsigned long flags;
int resp_len, ret, i;
u8 srp_tm_status;
BUG_ON(!ch);
- spin_lock_irqsave(&ioctx->spinlock, flags);
state = ioctx->state;
switch (state) {
case SRPT_STATE_NEW:
@@ -2388,7 +2375,6 @@ static void srpt_queue_response(struct se_cmd *cmd)
ch, ioctx->ioctx.index, ioctx->state);
break;
}
- spin_unlock_irqrestore(&ioctx->spinlock, flags);
if (unlikely(WARN_ON_ONCE(state == SRPT_STATE_CMD_RSP_SENT)))
return;
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index ac76825152f1..0c68bee6ba1b 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -195,7 +195,6 @@ struct srpt_rw_ctx {
* @rw_ctxs: RDMA read/write contexts.
* @rdma_cqe: RDMA completion queue element.
* @free_list: Node in srpt_rdma_ch.free_list.
- * @spinlock: Protects 'state'.
* @state: I/O context state.
* @cmd: Target core command data structure.
* @sense_data: SCSI sense data.
@@ -213,7 +212,6 @@ struct srpt_send_ioctx {
struct ib_cqe rdma_cqe;
struct list_head free_list;
- spinlock_t spinlock;
enum srpt_command_state state;
struct se_cmd cmd;
u8 n_rdma;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 20/28] IB/srpt: Add P_Key support
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (16 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 19/28] IB/srpt: Micro-optimize I/O context state manipulation Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 21/28] IB/srpt: One target per port Bart Van Assche
` (8 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Process connection requests that use another P_Key than the default
correctly.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 14 +++++++++++---
drivers/infiniband/ulp/srpt/ib_srpt.h | 2 ++
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 323d55e6daf1..eeeb47739e9a 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -41,6 +41,7 @@
#include <linux/string.h>
#include <linux/delay.h>
#include <linux/atomic.h>
+#include <rdma/ib_cache.h>
#include <scsi/scsi_proto.h>
#include <scsi/scsi_tcq.h>
#include <target/target_core_base.h>
@@ -1009,7 +1010,12 @@ static int srpt_init_ch_qp(struct srpt_rdma_ch *ch, struct ib_qp *qp)
attr->qp_state = IB_QPS_INIT;
attr->qp_access_flags = IB_ACCESS_LOCAL_WRITE;
attr->port_num = ch->sport->port;
- attr->pkey_index = 0;
+
+ ret = ib_find_cached_pkey(ch->sport->sdev->device, ch->sport->port,
+ ch->pkey, &attr->pkey_index);
+ if (ret < 0)
+ pr_err("Translating pkey %#x failed (%d) - using index 0\n",
+ ch->pkey, ret);
ret = ib_modify_qp(qp, attr,
IB_QP_STATE | IB_QP_ACCESS_FLAGS | IB_QP_PORT |
@@ -1926,9 +1932,10 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
it_iu_len = be32_to_cpu(req->req_it_iu_len);
- pr_info("Received SRP_LOGIN_REQ with i_port_id %pI6, t_port_id %pI6 and it_iu_len %d on port %d (guid=%pI6)\n",
+ pr_info("Received SRP_LOGIN_REQ with i_port_id %pI6, t_port_id %pI6 and it_iu_len %d on port %d (guid=%pI6); pkey %#04x\n",
req->initiator_port_id, req->target_port_id, it_iu_len,
- param->port, &sport->gid);
+ param->port, &sport->gid,
+ be16_to_cpu(param->primary_path->pkey));
rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
rej = kzalloc(sizeof(*rej), GFP_KERNEL);
@@ -2005,6 +2012,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
init_rcu_head(&ch->rcu);
kref_init(&ch->kref);
+ ch->pkey = be16_to_cpu(param->primary_path->pkey);
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
memcpy(ch->i_port_id, req->initiator_port_id, 16);
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 0c68bee6ba1b..dd44e51fa167 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -266,6 +266,7 @@ enum rdma_ch_state {
* @cmd_wait_list: List of SCSI commands that arrived before the RTU event. This
* list contains struct srpt_ioctx elements and is protected
* against concurrent modification by the cm_id spinlock.
+ * @pkey: P_Key of the IB partition for this SRP channel.
* @sess: Session information associated with this SRP channel.
* @sess_name: Session name.
* @ini_guid: Initiator port GUID.
@@ -294,6 +295,7 @@ struct srpt_rdma_ch {
struct srpt_recv_ioctx **ioctx_recv_ring;
struct list_head list;
struct list_head cmd_wait_list;
+ uint16_t pkey;
struct se_session *sess;
u8 sess_name[36];
u8 ini_guid[24];
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 21/28] IB/srpt: One target per port
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (17 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 20/28] IB/srpt: Add P_Key support Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 22/28] IB/srpt: Rework multi-channel support Bart Van Assche
` (7 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
In multipathing setups where a target system is equipped with
dual-port HCAs it is useful to have one connection per target port
instead of one connection per target HCA. Hence move the connection
list (rch_list) from struct srpt_device into struct srpt_port.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 102 +++++++++++++++++-----------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 20 +++----
2 files changed, 62 insertions(+), 60 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index eeeb47739e9a..d8c695135024 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1781,13 +1781,13 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
return ret;
}
-static bool srpt_ch_closed(struct srpt_device *sdev, struct srpt_rdma_ch *ch)
+static bool srpt_ch_closed(struct srpt_port *sport, struct srpt_rdma_ch *ch)
{
struct srpt_rdma_ch *ch2;
bool res = true;
rcu_read_lock();
- list_for_each_entry(ch2, &sdev->rch_list, list) {
+ list_for_each_entry(ch2, &sport->rch_list, list) {
if (ch2 == ch) {
res = false;
goto done;
@@ -1806,33 +1806,32 @@ static bool srpt_ch_closed(struct srpt_device *sdev, struct srpt_rdma_ch *ch)
static bool srpt_disconnect_ch_sync(struct srpt_rdma_ch *ch)
__must_hold(&sdev->mutex)
{
- struct srpt_device *sdev = ch->sport->sdev;
+ struct srpt_port *sport = ch->sport;
int ret;
- lockdep_assert_held(&sdev->mutex);
+ lockdep_assert_held(&sport->mutex);
pr_debug("ch %s-%d state %d\n", ch->sess_name, ch->qp->qp_num,
ch->state);
ret = srpt_disconnect_ch(ch);
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sport->mutex);
- while (wait_event_timeout(sdev->ch_releaseQ, srpt_ch_closed(sdev, ch),
+ while (wait_event_timeout(sport->ch_releaseQ, srpt_ch_closed(sport, ch),
5 * HZ) == 0)
pr_info("%s(%s-%d state %d): still waiting ...\n", __func__,
ch->sess_name, ch->qp->qp_num, ch->state);
- mutex_lock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
return ret == 0;
}
static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
- __must_hold(&sdev->mutex)
+ __must_hold(&sport->mutex)
{
- struct srpt_device *sdev = sport->sdev;
struct srpt_rdma_ch *ch;
- lockdep_assert_held(&sdev->mutex);
+ lockdep_assert_held(&sport->mutex);
if (sport->enabled == enabled)
return;
@@ -1841,10 +1840,10 @@ static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
return;
again:
- list_for_each_entry(ch, &sdev->rch_list, list) {
+ list_for_each_entry(ch, &sport->rch_list, list) {
if (ch->sport == sport) {
pr_info("%s: closing channel %s-%d\n",
- sdev->device->name, ch->sess_name,
+ sport->sdev->device->name, ch->sess_name,
ch->qp->qp_num);
if (srpt_disconnect_ch_sync(ch))
goto again;
@@ -1864,6 +1863,7 @@ static void srpt_release_channel_work(struct work_struct *w)
{
struct srpt_rdma_ch *ch;
struct srpt_device *sdev;
+ struct srpt_port *sport;
struct se_session *se_sess;
ch = container_of(w, struct srpt_rdma_ch, release_work);
@@ -1894,11 +1894,12 @@ static void srpt_release_channel_work(struct work_struct *w)
sdev, ch->rq_size,
srp_max_req_size, DMA_FROM_DEVICE);
- mutex_lock(&sdev->mutex);
+ sport = ch->sport;
+ mutex_lock(&sport->mutex);
list_del_init(&ch->list);
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sport->mutex);
- wake_up(&sdev->ch_releaseQ);
+ wake_up(&sport->ch_releaseQ);
kref_put(&ch->kref, srpt_free_ch);
}
@@ -1920,6 +1921,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
struct srp_login_rej *rej;
struct ib_cm_rep_param *rep_param;
struct srpt_rdma_ch *ch, *tmp_ch;
+ char *ini_guid, i_port_id[36];
u32 it_iu_len;
int i, ret = 0;
@@ -1968,9 +1970,9 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
if ((req->req_flags & SRP_MTCH_ACTION) == SRP_MULTICHAN_SINGLE) {
rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_NO_CHAN;
- mutex_lock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
- list_for_each_entry_safe(ch, tmp_ch, &sdev->rch_list, list) {
+ list_for_each_entry_safe(ch, tmp_ch, &sport->rch_list, list) {
if (!memcmp(ch->i_port_id, req->initiator_port_id, 16)
&& !memcmp(ch->t_port_id, req->target_port_id, 16)
&& param->port == ch->sport->port
@@ -1985,7 +1987,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
}
}
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sport->mutex);
} else
rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_MAINTAINED;
@@ -2074,9 +2076,10 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto destroy_ib;
}
- srpt_format_guid(ch->ini_guid, sizeof(ch->ini_guid),
+ srpt_format_guid(ch->sess_name, sizeof(ch->sess_name),
¶m->primary_path->dgid.global.interface_id);
- snprintf(ch->sess_name, sizeof(ch->sess_name), "0x%016llx%016llx",
+ ini_guid = ch->sess_name;
+ snprintf(i_port_id, sizeof(i_port_id), "0x%016llx%016llx",
be64_to_cpu(*(__be64 *)ch->i_port_id),
be64_to_cpu(*(__be64 *)(ch->i_port_id + 8)));
@@ -2084,17 +2087,17 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
if (sport->port_guid_tpg.se_tpg_wwn)
ch->sess = target_alloc_session(&sport->port_guid_tpg, 0, 0,
- TARGET_PROT_NORMAL,
- ch->ini_guid, ch, NULL);
+ TARGET_PROT_NORMAL, ini_guid,
+ ch, NULL);
if (sport->port_gid_tpg.se_tpg_wwn && IS_ERR_OR_NULL(ch->sess))
ch->sess = target_alloc_session(&sport->port_gid_tpg, 0, 0,
- TARGET_PROT_NORMAL, ch->sess_name, ch,
+ TARGET_PROT_NORMAL, i_port_id, ch,
NULL);
/* Retry without leading "0x" */
if (sport->port_gid_tpg.se_tpg_wwn && IS_ERR_OR_NULL(ch->sess))
ch->sess = target_alloc_session(&sport->port_gid_tpg, 0, 0,
TARGET_PROT_NORMAL,
- ch->sess_name + 2, ch, NULL);
+ i_port_id + 2, ch, NULL);
if (IS_ERR_OR_NULL(ch->sess)) {
pr_info("Rejected login because no ACL has been configured yet for initiator %s.\n",
ch->sess_name);
@@ -2137,9 +2140,9 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto release_channel;
}
- mutex_lock(&sdev->mutex);
- list_add_tail_rcu(&ch->list, &sdev->rch_list);
- mutex_unlock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
+ list_add_tail_rcu(&ch->list, &sport->rch_list);
+ mutex_unlock(&sport->mutex);
goto out;
@@ -2487,23 +2490,20 @@ static void srpt_refresh_port_work(struct work_struct *work)
}
/*
- * srpt_release_sdev() - Free the channel resources associated with a target.
+ * srpt_release_sport() - Free the channel resources associated with a target.
*/
-static int srpt_release_sdev(struct srpt_device *sdev)
+static int srpt_release_sport(struct srpt_port *sport)
{
- int i, res;
+ int res;
WARN_ON_ONCE(irqs_disabled());
- BUG_ON(!sdev);
-
- mutex_lock(&sdev->mutex);
- for (i = 0; i < ARRAY_SIZE(sdev->port); i++)
- srpt_set_enabled(&sdev->port[i], false);
- mutex_unlock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
+ srpt_set_enabled(sport, false);
+ mutex_unlock(&sport->mutex);
- res = wait_event_interruptible(sdev->ch_releaseQ,
- list_empty_careful(&sdev->rch_list));
+ res = wait_event_interruptible(sport->ch_releaseQ,
+ list_empty_careful(&sport->rch_list));
if (res)
pr_err("%s: interrupted.\n", __func__);
@@ -2630,9 +2630,7 @@ static void srpt_add_one(struct ib_device *device)
goto err;
sdev->device = device;
- INIT_LIST_HEAD(&sdev->rch_list);
- init_waitqueue_head(&sdev->ch_releaseQ);
- mutex_init(&sdev->mutex);
+ mutex_init(&sdev->sdev_mutex);
sdev->pd = ib_alloc_pd(device, 0);
if (IS_ERR(sdev->pd))
@@ -2673,6 +2671,9 @@ static void srpt_add_one(struct ib_device *device)
for (i = 1; i <= sdev->device->phys_port_cnt; i++) {
sport = &sdev->port[i - 1];
+ INIT_LIST_HEAD(&sport->rch_list);
+ init_waitqueue_head(&sport->ch_releaseQ);
+ mutex_init(&sport->mutex);
sport->sdev = sdev;
sport->port = i;
sport->port_attrib.srp_max_rdma_size = DEFAULT_MAX_RDMA_SIZE;
@@ -2743,7 +2744,9 @@ static void srpt_remove_one(struct ib_device *device, void *client_data)
spin_lock(&srpt_dev_lock);
list_del(&sdev->list);
spin_unlock(&srpt_dev_lock);
- srpt_release_sdev(sdev);
+
+ for (i = 0; i < sdev->device->phys_port_cnt; i++)
+ srpt_release_sport(&sdev->port[i]);
srpt_free_srq(sdev);
@@ -2828,11 +2831,11 @@ static void srpt_release_cmd(struct se_cmd *se_cmd)
static void srpt_close_session(struct se_session *se_sess)
{
struct srpt_rdma_ch *ch = se_sess->fabric_sess_ptr;
- struct srpt_device *sdev = ch->sport->sdev;
+ struct srpt_port *sport = ch->sport;
- mutex_lock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
srpt_disconnect_ch_sync(ch);
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sport->mutex);
}
/*
@@ -3056,7 +3059,7 @@ static ssize_t srpt_tpg_attrib_use_srq_store(struct config_item *item,
if (val != !!val)
return -EINVAL;
- ret = mutex_lock_interruptible(&sdev->mutex);
+ ret = mutex_lock_interruptible(&sdev->sdev_mutex);
if (ret < 0)
return ret;
enabled = sport->enabled;
@@ -3065,7 +3068,7 @@ static ssize_t srpt_tpg_attrib_use_srq_store(struct config_item *item,
sport->port_attrib.use_srq = val;
srpt_use_srq(sdev, sport->port_attrib.use_srq);
srpt_set_enabled(sport, enabled);
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sdev->sdev_mutex);
return count;
}
@@ -3096,7 +3099,6 @@ static ssize_t srpt_tpg_enable_store(struct config_item *item,
{
struct se_portal_group *se_tpg = to_tpg(item);
struct srpt_port *sport = srpt_tpg_to_sport(se_tpg);
- struct srpt_device *sdev = sport->sdev;
unsigned long tmp;
int ret;
@@ -3111,9 +3113,9 @@ static ssize_t srpt_tpg_enable_store(struct config_item *item,
return -EINVAL;
}
- mutex_lock(&sdev->mutex);
+ mutex_lock(&sport->mutex);
srpt_set_enabled(sport, tmp);
- mutex_unlock(&sdev->mutex);
+ mutex_unlock(&sport->mutex);
return count;
}
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index dd44e51fa167..82b93e2a2efb 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -262,14 +262,13 @@ enum rdma_ch_state {
* @state: channel state. See also enum rdma_ch_state.
* @ioctx_ring: Send ring.
* @ioctx_recv_ring: Receive I/O context ring.
- * @list: Node for insertion in the srpt_device.rch_list list.
+ * @list: Node in srpt_port.rch_list.
* @cmd_wait_list: List of SCSI commands that arrived before the RTU event. This
* list contains struct srpt_ioctx elements and is protected
* against concurrent modification by the cm_id spinlock.
* @pkey: P_Key of the IB partition for this SRP channel.
* @sess: Session information associated with this SRP channel.
* @sess_name: Session name.
- * @ini_guid: Initiator port GUID.
* @release_work: Allows scheduling of srpt_release_channel().
*/
struct srpt_rdma_ch {
@@ -297,8 +296,7 @@ struct srpt_rdma_ch {
struct list_head cmd_wait_list;
uint16_t pkey;
struct se_session *sess;
- u8 sess_name[36];
- u8 ini_guid[24];
+ u8 sess_name[24];
struct work_struct release_work;
};
@@ -334,6 +332,9 @@ struct srpt_port_attrib {
* @port_gid_tpg: TPG associated with target port GID.
* @port_gid_wwn: WWN associated with target port GID.
* @port_attrib: Port attributes that can be accessed through configfs.
+ * @ch_releaseQ: Enables waiting for removal from rch_list.
+ * @mutex: Protects rch_list.
+ * @rch_list: Channel list. See also srpt_rdma_ch.list.
*/
struct srpt_port {
struct srpt_device *sdev;
@@ -351,6 +352,9 @@ struct srpt_port {
struct se_portal_group port_gid_tpg;
struct se_wwn port_gid_wwn;
struct srpt_port_attrib port_attrib;
+ wait_queue_head_t ch_releaseQ;
+ struct mutex mutex;
+ struct list_head rch_list;
};
/**
@@ -361,11 +365,9 @@ struct srpt_port {
* @srq: Per-HCA SRQ (shared receive queue).
* @cm_id: Connection identifier.
* @srq_size: SRQ size.
+ * @sdev_mutex: Serializes use_srq changes.
* @use_srq: Whether or not to use SRQ.
* @ioctx_ring: Per-HCA SRQ.
- * @rch_list: Per-device channel list -- see also srpt_rdma_ch.list.
- * @ch_releaseQ: Enables waiting for removal from rch_list.
- * @mutex: Protects rch_list.
* @port: Information about the ports owned by this HCA.
* @event_handler: Per-HCA asynchronous IB event handler.
* @list: Node in srpt_dev_list.
@@ -377,11 +379,9 @@ struct srpt_device {
struct ib_srq *srq;
struct ib_cm_id *cm_id;
int srq_size;
+ struct mutex sdev_mutex;
bool use_srq;
struct srpt_recv_ioctx **ioctx_ring;
- struct list_head rch_list;
- wait_queue_head_t ch_releaseQ;
- struct mutex mutex;
struct srpt_port port[2];
struct ib_event_handler event_handler;
struct list_head list;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 22/28] IB/srpt: Rework multi-channel support
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (18 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 21/28] IB/srpt: One target per port Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 23/28] IB/srpt: Fix login-related race conditions Bart Van Assche
` (6 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Store initiator and target port ID's once per nexus instead of in each
channel data structure. This change simplifies the duplicate connection
check in srpt_cm_req_recv().
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 185 ++++++++++++++++++++++++----------
drivers/infiniband/ulp/srpt/ib_srpt.h | 34 +++++--
2 files changed, 159 insertions(+), 60 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index d8c695135024..3a25eca81871 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1783,14 +1783,17 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
static bool srpt_ch_closed(struct srpt_port *sport, struct srpt_rdma_ch *ch)
{
+ struct srpt_nexus *nexus;
struct srpt_rdma_ch *ch2;
bool res = true;
rcu_read_lock();
- list_for_each_entry(ch2, &sport->rch_list, list) {
- if (ch2 == ch) {
- res = false;
- goto done;
+ list_for_each_entry(nexus, &sport->nexus_list, entry) {
+ list_for_each_entry(ch2, &nexus->ch_list, list) {
+ if (ch2 == ch) {
+ res = false;
+ goto done;
+ }
}
}
done:
@@ -1826,30 +1829,78 @@ static bool srpt_disconnect_ch_sync(struct srpt_rdma_ch *ch)
return ret == 0;
}
-static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
- __must_hold(&sport->mutex)
+static void __srpt_close_all_ch(struct srpt_port *sport)
{
+ struct srpt_nexus *nexus;
struct srpt_rdma_ch *ch;
lockdep_assert_held(&sport->mutex);
- if (sport->enabled == enabled)
- return;
- sport->enabled = enabled;
- if (sport->enabled)
- return;
+ list_for_each_entry(nexus, &sport->nexus_list, entry) {
+ list_for_each_entry(ch, &nexus->ch_list, list) {
+ if (srpt_disconnect_ch(ch) >= 0)
+ pr_info("Closing channel %s-%d because target %s_%d has been disabled\n",
+ ch->sess_name, ch->qp->qp_num,
+ sport->sdev->device->name, sport->port);
+ srpt_close_ch(ch);
+ }
+ }
+}
+
+/*
+ * Look up (i_port_id, t_port_id) in sport->nexus_list. Create an entry if
+ * it does not yet exist.
+ */
+static struct srpt_nexus *srpt_get_nexus(struct srpt_port *sport,
+ const u8 i_port_id[16],
+ const u8 t_port_id[16])
+{
+ struct srpt_nexus *nexus = NULL, *tmp_nexus = NULL, *n;
-again:
- list_for_each_entry(ch, &sport->rch_list, list) {
- if (ch->sport == sport) {
- pr_info("%s: closing channel %s-%d\n",
- sport->sdev->device->name, ch->sess_name,
- ch->qp->qp_num);
- if (srpt_disconnect_ch_sync(ch))
- goto again;
+ for (;;) {
+ mutex_lock(&sport->mutex);
+ list_for_each_entry(n, &sport->nexus_list, entry) {
+ if (memcmp(n->i_port_id, i_port_id, 16) == 0 &&
+ memcmp(n->t_port_id, t_port_id, 16) == 0) {
+ nexus = n;
+ break;
+ }
+ }
+ if (!nexus && tmp_nexus) {
+ list_add_tail_rcu(&tmp_nexus->entry,
+ &sport->nexus_list);
+ swap(nexus, tmp_nexus);
+ }
+ mutex_unlock(&sport->mutex);
+
+ if (nexus)
+ break;
+ tmp_nexus = kzalloc(sizeof(*nexus), GFP_KERNEL);
+ if (!tmp_nexus) {
+ nexus = ERR_PTR(-ENOMEM);
+ break;
}
+ init_rcu_head(&tmp_nexus->rcu);
+ INIT_LIST_HEAD(&tmp_nexus->ch_list);
+ memcpy(tmp_nexus->i_port_id, i_port_id, 16);
+ memcpy(tmp_nexus->t_port_id, t_port_id, 16);
}
+ kfree(tmp_nexus);
+
+ return nexus;
+}
+
+static void srpt_set_enabled(struct srpt_port *sport, bool enabled)
+ __must_hold(&sport->mutex)
+{
+ lockdep_assert_held(&sport->mutex);
+
+ if (sport->enabled == enabled)
+ return;
+ sport->enabled = enabled;
+ if (!enabled)
+ __srpt_close_all_ch(sport);
}
static void srpt_free_ch(struct kref *kref)
@@ -1916,11 +1967,12 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
{
struct srpt_device *sdev = cm_id->context;
struct srpt_port *sport = &sdev->port[param->port - 1];
+ struct srpt_nexus *nexus;
struct srp_login_req *req;
- struct srp_login_rsp *rsp;
- struct srp_login_rej *rej;
- struct ib_cm_rep_param *rep_param;
- struct srpt_rdma_ch *ch, *tmp_ch;
+ struct srp_login_rsp *rsp = NULL;
+ struct srp_login_rej *rej = NULL;
+ struct ib_cm_rep_param *rep_param = NULL;
+ struct srpt_rdma_ch *ch;
char *ini_guid, i_port_id[36];
u32 it_iu_len;
int i, ret = 0;
@@ -1939,6 +1991,13 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
param->port, &sport->gid,
be16_to_cpu(param->primary_path->pkey));
+ nexus = srpt_get_nexus(sport, req->initiator_port_id,
+ req->target_port_id);
+ if (IS_ERR(nexus)) {
+ ret = PTR_ERR(nexus);
+ goto out;
+ }
+
rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
rej = kzalloc(sizeof(*rej), GFP_KERNEL);
rep_param = kzalloc(sizeof(*rep_param), GFP_KERNEL);
@@ -1968,29 +2027,22 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
}
if ((req->req_flags & SRP_MTCH_ACTION) == SRP_MULTICHAN_SINGLE) {
+ struct srpt_rdma_ch *ch2;
+
rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_NO_CHAN;
mutex_lock(&sport->mutex);
-
- list_for_each_entry_safe(ch, tmp_ch, &sport->rch_list, list) {
- if (!memcmp(ch->i_port_id, req->initiator_port_id, 16)
- && !memcmp(ch->t_port_id, req->target_port_id, 16)
- && param->port == ch->sport->port
- && param->listen_id == ch->sport->sdev->cm_id
- && ch->cm_id) {
- if (srpt_disconnect_ch(ch) < 0)
- continue;
- pr_info("Relogin - closed existing channel %s\n",
- ch->sess_name);
- rsp->rsp_flags =
- SRP_LOGIN_RSP_MULTICHAN_TERMINATED;
- }
+ list_for_each_entry(ch2, &nexus->ch_list, list) {
+ if (srpt_disconnect_ch(ch2) < 0)
+ continue;
+ pr_info("Relogin - closed existing channel %s\n",
+ ch2->sess_name);
+ rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_TERMINATED;
}
-
mutex_unlock(&sport->mutex);
-
- } else
+ } else {
rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_MAINTAINED;
+ }
if (*(__be64 *)req->target_port_id != cpu_to_be64(srpt_service_guid)
|| *(__be64 *)(req->target_port_id + 8) !=
@@ -2015,10 +2067,9 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
init_rcu_head(&ch->rcu);
kref_init(&ch->kref);
ch->pkey = be16_to_cpu(param->primary_path->pkey);
+ ch->nexus = nexus;
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
- memcpy(ch->i_port_id, req->initiator_port_id, 16);
- memcpy(ch->t_port_id, req->target_port_id, 16);
ch->sport = &sdev->port[param->port - 1];
ch->cm_id = cm_id;
cm_id->context = ch;
@@ -2080,8 +2131,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
¶m->primary_path->dgid.global.interface_id);
ini_guid = ch->sess_name;
snprintf(i_port_id, sizeof(i_port_id), "0x%016llx%016llx",
- be64_to_cpu(*(__be64 *)ch->i_port_id),
- be64_to_cpu(*(__be64 *)(ch->i_port_id + 8)));
+ be64_to_cpu(*(__be64 *)nexus->i_port_id),
+ be64_to_cpu(*(__be64 *)(nexus->i_port_id + 8)));
pr_debug("registering session %s\n", ch->sess_name);
@@ -2141,7 +2192,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
}
mutex_lock(&sport->mutex);
- list_add_tail_rcu(&ch->list, &sport->rch_list);
+ list_add_tail_rcu(&ch->list, &nexus->ch_list);
mutex_unlock(&sport->mutex);
goto out;
@@ -2489,12 +2540,27 @@ static void srpt_refresh_port_work(struct work_struct *work)
srpt_refresh_port(sport);
}
+static bool srpt_ch_list_empty(struct srpt_port *sport)
+{
+ struct srpt_nexus *nexus;
+ bool res = true;
+
+ rcu_read_lock();
+ list_for_each_entry(nexus, &sport->nexus_list, entry)
+ if (!list_empty(&nexus->ch_list))
+ res = false;
+ rcu_read_unlock();
+
+ return res;
+}
+
/*
* srpt_release_sport() - Free the channel resources associated with a target.
*/
static int srpt_release_sport(struct srpt_port *sport)
{
- int res;
+ struct srpt_nexus *nexus, *next_n;
+ struct srpt_rdma_ch *ch;
WARN_ON_ONCE(irqs_disabled());
@@ -2502,10 +2568,27 @@ static int srpt_release_sport(struct srpt_port *sport)
srpt_set_enabled(sport, false);
mutex_unlock(&sport->mutex);
- res = wait_event_interruptible(sport->ch_releaseQ,
- list_empty_careful(&sport->rch_list));
- if (res)
- pr_err("%s: interrupted.\n", __func__);
+ while (wait_event_timeout(sport->ch_releaseQ,
+ srpt_ch_list_empty(sport), 5 * HZ) <= 0) {
+ pr_info("%s_%d: waiting for session unregistration ...\n",
+ sport->sdev->device->name, sport->port);
+ rcu_read_lock();
+ list_for_each_entry(nexus, &sport->nexus_list, entry) {
+ list_for_each_entry(ch, &nexus->ch_list, list) {
+ pr_info("%s-%d: state %s\n",
+ ch->sess_name, ch->qp->qp_num,
+ get_ch_state_name(ch->state));
+ }
+ }
+ rcu_read_unlock();
+ }
+
+ mutex_lock(&sport->mutex);
+ list_for_each_entry_safe(nexus, next_n, &sport->nexus_list, entry) {
+ list_del(&nexus->entry);
+ kfree_rcu(nexus, rcu);
+ }
+ mutex_unlock(&sport->mutex);
return 0;
}
@@ -2671,7 +2754,7 @@ static void srpt_add_one(struct ib_device *device)
for (i = 1; i <= sdev->device->phys_port_cnt; i++) {
sport = &sdev->port[i - 1];
- INIT_LIST_HEAD(&sport->rch_list);
+ INIT_LIST_HEAD(&sport->nexus_list);
init_waitqueue_head(&sport->ch_releaseQ);
mutex_init(&sport->mutex);
sport->sdev = sdev;
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 82b93e2a2efb..0ffb95cb117f 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -54,6 +54,8 @@
*/
#define SRP_SERVICE_NAME_PREFIX "SRP.T10:"
+struct srpt_nexus;
+
enum {
/*
* SRP IOControllerProfile attributes for SRP target ports that have
@@ -240,6 +242,7 @@ enum rdma_ch_state {
/**
* struct srpt_rdma_ch - RDMA channel.
+ * @nexus: I_T nexus this channel is associated with.
* @cm_id: IB CM ID associated with the channel.
* @qp: IB queue pair used for communicating over this channel.
* @cq: IB completion queue for this channel.
@@ -251,8 +254,6 @@ enum rdma_ch_state {
* @sq_wr_avail: number of work requests available in the send queue.
* @sport: pointer to the information of the HCA port used by this
* channel.
- * @i_port_id: 128-bit initiator port identifier copied from SRP_LOGIN_REQ.
- * @t_port_id: 128-bit target port identifier copied from SRP_LOGIN_REQ.
* @max_ti_iu_len: maximum target-to-initiator information unit length.
* @req_lim: request limit: maximum number of requests that may be sent
* by the initiator without having received a response.
@@ -262,7 +263,7 @@ enum rdma_ch_state {
* @state: channel state. See also enum rdma_ch_state.
* @ioctx_ring: Send ring.
* @ioctx_recv_ring: Receive I/O context ring.
- * @list: Node in srpt_port.rch_list.
+ * @list: Node in srpt_nexus.ch_list.
* @cmd_wait_list: List of SCSI commands that arrived before the RTU event. This
* list contains struct srpt_ioctx elements and is protected
* against concurrent modification by the cm_id spinlock.
@@ -272,6 +273,7 @@ enum rdma_ch_state {
* @release_work: Allows scheduling of srpt_release_channel().
*/
struct srpt_rdma_ch {
+ struct srpt_nexus *nexus;
struct ib_cm_id *cm_id;
struct ib_qp *qp;
struct ib_cq *cq;
@@ -282,8 +284,6 @@ struct srpt_rdma_ch {
u32 max_rsp_size;
atomic_t sq_wr_avail;
struct srpt_port *sport;
- u8 i_port_id[16];
- u8 t_port_id[16];
int max_ti_iu_len;
atomic_t req_lim;
atomic_t req_lim_delta;
@@ -300,6 +300,22 @@ struct srpt_rdma_ch {
struct work_struct release_work;
};
+/**
+ * struct srpt_nexus - I_T nexus
+ * @rcu: RCU head for this data structure.
+ * @entry: srpt_port.nexus_list list node.
+ * @ch_list: struct srpt_rdma_ch list. Protected by srpt_port.mutex.
+ * @i_port_id: 128-bit initiator port identifier copied from SRP_LOGIN_REQ.
+ * @t_port_id: 128-bit target port identifier copied from SRP_LOGIN_REQ.
+ */
+struct srpt_nexus {
+ struct rcu_head rcu;
+ struct list_head entry;
+ struct list_head ch_list;
+ u8 i_port_id[16];
+ u8 t_port_id[16];
+};
+
/**
* struct srpt_port_attib - Attributes for SRPT port
* @srp_max_rdma_size: Maximum size of SRP RDMA transfers for new connections.
@@ -332,9 +348,9 @@ struct srpt_port_attrib {
* @port_gid_tpg: TPG associated with target port GID.
* @port_gid_wwn: WWN associated with target port GID.
* @port_attrib: Port attributes that can be accessed through configfs.
- * @ch_releaseQ: Enables waiting for removal from rch_list.
- * @mutex: Protects rch_list.
- * @rch_list: Channel list. See also srpt_rdma_ch.list.
+ * @ch_releaseQ: Enables waiting for removal from nexus_list.
+ * @mutex: Protects nexus_list.
+ * @nexus_list: Nexus list. See also srpt_nexus.entry.
*/
struct srpt_port {
struct srpt_device *sdev;
@@ -354,7 +370,7 @@ struct srpt_port {
struct srpt_port_attrib port_attrib;
wait_queue_head_t ch_releaseQ;
struct mutex mutex;
- struct list_head rch_list;
+ struct list_head nexus_list;
};
/**
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 23/28] IB/srpt: Fix login-related race conditions
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (19 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 22/28] IB/srpt: Rework multi-channel support Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 24/28] IB/srpt: Prepare RDMA/CM support Bart Van Assche
` (5 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Make sure that sport->mutex is not released between the duplicate
channel check, adding a channel to the channel list and performing
the sport enabled check. Avoid that srpt_disconnect_ch() can be
invoked concurrently with the ib_send_cm_rep() call by
srpt_cm_req_recv().
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 195 +++++++++++++++++++---------------
1 file changed, 111 insertions(+), 84 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 3a25eca81871..b6544112a321 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1975,7 +1975,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
struct srpt_rdma_ch *ch;
char *ini_guid, i_port_id[36];
u32 it_iu_len;
- int i, ret = 0;
+ int i, ret;
WARN_ON_ONCE(irqs_disabled());
@@ -1998,69 +1998,43 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto out;
}
+ ret = -ENOMEM;
rsp = kzalloc(sizeof(*rsp), GFP_KERNEL);
rej = kzalloc(sizeof(*rej), GFP_KERNEL);
rep_param = kzalloc(sizeof(*rep_param), GFP_KERNEL);
-
- if (!rsp || !rej || !rep_param) {
- ret = -ENOMEM;
+ if (!rsp || !rej || !rep_param)
goto out;
- }
+ ret = -EINVAL;
if (it_iu_len > srp_max_req_size || it_iu_len < 64) {
rej->reason = cpu_to_be32(
- SRP_LOGIN_REJ_REQ_IT_IU_LENGTH_TOO_LARGE);
- ret = -EINVAL;
- pr_err("rejected SRP_LOGIN_REQ because its"
- " length (%d bytes) is out of range (%d .. %d)\n",
+ SRP_LOGIN_REJ_REQ_IT_IU_LENGTH_TOO_LARGE);
+ pr_err("rejected SRP_LOGIN_REQ because its length (%d bytes) is out of range (%d .. %d)\n",
it_iu_len, 64, srp_max_req_size);
goto reject;
}
if (!sport->enabled) {
- rej->reason = cpu_to_be32(
- SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
- ret = -EINVAL;
- pr_err("rejected SRP_LOGIN_REQ because the target port"
- " has not yet been enabled\n");
+ rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
+ pr_info("rejected SRP_LOGIN_REQ because target port %s_%d has not yet been enabled\n",
+ sport->sdev->device->name, param->port);
goto reject;
}
- if ((req->req_flags & SRP_MTCH_ACTION) == SRP_MULTICHAN_SINGLE) {
- struct srpt_rdma_ch *ch2;
-
- rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_NO_CHAN;
-
- mutex_lock(&sport->mutex);
- list_for_each_entry(ch2, &nexus->ch_list, list) {
- if (srpt_disconnect_ch(ch2) < 0)
- continue;
- pr_info("Relogin - closed existing channel %s\n",
- ch2->sess_name);
- rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_TERMINATED;
- }
- mutex_unlock(&sport->mutex);
- } else {
- rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_MAINTAINED;
- }
-
if (*(__be64 *)req->target_port_id != cpu_to_be64(srpt_service_guid)
|| *(__be64 *)(req->target_port_id + 8) !=
cpu_to_be64(srpt_service_guid)) {
rej->reason = cpu_to_be32(
- SRP_LOGIN_REJ_UNABLE_ASSOCIATE_CHANNEL);
- ret = -ENOMEM;
- pr_err("rejected SRP_LOGIN_REQ because it"
- " has an invalid target port identifier.\n");
+ SRP_LOGIN_REJ_UNABLE_ASSOCIATE_CHANNEL);
+ pr_err("rejected SRP_LOGIN_REQ because it has an invalid target port identifier.\n");
goto reject;
}
+ ret = -ENOMEM;
ch = kzalloc(sizeof(*ch), GFP_KERNEL);
if (!ch) {
- rej->reason = cpu_to_be32(
- SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
- pr_err("rejected SRP_LOGIN_REQ because no memory.\n");
- ret = -ENOMEM;
+ rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
+ pr_err("rejected SRP_LOGIN_REQ because out of memory.\n");
goto reject;
}
@@ -2088,8 +2062,11 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
srpt_alloc_ioctx_ring(ch->sport->sdev, ch->rq_size,
sizeof(*ch->ioctx_ring[0]),
ch->max_rsp_size, DMA_TO_DEVICE);
- if (!ch->ioctx_ring)
+ if (!ch->ioctx_ring) {
+ pr_err("rejected SRP_LOGIN_REQ because creating a new QP SQ ring failed.\n");
+ rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
goto free_ch;
+ }
INIT_LIST_HEAD(&ch->free_list);
for (i = 0; i < ch->rq_size; i++) {
@@ -2111,20 +2088,10 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
}
ret = srpt_create_ch_ib(ch);
- if (ret) {
- rej->reason = cpu_to_be32(
- SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
- pr_err("rejected SRP_LOGIN_REQ because creating"
- " a new RDMA channel failed.\n");
- goto free_recv_ring;
- }
-
- ret = srpt_ch_qp_rtr(ch, ch->qp);
if (ret) {
rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
- pr_err("rejected SRP_LOGIN_REQ because enabling"
- " RTR failed (error code = %d)\n", ret);
- goto destroy_ib;
+ pr_err("rejected SRP_LOGIN_REQ because creating a new RDMA channel failed.\n");
+ goto free_recv_ring;
}
srpt_format_guid(ch->sess_name, sizeof(ch->sess_name),
@@ -2150,11 +2117,51 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
TARGET_PROT_NORMAL,
i_port_id + 2, ch, NULL);
if (IS_ERR_OR_NULL(ch->sess)) {
- pr_info("Rejected login because no ACL has been configured yet for initiator %s.\n",
- ch->sess_name);
- rej->reason = cpu_to_be32((PTR_ERR(ch->sess) == -ENOMEM) ?
+ ret = PTR_ERR(ch->sess);
+ pr_info("Rejected login for initiator %s: ret = %d.\n",
+ ch->sess_name, ret);
+ rej->reason = cpu_to_be32(ret == -ENOMEM ?
SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES :
SRP_LOGIN_REJ_CHANNEL_LIMIT_REACHED);
+ goto reject;
+ }
+
+ mutex_lock(&sport->mutex);
+
+ if ((req->req_flags & SRP_MTCH_ACTION) == SRP_MULTICHAN_SINGLE) {
+ struct srpt_rdma_ch *ch2;
+
+ rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_NO_CHAN;
+
+ list_for_each_entry(ch2, &nexus->ch_list, list) {
+ if (srpt_disconnect_ch(ch2) < 0)
+ continue;
+ pr_info("Relogin - closed existing channel %s\n",
+ ch2->sess_name);
+ rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_TERMINATED;
+ }
+ } else {
+ rsp->rsp_flags = SRP_LOGIN_RSP_MULTICHAN_MAINTAINED;
+ }
+
+ list_add_tail_rcu(&ch->list, &nexus->ch_list);
+
+ if (!sport->enabled) {
+ rej->reason = cpu_to_be32(
+ SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
+ pr_info("rejected SRP_LOGIN_REQ because target %s_%d is not enabled\n",
+ sdev->device->name, param->port);
+ mutex_unlock(&sport->mutex);
+ goto reject;
+ }
+
+ mutex_unlock(&sport->mutex);
+
+ ret = srpt_ch_qp_rtr(ch, ch->qp);
+ if (ret) {
+ rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
+ pr_err("rejected SRP_LOGIN_REQ because enabling RTR failed (error code = %d)\n",
+ ret);
goto destroy_ib;
}
@@ -2167,8 +2174,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
rsp->max_it_iu_len = req->req_it_iu_len;
rsp->max_ti_iu_len = req->req_it_iu_len;
ch->max_ti_iu_len = it_iu_len;
- rsp->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT
- | SRP_BUF_FORMAT_INDIRECT);
+ rsp->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
+ SRP_BUF_FORMAT_INDIRECT);
rsp->req_lim_delta = cpu_to_be32(ch->rq_size);
atomic_set(&ch->req_lim, ch->rq_size);
atomic_set(&ch->req_lim_delta, 0);
@@ -2184,24 +2191,30 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
rep_param->responder_resources = 4;
rep_param->initiator_depth = 4;
- ret = ib_send_cm_rep(cm_id, rep_param);
- if (ret) {
- pr_err("sending SRP_LOGIN_REQ response failed"
- " (error code = %d)\n", ret);
- goto release_channel;
- }
-
+ /*
+ * Hold the sport mutex while accepting a connection to avoid that
+ * srpt_disconnect_ch() is invoked concurrently with this code.
+ */
mutex_lock(&sport->mutex);
- list_add_tail_rcu(&ch->list, &nexus->ch_list);
+ if (sport->enabled && ch->state == CH_CONNECTING) {
+ ret = ib_send_cm_rep(cm_id, rep_param);
+ } else {
+ ret = -EINVAL;
+ }
mutex_unlock(&sport->mutex);
- goto out;
+ switch (ret) {
+ case 0:
+ break;
+ case -EINVAL:
+ goto reject;
+ default:
+ rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
+ pr_err("sending SRP_LOGIN_REQ response failed (error code = %d)\n", ret);
+ goto reject;
+ }
-release_channel:
- srpt_disconnect_ch(ch);
- transport_deregister_session_configfs(ch->sess);
- transport_deregister_session(ch->sess);
- ch->sess = NULL;
+ goto out;
destroy_ib:
srpt_destroy_ch_ib(ch);
@@ -2216,13 +2229,18 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
ch->sport->sdev, ch->rq_size,
ch->max_rsp_size, DMA_TO_DEVICE);
free_ch:
+ cm_id->context = NULL;
kfree(ch);
+ ch = NULL;
+
+ WARN_ON_ONCE(ret == 0);
reject:
+ pr_info("Rejecting login with reason %#x\n", be32_to_cpu(rej->reason));
rej->opcode = SRP_LOGIN_REJ;
rej->tag = req->tag;
- rej->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT
- | SRP_BUF_FORMAT_INDIRECT);
+ rej->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
+ SRP_BUF_FORMAT_INDIRECT);
ib_send_cm_rej(cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0,
(void *)rej, sizeof(*rej));
@@ -2264,17 +2282,26 @@ static void srpt_cm_rtu_recv(struct srpt_rdma_ch *ch)
{
int ret;
- if (srpt_set_ch_state(ch, CH_LIVE)) {
- ret = srpt_ch_qp_rts(ch, ch->qp);
-
- if (ret == 0) {
- /* Trigger wait list processing. */
- ret = srpt_zerolength_write(ch);
- WARN_ONCE(ret < 0, "%d\n", ret);
- } else {
- srpt_close_ch(ch);
- }
+ ret = srpt_ch_qp_rts(ch, ch->qp);
+ if (ret < 0) {
+ pr_err("%s-%d: QP transition to RTS failed\n", ch->sess_name,
+ ch->qp->qp_num);
+ srpt_close_ch(ch);
+ return;
}
+
+ /* Trigger wait list processing. */
+ ret = srpt_zerolength_write(ch);
+ WARN_ONCE(ret < 0, "%d\n", ret);
+
+ /*
+ * Note: calling srpt_close_ch() if the transition to the LIVE state
+ * fails is not necessary since that means that that function has
+ * already been invoked from another thread.
+ */
+ if (!srpt_set_ch_state(ch, CH_LIVE))
+ pr_err("%s-%d: channel transition to LIVE state failed\n",
+ ch->sess_name, ch->qp->qp_num);
}
/*
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 24/28] IB/srpt: Prepare RDMA/CM support
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (20 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 23/28] IB/srpt: Fix login-related race conditions Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 25/28] IB/srpt: Move the code for parsing struct ib_cm_req_event_param Bart Van Assche
` (4 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Introduce a union in struct srpt_rdma_ch for member variables that
depend on the type of connection manager. Avoid that error messages
report the IB/CM ID.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 38 +++++++++++++++++------------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 8 ++++++--
2 files changed, 25 insertions(+), 21 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index b6544112a321..8c67cbff93b6 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -209,12 +209,12 @@ static const char *get_ch_state_name(enum rdma_ch_state s)
*/
static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch)
{
- pr_debug("QP event %d on cm_id=%p sess_name=%s state=%d\n",
- event->event, ch->cm_id, ch->sess_name, ch->state);
+ pr_debug("QP event %d on ch=%p sess_name=%s state=%d\n",
+ event->event, ch, ch->sess_name, ch->state);
switch (event->event) {
case IB_EVENT_COMM_EST:
- ib_cm_notify(ch->cm_id, event->event);
+ ib_cm_notify(ch->ib_cm.cm_id, event->event);
break;
case IB_EVENT_QP_LAST_WQE_REACHED:
pr_debug("%s-%d, state %s: received Last WQE event.\n",
@@ -1043,7 +1043,7 @@ static int srpt_ch_qp_rtr(struct srpt_rdma_ch *ch, struct ib_qp *qp)
int ret;
qp_attr.qp_state = IB_QPS_RTR;
- ret = ib_cm_init_qp_attr(ch->cm_id, &qp_attr, &attr_mask);
+ ret = ib_cm_init_qp_attr(ch->ib_cm.cm_id, &qp_attr, &attr_mask);
if (ret)
goto out;
@@ -1073,7 +1073,7 @@ static int srpt_ch_qp_rts(struct srpt_rdma_ch *ch, struct ib_qp *qp)
int ret;
qp_attr.qp_state = IB_QPS_RTS;
- ret = ib_cm_init_qp_attr(ch->cm_id, &qp_attr, &attr_mask);
+ ret = ib_cm_init_qp_attr(ch->ib_cm.cm_id, &qp_attr, &attr_mask);
if (ret)
goto out;
@@ -1443,9 +1443,9 @@ static void srpt_handle_tsk_mgmt(struct srpt_rdma_ch *ch,
srp_tsk = recv_ioctx->ioctx.buf;
cmd = &send_ioctx->cmd;
- pr_debug("recv tsk_mgmt fn %d for task_tag %lld and cmd tag %lld"
- " cm_id %p sess %p\n", srp_tsk->tsk_mgmt_func,
- srp_tsk->task_tag, srp_tsk->tag, ch->cm_id, ch->sess);
+ pr_debug("recv tsk_mgmt fn %d for task_tag %lld and cmd tag %lld ch %p sess %p\n",
+ srp_tsk->tsk_mgmt_func, srp_tsk->task_tag, srp_tsk->tag, ch,
+ ch->sess);
srpt_set_cmd_state(send_ioctx, SRPT_STATE_MGMT);
send_ioctx->cmd.tag = srp_tsk->tag;
@@ -1683,9 +1683,9 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
atomic_set(&ch->sq_wr_avail, qp_init->cap.max_send_wr);
- pr_debug("%s: max_cqe= %d max_sge= %d sq_size = %d cm_id= %p\n",
+ pr_debug("%s: max_cqe= %d max_sge= %d sq_size = %d ch= %p\n",
__func__, ch->cq->cqe, qp_init->cap.max_send_sge,
- qp_init->cap.max_send_wr, ch->cm_id);
+ qp_init->cap.max_send_wr, ch);
ret = srpt_init_ch_qp(ch, ch->qp);
if (ret)
@@ -1771,9 +1771,9 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
if (!srpt_set_ch_state(ch, CH_DISCONNECTING))
return -ENOTCONN;
- ret = ib_send_cm_dreq(ch->cm_id, NULL, 0);
+ ret = ib_send_cm_dreq(ch->ib_cm.cm_id, NULL, 0);
if (ret < 0)
- ret = ib_send_cm_drep(ch->cm_id, NULL, 0);
+ ret = ib_send_cm_drep(ch->ib_cm.cm_id, NULL, 0);
if (ret < 0 && srpt_close_ch(ch))
ret = 0;
@@ -1933,7 +1933,7 @@ static void srpt_release_channel_work(struct work_struct *w)
transport_deregister_session(se_sess);
ch->sess = NULL;
- ib_destroy_cm_id(ch->cm_id);
+ ib_destroy_cm_id(ch->ib_cm.cm_id);
srpt_destroy_ch_ib(ch);
@@ -1973,7 +1973,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
struct srp_login_rej *rej = NULL;
struct ib_cm_rep_param *rep_param = NULL;
struct srpt_rdma_ch *ch;
- char *ini_guid, i_port_id[36];
+ char *ini_id, i_port_id[36];
u32 it_iu_len;
int i, ret;
@@ -2045,7 +2045,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
ch->sport = &sdev->port[param->port - 1];
- ch->cm_id = cm_id;
+ ch->ib_cm.cm_id = cm_id;
cm_id->context = ch;
/*
* ch->rq_size should be at least as large as the initiator queue
@@ -2096,7 +2096,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
srpt_format_guid(ch->sess_name, sizeof(ch->sess_name),
¶m->primary_path->dgid.global.interface_id);
- ini_guid = ch->sess_name;
+ ini_id = ch->sess_name;
snprintf(i_port_id, sizeof(i_port_id), "0x%016llx%016llx",
be64_to_cpu(*(__be64 *)nexus->i_port_id),
be64_to_cpu(*(__be64 *)(nexus->i_port_id + 8)));
@@ -2105,7 +2105,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
if (sport->port_guid_tpg.se_tpg_wwn)
ch->sess = target_alloc_session(&sport->port_guid_tpg, 0, 0,
- TARGET_PROT_NORMAL, ini_guid,
+ TARGET_PROT_NORMAL, ini_id,
ch, NULL);
if (sport->port_gid_tpg.se_tpg_wwn && IS_ERR_OR_NULL(ch->sess))
ch->sess = target_alloc_session(&sport->port_gid_tpg, 0, 0,
@@ -2165,8 +2165,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto destroy_ib;
}
- pr_debug("Establish connection sess=%p name=%s cm_id=%p\n", ch->sess,
- ch->sess_name, ch->cm_id);
+ pr_debug("Establish connection sess=%p name=%s ch=%p\n", ch->sess,
+ ch->sess_name, ch);
/* create srp_login_response */
rsp->opcode = SRP_LOGIN_RSP;
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 0ffb95cb117f..2e40e6574dca 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -243,8 +243,8 @@ enum rdma_ch_state {
/**
* struct srpt_rdma_ch - RDMA channel.
* @nexus: I_T nexus this channel is associated with.
- * @cm_id: IB CM ID associated with the channel.
* @qp: IB queue pair used for communicating over this channel.
+ * @cm_id: IB CM ID associated with the channel.
* @cq: IB completion queue for this channel.
* @zw_cqe: Zero-length write CQE.
* @rcu: RCU head.
@@ -274,8 +274,12 @@ enum rdma_ch_state {
*/
struct srpt_rdma_ch {
struct srpt_nexus *nexus;
- struct ib_cm_id *cm_id;
struct ib_qp *qp;
+ union {
+ struct {
+ struct ib_cm_id *cm_id;
+ } ib_cm;
+ };
struct ib_cq *cq;
struct ib_cqe zw_cqe;
struct rcu_head rcu;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 25/28] IB/srpt: Move the code for parsing struct ib_cm_req_event_param
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (21 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 24/28] IB/srpt: Prepare RDMA/CM support Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 26/28] IB/srpt: Fix a race condition related to wait list processing Bart Van Assche
` (3 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
This patch does not change any functionality but makes srpt_cm_req_recv()
independent of the IB/CM and hence simplifies the patch that introduces
RDMA/CM support.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 48 +++++++++++++++++++++--------------
1 file changed, 29 insertions(+), 19 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 8c67cbff93b6..e222794ec1aa 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1962,34 +1962,31 @@ static void srpt_release_channel_work(struct work_struct *w)
* functions returns zero. Otherwise the caller remains the owner of cm_id.
*/
static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
- struct ib_cm_req_event_param *param,
- void *private_data)
+ u8 port_num, __be16 pkey,
+ const struct srp_login_req *req,
+ const char *sess_name, const char *ini_id)
{
struct srpt_device *sdev = cm_id->context;
- struct srpt_port *sport = &sdev->port[param->port - 1];
+ struct srpt_port *sport = &sdev->port[port_num - 1];
struct srpt_nexus *nexus;
- struct srp_login_req *req;
struct srp_login_rsp *rsp = NULL;
struct srp_login_rej *rej = NULL;
struct ib_cm_rep_param *rep_param = NULL;
struct srpt_rdma_ch *ch;
- char *ini_id, i_port_id[36];
+ char i_port_id[36];
u32 it_iu_len;
int i, ret;
WARN_ON_ONCE(irqs_disabled());
- if (WARN_ON(!sdev || !private_data))
+ if (WARN_ON(!sdev || !req))
return -EINVAL;
- req = (struct srp_login_req *)private_data;
-
it_iu_len = be32_to_cpu(req->req_it_iu_len);
pr_info("Received SRP_LOGIN_REQ with i_port_id %pI6, t_port_id %pI6 and it_iu_len %d on port %d (guid=%pI6); pkey %#04x\n",
req->initiator_port_id, req->target_port_id, it_iu_len,
- param->port, &sport->gid,
- be16_to_cpu(param->primary_path->pkey));
+ port_num, &sport->gid, be16_to_cpu(pkey));
nexus = srpt_get_nexus(sport, req->initiator_port_id,
req->target_port_id);
@@ -2017,7 +2014,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
if (!sport->enabled) {
rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
pr_info("rejected SRP_LOGIN_REQ because target port %s_%d has not yet been enabled\n",
- sport->sdev->device->name, param->port);
+ sport->sdev->device->name, port_num);
goto reject;
}
@@ -2040,11 +2037,11 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
init_rcu_head(&ch->rcu);
kref_init(&ch->kref);
- ch->pkey = be16_to_cpu(param->primary_path->pkey);
+ ch->pkey = be16_to_cpu(pkey);
ch->nexus = nexus;
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
- ch->sport = &sdev->port[param->port - 1];
+ ch->sport = sport;
ch->ib_cm.cm_id = cm_id;
cm_id->context = ch;
/*
@@ -2094,9 +2091,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
goto free_recv_ring;
}
- srpt_format_guid(ch->sess_name, sizeof(ch->sess_name),
- ¶m->primary_path->dgid.global.interface_id);
- ini_id = ch->sess_name;
+ strlcpy(ch->sess_name, ini_id, sizeof(ch->sess_name));
snprintf(i_port_id, sizeof(i_port_id), "0x%016llx%016llx",
be64_to_cpu(*(__be64 *)nexus->i_port_id),
be64_to_cpu(*(__be64 *)(nexus->i_port_id + 8)));
@@ -2150,7 +2145,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
rej->reason = cpu_to_be32(
SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
pr_info("rejected SRP_LOGIN_REQ because target %s_%d is not enabled\n",
- sdev->device->name, param->port);
+ sdev->device->name, port_num);
mutex_unlock(&sport->mutex);
goto reject;
}
@@ -2253,6 +2248,21 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
return ret;
}
+static int srpt_ib_cm_req_recv(struct ib_cm_id *cm_id,
+ struct ib_cm_req_event_param *param,
+ void *private_data)
+{
+ char sess_name[40], ini_id[40];
+
+ snprintf(sess_name, sizeof(sess_name), "%pI6",
+ ¶m->primary_path->dgid);
+ srpt_format_guid(ini_id, sizeof(ini_id),
+ ¶m->primary_path->dgid.global.interface_id);
+
+ return srpt_cm_req_recv(cm_id, param->port, param->primary_path->pkey,
+ private_data, sess_name, ini_id);
+}
+
static void srpt_cm_rej_recv(struct srpt_rdma_ch *ch,
enum ib_cm_rej_reason reason,
const u8 *private_data,
@@ -2322,8 +2332,8 @@ static int srpt_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
ret = 0;
switch (event->event) {
case IB_CM_REQ_RECEIVED:
- ret = srpt_cm_req_recv(cm_id, &event->param.req_rcvd,
- event->private_data);
+ ret = srpt_ib_cm_req_recv(cm_id, &event->param.req_rcvd,
+ event->private_data);
break;
case IB_CM_REJ_RECEIVED:
srpt_cm_rej_recv(ch, event->param.rej_rcvd.reason,
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 26/28] IB/srpt: Fix a race condition related to wait list processing
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (22 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 25/28] IB/srpt: Move the code for parsing struct ib_cm_req_event_param Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 27/28] IB/srpt: Avoid that wait list processing triggers command reordering Bart Van Assche
` (2 subsequent siblings)
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Wait list processing only occurs if the channel state >= CH_LIVE. Hence
set the channel state to CH_LIVE before triggering wait list processing
asynchronously.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index e222794ec1aa..a509e0d0df0b 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -2300,18 +2300,20 @@ static void srpt_cm_rtu_recv(struct srpt_rdma_ch *ch)
return;
}
- /* Trigger wait list processing. */
- ret = srpt_zerolength_write(ch);
- WARN_ONCE(ret < 0, "%d\n", ret);
-
/*
* Note: calling srpt_close_ch() if the transition to the LIVE state
* fails is not necessary since that means that that function has
* already been invoked from another thread.
*/
- if (!srpt_set_ch_state(ch, CH_LIVE))
+ if (!srpt_set_ch_state(ch, CH_LIVE)) {
pr_err("%s-%d: channel transition to LIVE state failed\n",
ch->sess_name, ch->qp->qp_num);
+ return;
+ }
+
+ /* Trigger wait list processing. */
+ ret = srpt_zerolength_write(ch);
+ WARN_ONCE(ret < 0, "%d\n", ret);
}
/*
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 27/28] IB/srpt: Avoid that wait list processing triggers command reordering
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (23 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 26/28] IB/srpt: Fix a race condition related to wait list processing Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:39 ` [PATCH 28/28] IB/srpt: Add RDMA/CM support Bart Van Assche
2018-01-03 21:51 ` [PATCH 00/28] IB/srp and IB/srpt patches Jason Gunthorpe
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
If a receive I/O context is removed from the wait list and
srpt_handle_new_iu() fails to allocate a send I/O context then
re-adding the receive I/O context to the wait list can cause
reordering. Avoid this by only removing a receive I/O context
from the wait list after allocating a send I/O context succeeded.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 82 ++++++++++++++++++++---------------
drivers/infiniband/ulp/srpt/ib_srpt.h | 2 +
2 files changed, 49 insertions(+), 35 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index a509e0d0df0b..00e959fe4bac 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1467,39 +1467,39 @@ static void srpt_handle_tsk_mgmt(struct srpt_rdma_ch *ch,
* srpt_handle_new_iu() - Process a newly received information unit.
* @ch: RDMA channel through which the information unit has been received.
* @recv_ioctx: Receive I/O context associated with the information unit.
- * @send_ioctx: Send I/O context associated with the information unit.
*/
-static void srpt_handle_new_iu(struct srpt_rdma_ch *ch,
- struct srpt_recv_ioctx *recv_ioctx,
- struct srpt_send_ioctx *send_ioctx)
+static bool
+srpt_handle_new_iu(struct srpt_rdma_ch *ch, struct srpt_recv_ioctx *recv_ioctx)
{
+ struct srpt_send_ioctx *send_ioctx = NULL;
struct srp_cmd *srp_cmd;
+ bool res = false;
+ u8 opcode;
BUG_ON(!ch);
BUG_ON(!recv_ioctx);
+ if (unlikely(ch->state == CH_CONNECTING))
+ goto push;
+
ib_dma_sync_single_for_cpu(ch->sport->sdev->device,
recv_ioctx->ioctx.dma, srp_max_req_size,
DMA_FROM_DEVICE);
- if (unlikely(ch->state == CH_CONNECTING))
- goto out_wait;
-
- if (unlikely(ch->state != CH_LIVE))
- return;
-
srp_cmd = recv_ioctx->ioctx.buf;
- if (srp_cmd->opcode == SRP_CMD || srp_cmd->opcode == SRP_TSK_MGMT) {
- if (!send_ioctx) {
- if (!list_empty(&ch->cmd_wait_list))
- goto out_wait;
- send_ioctx = srpt_get_send_ioctx(ch);
- }
+ opcode = srp_cmd->opcode;
+ if (opcode == SRP_CMD || opcode == SRP_TSK_MGMT) {
+ send_ioctx = srpt_get_send_ioctx(ch);
if (unlikely(!send_ioctx))
- goto out_wait;
+ goto push;
+ }
+
+ if (!list_empty(&recv_ioctx->wait_list)) {
+ WARN_ON_ONCE(!ch->processing_wait_list);
+ list_del_init(&recv_ioctx->wait_list);
}
- switch (srp_cmd->opcode) {
+ switch (opcode) {
case SRP_CMD:
srpt_handle_cmd(ch, recv_ioctx, send_ioctx);
break;
@@ -1519,16 +1519,22 @@ static void srpt_handle_new_iu(struct srpt_rdma_ch *ch,
pr_err("Received SRP_RSP\n");
break;
default:
- pr_err("received IU with unknown opcode 0x%x\n",
- srp_cmd->opcode);
+ pr_err("received IU with unknown opcode 0x%x\n", opcode);
break;
}
srpt_post_recv(ch->sport->sdev, ch, recv_ioctx);
- return;
+ res = true;
-out_wait:
- list_add_tail(&recv_ioctx->wait_list, &ch->cmd_wait_list);
+out:
+ return res;
+
+push:
+ if (list_empty(&recv_ioctx->wait_list)) {
+ WARN_ON_ONCE(ch->processing_wait_list);
+ list_add_tail(&recv_ioctx->wait_list, &ch->cmd_wait_list);
+ }
+ goto out;
}
static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc)
@@ -1543,7 +1549,7 @@ static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc)
req_lim = atomic_dec_return(&ch->req_lim);
if (unlikely(req_lim < 0))
pr_err("req_lim = %d < 0\n", req_lim);
- srpt_handle_new_iu(ch, ioctx, NULL);
+ srpt_handle_new_iu(ch, ioctx);
} else {
pr_info_ratelimited("receiving failed for ioctx %p with status %d\n",
ioctx, wc->status);
@@ -1557,19 +1563,21 @@ static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc)
*/
static void srpt_process_wait_list(struct srpt_rdma_ch *ch)
{
- struct srpt_send_ioctx *ioctx;
+ struct srpt_recv_ioctx *recv_ioctx, *tmp;
- while (!list_empty(&ch->cmd_wait_list) &&
- ch->state >= CH_LIVE &&
- (ioctx = srpt_get_send_ioctx(ch)) != NULL) {
- struct srpt_recv_ioctx *recv_ioctx;
+ WARN_ON_ONCE(ch->state == CH_CONNECTING);
- recv_ioctx = list_first_entry(&ch->cmd_wait_list,
- struct srpt_recv_ioctx,
- wait_list);
- list_del(&recv_ioctx->wait_list);
- srpt_handle_new_iu(ch, recv_ioctx, ioctx);
+ if (list_empty(&ch->cmd_wait_list))
+ return;
+
+ WARN_ON_ONCE(ch->processing_wait_list);
+ ch->processing_wait_list = true;
+ list_for_each_entry_safe(recv_ioctx, tmp, &ch->cmd_wait_list,
+ wait_list) {
+ if (!srpt_handle_new_iu(ch, recv_ioctx))
+ break;
}
+ ch->processing_wait_list = false;
}
/*
@@ -2082,6 +2090,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
goto free_ring;
}
+ for (i = 0; i < ch->rq_size; i++)
+ INIT_LIST_HEAD(&ch->ioctx_recv_ring[i]->wait_list);
}
ret = srpt_create_ch_ib(ch);
@@ -2714,8 +2724,10 @@ static int srpt_alloc_srq(struct srpt_device *sdev)
sdev->use_srq = true;
sdev->srq = srq;
- for (i = 0; i < sdev->srq_size; ++i)
+ for (i = 0; i < sdev->srq_size; ++i) {
+ INIT_LIST_HEAD(&sdev->ioctx_ring[i]->wait_list);
srpt_post_recv(sdev, NULL, sdev->ioctx_ring[i]);
+ }
return 0;
}
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 2e40e6574dca..5a7eee561c2f 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -261,6 +261,7 @@ enum rdma_ch_state {
* @spinlock: Protects free_list and state.
* @free_list: Head of list with free send I/O contexts.
* @state: channel state. See also enum rdma_ch_state.
+ * @processing_wait_list: Whether or not cmd_wait_list is being processed.
* @ioctx_ring: Send ring.
* @ioctx_recv_ring: Receive I/O context ring.
* @list: Node in srpt_nexus.ch_list.
@@ -299,6 +300,7 @@ struct srpt_rdma_ch {
struct list_head list;
struct list_head cmd_wait_list;
uint16_t pkey;
+ bool processing_wait_list;
struct se_session *sess;
u8 sess_name[24];
struct work_struct release_work;
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* [PATCH 28/28] IB/srpt: Add RDMA/CM support
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (24 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 27/28] IB/srpt: Avoid that wait list processing triggers command reordering Bart Van Assche
@ 2018-01-03 21:39 ` Bart Van Assche
2018-01-03 21:51 ` [PATCH 00/28] IB/srp and IB/srpt patches Jason Gunthorpe
26 siblings, 0 replies; 35+ messages in thread
From: Bart Van Assche @ 2018-01-03 21:39 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Bart Van Assche
Add a kernel module parameter for configuring the port on which the
ib_srpt driver listens for incoming RDMA/CM connections. The default
value for this kernel module parameter is 0 which means "do not listen
for incoming RDMA/CM connections". Add RDMA/CM support to all code
that handles connection state changes. Modify srpt_init_nodeacl()
such that ACLs can be configured for IPv4 addresses.
Signed-off-by: Bart Van Assche <bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
---
drivers/infiniband/ulp/srpt/ib_srpt.c | 323 +++++++++++++++++++++++++++-------
drivers/infiniband/ulp/srpt/ib_srpt.h | 6 +
2 files changed, 270 insertions(+), 59 deletions(-)
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 00e959fe4bac..65b9ff0a5378 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -41,6 +41,7 @@
#include <linux/string.h>
#include <linux/delay.h>
#include <linux/atomic.h>
+#include <linux/inet.h>
#include <rdma/ib_cache.h>
#include <scsi/scsi_proto.h>
#include <scsi/scsi_tcq.h>
@@ -71,6 +72,10 @@ static u64 srpt_service_guid;
static DEFINE_SPINLOCK(srpt_dev_lock); /* Protects srpt_dev_list. */
static LIST_HEAD(srpt_dev_list); /* List of srpt_device structures. */
+static u16 rdma_cm_port;
+module_param(rdma_cm_port, short, 0444);
+MODULE_PARM_DESC(rdma_cm_port, "Port number RDMA/CM will bind to.");
+
static unsigned srp_max_req_size = DEFAULT_MAX_REQ_SIZE;
module_param(srp_max_req_size, int, 0444);
MODULE_PARM_DESC(srp_max_req_size,
@@ -92,6 +97,7 @@ MODULE_PARM_DESC(srpt_service_guid,
" instead of using the node_guid of the first HCA.");
static struct ib_client srpt_client;
+static struct rdma_cm_id *rdma_cm_id;
static void srpt_release_cmd(struct se_cmd *se_cmd);
static void srpt_free_ch(struct kref *kref);
static int srpt_queue_status(struct se_cmd *cmd);
@@ -214,7 +220,10 @@ static void srpt_qp_event(struct ib_event *event, struct srpt_rdma_ch *ch)
switch (event->event) {
case IB_EVENT_COMM_EST:
- ib_cm_notify(ch->ib_cm.cm_id, event->event);
+ if (ch->using_rdma_cm)
+ rdma_notify(ch->rdma_cm.cm_id, event->event);
+ else
+ ib_cm_notify(ch->ib_cm.cm_id, event->event);
break;
case IB_EVENT_QP_LAST_WQE_REACHED:
pr_debug("%s-%d, state %s: received Last WQE event.\n",
@@ -1003,6 +1012,8 @@ static int srpt_init_ch_qp(struct srpt_rdma_ch *ch, struct ib_qp *qp)
struct ib_qp_attr *attr;
int ret;
+ WARN_ON_ONCE(ch->using_rdma_cm);
+
attr = kzalloc(sizeof(*attr), GFP_KERNEL);
if (!attr)
return -ENOMEM;
@@ -1042,6 +1053,8 @@ static int srpt_ch_qp_rtr(struct srpt_rdma_ch *ch, struct ib_qp *qp)
int attr_mask;
int ret;
+ WARN_ON_ONCE(ch->using_rdma_cm);
+
qp_attr.qp_state = IB_QPS_RTR;
ret = ib_cm_init_qp_attr(ch->ib_cm.cm_id, &qp_attr, &attr_mask);
if (ret)
@@ -1675,18 +1688,31 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
qp_init->cap.max_recv_sge = qp_init->cap.max_send_sge;
}
- ch->qp = ib_create_qp(sdev->pd, qp_init);
- if (IS_ERR(ch->qp)) {
- ret = PTR_ERR(ch->qp);
- if (ret == -ENOMEM) {
- sq_size /= 2;
- if (sq_size >= MIN_SRPT_SQ_SIZE) {
- ib_destroy_cq(ch->cq);
- goto retry;
- }
+ if (ch->using_rdma_cm) {
+ ret = rdma_create_qp(ch->rdma_cm.cm_id, sdev->pd, qp_init);
+ ch->qp = ch->rdma_cm.cm_id->qp;
+ } else {
+ ch->qp = ib_create_qp(sdev->pd, qp_init);
+ if (!IS_ERR(ch->qp)) {
+ ret = srpt_init_ch_qp(ch, ch->qp);
+ if (ret)
+ ib_destroy_qp(ch->qp);
+ } else {
+ ret = PTR_ERR(ch->qp);
+ }
+ }
+ if (ret) {
+ bool retry = sq_size > MIN_SRPT_SQ_SIZE;
+
+ pr_err("failed to create queue pair with sq_size = %d (%d)%s\n",
+ sq_size, ret, retry ? " - retrying" : "");
+ if (retry) {
+ ib_free_cq(ch->cq);
+ sq_size = max(sq_size / 2, MIN_SRPT_SQ_SIZE);
+ goto retry;
+ } else {
+ goto err_destroy_cq;
}
- pr_err("failed to create_qp ret= %d\n", ret);
- goto err_destroy_cq;
}
atomic_set(&ch->sq_wr_avail, qp_init->cap.max_send_wr);
@@ -1695,10 +1721,6 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
__func__, ch->cq->cqe, qp_init->cap.max_send_sge,
qp_init->cap.max_send_wr, ch);
- ret = srpt_init_ch_qp(ch, ch->qp);
- if (ret)
- goto err_destroy_qp;
-
if (!sdev->use_srq)
for (i = 0; i < ch->rq_size; i++)
srpt_post_recv(sdev, ch, ch->ioctx_recv_ring[i]);
@@ -1707,9 +1729,8 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch)
kfree(qp_init);
return ret;
-err_destroy_qp:
- ib_destroy_qp(ch->qp);
err_destroy_cq:
+ ch->qp = NULL;
ib_free_cq(ch->cq);
goto out;
}
@@ -1779,9 +1800,13 @@ static int srpt_disconnect_ch(struct srpt_rdma_ch *ch)
if (!srpt_set_ch_state(ch, CH_DISCONNECTING))
return -ENOTCONN;
- ret = ib_send_cm_dreq(ch->ib_cm.cm_id, NULL, 0);
- if (ret < 0)
- ret = ib_send_cm_drep(ch->ib_cm.cm_id, NULL, 0);
+ if (ch->using_rdma_cm) {
+ ret = rdma_disconnect(ch->rdma_cm.cm_id);
+ } else {
+ ret = ib_send_cm_dreq(ch->ib_cm.cm_id, NULL, 0);
+ if (ret < 0)
+ ret = ib_send_cm_drep(ch->ib_cm.cm_id, NULL, 0);
+ }
if (ret < 0 && srpt_close_ch(ch))
ret = 0;
@@ -1941,7 +1966,10 @@ static void srpt_release_channel_work(struct work_struct *w)
transport_deregister_session(se_sess);
ch->sess = NULL;
- ib_destroy_cm_id(ch->ib_cm.cm_id);
+ if (ch->using_rdma_cm)
+ rdma_destroy_id(ch->rdma_cm.cm_id);
+ else
+ ib_destroy_cm_id(ch->ib_cm.cm_id);
srpt_destroy_ch_ib(ch);
@@ -1964,22 +1992,26 @@ static void srpt_release_channel_work(struct work_struct *w)
}
/*
- * srpt_cm_req_recv() - Process the event IB_CM_REQ_RECEIVED.
+ * srpt_cm_req_recv() - Process the IB_CM_REQ_RECEIVED event.
*
* Ownership of the cm_id is transferred to the target session if this
- * functions returns zero. Otherwise the caller remains the owner of cm_id.
+ * function returns zero. Otherwise the caller remains the owner of cm_id.
*/
-static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
+static int srpt_cm_req_recv(struct srpt_device *const sdev,
+ struct ib_cm_id *ib_cm_id,
+ struct rdma_cm_id *rdma_cm_id,
u8 port_num, __be16 pkey,
const struct srp_login_req *req,
const char *sess_name, const char *ini_id)
{
- struct srpt_device *sdev = cm_id->context;
struct srpt_port *sport = &sdev->port[port_num - 1];
struct srpt_nexus *nexus;
struct srp_login_rsp *rsp = NULL;
struct srp_login_rej *rej = NULL;
- struct ib_cm_rep_param *rep_param = NULL;
+ union {
+ struct rdma_conn_param rdma_cm;
+ struct ib_cm_rep_param ib_cm;
+ } *rep_param = NULL;
struct srpt_rdma_ch *ch;
char i_port_id[36];
u32 it_iu_len;
@@ -2050,8 +2082,14 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
ch->zw_cqe.done = srpt_zerolength_write_done;
INIT_WORK(&ch->release_work, srpt_release_channel_work);
ch->sport = sport;
- ch->ib_cm.cm_id = cm_id;
- cm_id->context = ch;
+ if (ib_cm_id) {
+ ch->ib_cm.cm_id = ib_cm_id;
+ ib_cm_id->context = ch;
+ } else {
+ ch->using_rdma_cm = true;
+ ch->rdma_cm.cm_id = rdma_cm_id;
+ rdma_cm_id->context = ch;
+ }
/*
* ch->rq_size should be at least as large as the initiator queue
* depth to avoid that the initiator driver has to report QUEUE_FULL
@@ -2162,7 +2200,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
mutex_unlock(&sport->mutex);
- ret = srpt_ch_qp_rtr(ch, ch->qp);
+ ret = ch->using_rdma_cm ? 0 : srpt_ch_qp_rtr(ch, ch->qp);
if (ret) {
rej->reason = cpu_to_be32(SRP_LOGIN_REJ_INSUFFICIENT_RESOURCES);
pr_err("rejected SRP_LOGIN_REQ because enabling RTR failed (error code = %d)\n",
@@ -2186,15 +2224,24 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
atomic_set(&ch->req_lim_delta, 0);
/* create cm reply */
- rep_param->qp_num = ch->qp->qp_num;
- rep_param->private_data = (void *)rsp;
- rep_param->private_data_len = sizeof(*rsp);
- rep_param->rnr_retry_count = 7;
- rep_param->flow_control = 1;
- rep_param->failover_accepted = 0;
- rep_param->srq = 1;
- rep_param->responder_resources = 4;
- rep_param->initiator_depth = 4;
+ if (ch->using_rdma_cm) {
+ rep_param->rdma_cm.private_data = (void *)rsp;
+ rep_param->rdma_cm.private_data_len = sizeof(*rsp);
+ rep_param->rdma_cm.rnr_retry_count = 7;
+ rep_param->rdma_cm.flow_control = 1;
+ rep_param->rdma_cm.responder_resources = 4;
+ rep_param->rdma_cm.initiator_depth = 4;
+ } else {
+ rep_param->ib_cm.qp_num = ch->qp->qp_num;
+ rep_param->ib_cm.private_data = (void *)rsp;
+ rep_param->ib_cm.private_data_len = sizeof(*rsp);
+ rep_param->ib_cm.rnr_retry_count = 7;
+ rep_param->ib_cm.flow_control = 1;
+ rep_param->ib_cm.failover_accepted = 0;
+ rep_param->ib_cm.srq = 1;
+ rep_param->ib_cm.responder_resources = 4;
+ rep_param->ib_cm.initiator_depth = 4;
+ }
/*
* Hold the sport mutex while accepting a connection to avoid that
@@ -2202,7 +2249,10 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
*/
mutex_lock(&sport->mutex);
if (sport->enabled && ch->state == CH_CONNECTING) {
- ret = ib_send_cm_rep(cm_id, rep_param);
+ if (ch->using_rdma_cm)
+ ret = rdma_accept(rdma_cm_id, &rep_param->rdma_cm);
+ else
+ ret = ib_send_cm_rep(ib_cm_id, &rep_param->ib_cm);
} else {
ret = -EINVAL;
}
@@ -2234,7 +2284,8 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
ch->sport->sdev, ch->rq_size,
ch->max_rsp_size, DMA_TO_DEVICE);
free_ch:
- cm_id->context = NULL;
+ if (ib_cm_id)
+ ib_cm_id->context = NULL;
kfree(ch);
ch = NULL;
@@ -2247,8 +2298,11 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id,
rej->buf_fmt = cpu_to_be16(SRP_BUF_FORMAT_DIRECT |
SRP_BUF_FORMAT_INDIRECT);
- ib_send_cm_rej(cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0,
- (void *)rej, sizeof(*rej));
+ if (rdma_cm_id)
+ rdma_reject(rdma_cm_id, rej, sizeof(*rej));
+ else
+ ib_send_cm_rej(ib_cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0,
+ rej, sizeof(*rej));
out:
kfree(rep_param);
@@ -2269,10 +2323,62 @@ static int srpt_ib_cm_req_recv(struct ib_cm_id *cm_id,
srpt_format_guid(ini_id, sizeof(ini_id),
¶m->primary_path->dgid.global.interface_id);
- return srpt_cm_req_recv(cm_id, param->port, param->primary_path->pkey,
+ return srpt_cm_req_recv(cm_id->context, cm_id, NULL, param->port,
+ param->primary_path->pkey,
private_data, sess_name, ini_id);
}
+static const char *inet_ntop(const void *sa, char *dst, unsigned int size)
+{
+ switch (((struct sockaddr *)sa)->sa_family) {
+ case AF_INET:
+ snprintf(dst, size, "%pI4",
+ &((struct sockaddr_in *)sa)->sin_addr);
+ break;
+ case AF_INET6:
+ snprintf(dst, size, "%pI6",
+ &((struct sockaddr_in6 *)sa)->sin6_addr);
+ break;
+ default:
+ snprintf(dst, size, "???");
+ break;
+ }
+ return dst;
+}
+
+static int srpt_rdma_cm_req_recv(struct rdma_cm_id *cm_id,
+ struct rdma_cm_event *event)
+{
+ struct srpt_device *sdev;
+ struct srp_login_req req;
+ const struct srp_login_req_rdma *req_rdma;
+ char sess_name[40];
+
+ sdev = ib_get_client_data(cm_id->device, &srpt_client);
+ if (!sdev)
+ return -ECONNREFUSED;
+
+ if (event->param.conn.private_data_len < sizeof(*req_rdma))
+ return -EINVAL;
+
+ /* Transform srp_login_req_rdma into srp_login_req. */
+ req_rdma = event->param.conn.private_data;
+ memset(&req, 0, sizeof(req));
+ req.opcode = req_rdma->opcode;
+ req.tag = req_rdma->tag;
+ req.req_it_iu_len = req_rdma->req_it_iu_len;
+ req.req_buf_fmt = req_rdma->req_buf_fmt;
+ req.req_flags = req_rdma->req_flags;
+ memcpy(req.initiator_port_id, req_rdma->initiator_port_id, 16);
+ memcpy(req.target_port_id, req_rdma->target_port_id, 16);
+
+ inet_ntop(&cm_id->route.addr.src_addr, sess_name, sizeof(sess_name));
+
+ return srpt_cm_req_recv(sdev, NULL, cm_id, cm_id->port_num,
+ cm_id->route.path_rec->pkey, &req, sess_name,
+ sess_name);
+}
+
static void srpt_cm_rej_recv(struct srpt_rdma_ch *ch,
enum ib_cm_rej_reason reason,
const u8 *private_data,
@@ -2293,16 +2399,16 @@ static void srpt_cm_rej_recv(struct srpt_rdma_ch *ch,
}
/*
- * srpt_cm_rtu_recv() - Process an IB_CM_RTU_RECEIVED or USER_ESTABLISHED event.
+ * srpt_cm_rtu_recv() - Process RTU event.
*
- * An IB_CM_RTU_RECEIVED message indicates that the connection is established
- * and that the recipient may begin transmitting (RTU = ready to use).
+ * An RTU (ready to use) message indicates that the connection has been
+ * established and that the recipient may begin transmitting.
*/
static void srpt_cm_rtu_recv(struct srpt_rdma_ch *ch)
{
int ret;
- ret = srpt_ch_qp_rts(ch, ch->qp);
+ ret = ch->using_rdma_cm ? 0 : srpt_ch_qp_rts(ch, ch->qp);
if (ret < 0) {
pr_err("%s-%d: QP transition to RTS failed\n", ch->sess_name,
ch->qp->qp_num);
@@ -2387,6 +2493,49 @@ static int srpt_cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
return ret;
}
+static int srpt_rdma_cm_handler(struct rdma_cm_id *cm_id,
+ struct rdma_cm_event *event)
+{
+ struct srpt_rdma_ch *ch = cm_id->context;
+ int ret = 0;
+
+ switch (event->event) {
+ case RDMA_CM_EVENT_CONNECT_REQUEST:
+ ret = srpt_rdma_cm_req_recv(cm_id, event);
+ break;
+ case RDMA_CM_EVENT_REJECTED:
+ srpt_cm_rej_recv(ch, event->status,
+ event->param.conn.private_data,
+ event->param.conn.private_data_len);
+ break;
+ case RDMA_CM_EVENT_ESTABLISHED:
+ srpt_cm_rtu_recv(ch);
+ break;
+ case RDMA_CM_EVENT_DISCONNECTED:
+ if (ch->state < CH_DISCONNECTING)
+ srpt_disconnect_ch(ch);
+ else
+ srpt_close_ch(ch);
+ break;
+ case RDMA_CM_EVENT_TIMEWAIT_EXIT:
+ srpt_close_ch(ch);
+ break;
+ case RDMA_CM_EVENT_UNREACHABLE:
+ pr_info("Received CM REP error for ch %s-%d.\n", ch->sess_name,
+ ch->qp->qp_num);
+ break;
+ case RDMA_CM_EVENT_DEVICE_REMOVAL:
+ case RDMA_CM_EVENT_ADDR_CHANGE:
+ break;
+ default:
+ pr_err("received unrecognized RDMA CM event %d\n",
+ event->event);
+ break;
+ }
+
+ return ret;
+}
+
static int srpt_write_pending_status(struct se_cmd *se_cmd)
{
struct srpt_send_ioctx *ioctx;
@@ -2755,7 +2904,7 @@ static void srpt_add_one(struct ib_device *device)
{
struct srpt_device *sdev;
struct srpt_port *sport;
- int i;
+ int i, ret;
pr_debug("device = %p\n", device);
@@ -2779,9 +2928,15 @@ static void srpt_add_one(struct ib_device *device)
if (!srpt_service_guid)
srpt_service_guid = be64_to_cpu(device->node_guid);
- sdev->cm_id = ib_create_cm_id(device, srpt_cm_handler, sdev);
- if (IS_ERR(sdev->cm_id))
- goto err_ring;
+ if (rdma_port_get_link_layer(device, 1) == IB_LINK_LAYER_INFINIBAND)
+ sdev->cm_id = ib_create_cm_id(device, srpt_cm_handler, sdev);
+ if (IS_ERR(sdev->cm_id)) {
+ pr_info("ib_create_cm_id() failed: %ld\n",
+ PTR_ERR(sdev->cm_id));
+ sdev->cm_id = NULL;
+ if (!rdma_cm_id)
+ goto err_ring;
+ }
/* print out target login information */
pr_debug("Target login info: id_ext=%016llx,ioc_guid=%016llx,"
@@ -2794,8 +2949,13 @@ static void srpt_add_one(struct ib_device *device)
* in the system as service_id; therefore, the target_id will change
* if this HCA is gone bad and replaced by different HCA
*/
- if (ib_cm_listen(sdev->cm_id, cpu_to_be64(srpt_service_guid), 0))
+ ret = sdev->cm_id ?
+ ib_cm_listen(sdev->cm_id, cpu_to_be64(srpt_service_guid), 0) : 0;
+ if (ret < 0) {
+ pr_err("ib_cm_listen() failed: %d (cm_id state = %d)\n", ret,
+ sdev->cm_id->state);
goto err_cm;
+ }
INIT_IB_EVENT_HANDLER(&sdev->event_handler, sdev->device,
srpt_event_handler);
@@ -2835,7 +2995,8 @@ static void srpt_add_one(struct ib_device *device)
err_event:
ib_unregister_event_handler(&sdev->event_handler);
err_cm:
- ib_destroy_cm_id(sdev->cm_id);
+ if (sdev->cm_id)
+ ib_destroy_cm_id(sdev->cm_id);
err_ring:
srpt_free_srq(sdev);
ib_dealloc_pd(sdev->pd);
@@ -2868,7 +3029,10 @@ static void srpt_remove_one(struct ib_device *device, void *client_data)
for (i = 0; i < sdev->device->phys_port_cnt; i++)
cancel_work_sync(&sdev->port[i].work);
- ib_destroy_cm_id(sdev->cm_id);
+ if (sdev->cm_id)
+ ib_destroy_cm_id(sdev->cm_id);
+
+ ib_set_client_data(device, &srpt_client, NULL);
/*
* Unregistering a target must happen after destroying sdev->cm_id
@@ -3033,18 +3197,25 @@ static int srpt_parse_i_port_id(u8 i_port_id[16], const char *name)
leading_zero_bytes = 16 - count;
memset(i_port_id, 0, leading_zero_bytes);
ret = hex2bin(i_port_id + leading_zero_bytes, p, count);
- if (ret < 0)
- pr_debug("hex2bin failed for srpt_parse_i_port_id: %d\n", ret);
+
out:
return ret;
}
/*
- * configfs callback function invoked for
- * mkdir /sys/kernel/config/target/$driver/$port/$tpg/acls/$i_port_id
+ * configfs callback function invoked for mkdir
+ * /sys/kernel/config/target/$driver/$port/$tpg/acls/$i_port_id i_port_id must
+ * be an initiator port GUID, GID or IP address. See also the
+ * target_alloc_session() calls in this driver. Examples of valid initiator
+ * port IDs:
+ * 0x0000000000000000505400fffe4a0b7b
+ * 0000000000000000505400fffe4a0b7b
+ * 5054:00ff:fe4a:0b7b
+ * 192.168.122.76
*/
static int srpt_init_nodeacl(struct se_node_acl *se_nacl, const char *name)
{
+ struct sockaddr_storage sa;
u64 guid;
u8 i_port_id[16];
int ret;
@@ -3052,6 +3223,9 @@ static int srpt_init_nodeacl(struct se_node_acl *se_nacl, const char *name)
ret = srpt_parse_guid(&guid, name);
if (ret < 0)
ret = srpt_parse_i_port_id(i_port_id, name);
+ if (ret < 0)
+ ret = inet_pton_with_scope(&init_net, AF_UNSPEC, name, NULL,
+ &sa);
if (ret < 0)
pr_err("invalid initiator port ID %s\n", name);
return ret;
@@ -3404,8 +3578,37 @@ static int __init srpt_init_module(void)
goto out_unregister_target;
}
+ if (rdma_cm_port) {
+ struct sockaddr_in addr;
+
+ rdma_cm_id = rdma_create_id(&init_net, srpt_rdma_cm_handler,
+ NULL, RDMA_PS_TCP, IB_QPT_RC);
+ if (IS_ERR(rdma_cm_id)) {
+ rdma_cm_id = NULL;
+ pr_err("RDMA/CM ID creation failed\n");
+ goto out_unregister_client;
+ }
+
+ /* We will listen on any RDMA device. */
+ memset(&addr, 0, sizeof(addr));
+ addr.sin_family = AF_INET;
+ addr.sin_port = cpu_to_be16(rdma_cm_port);
+ if (rdma_bind_addr(rdma_cm_id, (void *)&addr)) {
+ pr_err("Binding RDMA/CM ID to port %u failed\n",
+ rdma_cm_port);
+ goto out_unregister_client;
+ }
+
+ if (rdma_listen(rdma_cm_id, 128)) {
+ pr_err("rdma_listen() failed\n");
+ goto out_unregister_client;
+ }
+ }
+
return 0;
+out_unregister_client:
+ ib_unregister_client(&srpt_client);
out_unregister_target:
target_unregister_template(&srpt_template);
out:
@@ -3414,6 +3617,8 @@ static int __init srpt_init_module(void)
static void __exit srpt_cleanup_module(void)
{
+ if (rdma_cm_id)
+ rdma_destroy_id(rdma_cm_id);
ib_unregister_client(&srpt_client);
target_unregister_template(&srpt_template);
}
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h
index 5a7eee561c2f..c79cffd98193 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.h
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.h
@@ -42,6 +42,7 @@
#include <rdma/ib_verbs.h>
#include <rdma/ib_sa.h>
#include <rdma/ib_cm.h>
+#include <rdma/rdma_cm.h>
#include <rdma/rw.h>
#include <scsi/srp.h>
@@ -261,6 +262,7 @@ enum rdma_ch_state {
* @spinlock: Protects free_list and state.
* @free_list: Head of list with free send I/O contexts.
* @state: channel state. See also enum rdma_ch_state.
+ * @using_rdma_cm: Whether the RDMA/CM or IB/CM is used for this channel.
* @processing_wait_list: Whether or not cmd_wait_list is being processed.
* @ioctx_ring: Send ring.
* @ioctx_recv_ring: Receive I/O context ring.
@@ -280,6 +282,9 @@ struct srpt_rdma_ch {
struct {
struct ib_cm_id *cm_id;
} ib_cm;
+ struct {
+ struct rdma_cm_id *cm_id;
+ } rdma_cm;
};
struct ib_cq *cq;
struct ib_cqe zw_cqe;
@@ -300,6 +305,7 @@ struct srpt_rdma_ch {
struct list_head list;
struct list_head cmd_wait_list;
uint16_t pkey;
+ bool using_rdma_cm;
bool processing_wait_list;
struct se_session *sess;
u8 sess_name[24];
--
2.15.1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 35+ messages in thread* Re: [PATCH 00/28] IB/srp and IB/srpt patches
[not found] ` <20180103213938.11664-1-bart.vanassche-Sjgp3cTcYWE@public.gmane.org>
` (25 preceding siblings ...)
2018-01-03 21:39 ` [PATCH 28/28] IB/srpt: Add RDMA/CM support Bart Van Assche
@ 2018-01-03 21:51 ` Jason Gunthorpe
[not found] ` <20180103215130.GN11348-uk2M96/98Pc@public.gmane.org>
26 siblings, 1 reply; 35+ messages in thread
From: Jason Gunthorpe @ 2018-01-03 21:51 UTC (permalink / raw)
To: Bart Van Assche; +Cc: Doug Ledford, linux-rdma-u79uwXL29TY76Z2rM5mHXA
On Wed, Jan 03, 2018 at 01:39:10PM -0800, Bart Van Assche wrote:
> Hello Jason,
>
> This patch series adds RDMA/CM support to the SRP initiator and target drivers
> and also includes a series of bug fixes for the SRPT driver. Please consider
> at least the SRPT bug fixes for kernel v4.16 since one of these fixes is a
> security fix.
Bart, this is really big for the rc6 timeframe.. This might make it if
there is a rc8 this cycle.
I'll grab the two patches marked for stable to for-rc, they look
trivial.
I'm with the DaveM school of thought here that patch series should be
10-15 at most, and have a clear purpose, even if that purpose is just
'assorted one patch fixes'.. If you resend it can you split the series?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 35+ messages in thread