[RFC 0/5] Hardware tag matching

[RFC 0/5] Hardware tag matching

[Artemy Kovalyov]

Introduction
============

The MPI standard defines a set of rules, known as tag-matching, for matching
source send operations to destination receives according to the following
attributes:

* Communicator
* User tag - wild card may be specified by the receiver
* Source rank - wild card may be specified by the receiver
* Destination rank - wild card may be specified by the receiver

These matching attributes are specified by all Send and Receive operations.
Send operations from a given source to a given destination are processed in
the order in which the Sends were posted. Receive operations are associated
with the earliest send operation (from any source) that matches the
attributes, in the order in which the Receives were posted. Note that Receive
tags are not necessarily consumed in the order they are created, e.g., a later
generated tag may be consumed if earlier tags do not satisfy the matching
rules.

When a message arrives at the receiver, MPI implementations often classify it
as either 'expected' or 'unexpected' according to whether a Receive operation
with a matching tag has already been posted by the application. In the
expected case, the message may be processed immediately. In the unexpected
case, the message is saved in an unexpected message queue, and will be
processed when a matching Receive operation is posted.

To bound the amount of memory to hold unexpected messages, MPI implementations
use 2 data transfer protocols. The 'eager' protocol is used for small
messages. Eager messages are sent without any prior synchronization and
processed/buffered at the receiver. Typically, with RDMA, a single RDMA-Send
operation is used to transfer the data.

The 'rendezvous' protocol is used for large messages. Initially, only the
message tag is sent along with some meta-data. Only when the tag is matched to
a Receive operation, will the receiver initiate the corresponding data
transfer. A common RDMA implementation is to send the message tag with an
RDMA-Send, and transfer the data with an RDMA-Read issued by the receiver.
When the transfer is complete, the receiver will notify the sender that its
buffer may be freed using an RDMA-Send.

RDMA tag-matching offload
=========================

Tag-matching offload satisfies the following principals:
-   Tag-matching is viewed as an RDMA application, and thus does not affect the
    RDMA transport in any way (*)
-   Tag-matching processing will be split between HW and SW.
    *   HW will hold a bounded prefix of Receive tags
-   HW will process and transfer any expected message that matches a tag held
    in HW.
    *   In case the message uses the rendezvous protocol, HW will also initiate
	the RDMA-Read data transfer and send a notification message when the
	data transfer completes.
-   SW will handle any message that is either unexpected or whose tag is not
    held in HW.

(*) This concept can apply to additional application-specific offloads in the
future.

Tag-matching is initially defined for RC transport. Tag-matching messages are
encapsulated in RDMA-Send messages and contain the following headers:

    0			1		    2			3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   Tag Matching Header (TMH):
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |	Operation  |		      reserved			   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |			  User data (optional)			   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |				 Tag				   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |				 Tag				   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Rendezvous Header (RVH):
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |			     Virtual Address			   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |			     Virtual Address			   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |			       Remote Key			   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |				 Length				   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Tag-matching messages always contain a TMH. An RHV is added for Rendezvous
request messages. The following message formats are defined:
-   Eager request: TMH | payload
-   Rendezvous request: TMH | RHV | optional meta-data (**)
-   Rendezvous response: TMH
Note that rendezvous data transfers are standard RDMA-Reads

(**) Rendezvous request messages may also arrive unexpected; in this case, the
message is handled in SW optionally leveraging additional meta-data passed by
the sender

As tag-matching messages are standard RDMA-Sends, no special HW support is
needed at the sender. At the receiver, we introduce a new SRQ type - a
Tag-Matching SRQ (TM-SRQ). The TM-SRQ forms the serialization point for
matching messages coming from any of the associated RC connections, and reports
all tag matching completions and events to a dedicated CQ.
2 kinds of buffers may be posted to the TM-SRQ:
-   Buffers associated with tags (tagged-buffers), which are used when a match
    is made by HW
-   Standard SRQ buffers, which are used for unexpected messages (from HW's
    perspective)
When a message is matched by HW, the payload is transferred directly to the
application buffer (both in the eager and the rendezvous case), while skipping
any TM headers. Otherwise, the entire message, including any TM headers, is
scattered to the SRQ buffer.

Since unexpected messages are handled in SW, there exists an inherent race
between the arrival of messages from the wire and posting of new tagged
buffers. For example, consider 2 incoming messages m1 and m2 and matching
buffers b1 and b2 that are posted asynchronously. If b1 is posted after m1
arrives but before m2, m1 would be delivered as an unexpected message while m2
would match b1, violating the ordering rules. Consequently, whenever HW deems
a message unexpected, tag matching must be disabled for new tags until SW and
HW synchronize. This synchronization is achieved by reporting to HW the number
of unexpected messages handled by SW (with respect to the current posted
tags). When the SW and HW are in synch, tag matching resumes normally.

This RFC describes the corresponding Verbs API and implementation for RDMA tag
matching offload.

Tag Matching Verbs
==================

Capabilities
------------

Tag matching capabilities are queried by ibv_query_device(), and report the
following attributes:

* max_rndv_hdr_size - Max size of rendezvous request message
* max_num_tags - Max number of tagged buffers in a TM-SRQ matching list
* max_ops - Max number of outstanding tag matching operations
* max_sge - Max number of SGEs in a tagged buffer
* flags - the following flags are currently defined:
    - IBV_TM_CAP_RC - Support tag matching on RC transport


TM-SRQ creation
---------------

TM-SRQs are created by the ibv_create_srq_ex() Verb, which accepts the
following new attributes:
* srq_type - set to IBV_SRQT_TAG_MATCHING
* comp_mask - set the IBV_SRQ_INIT_ATTR_TM flag
* tm_cap - TM properties for this TM-SRQ; defined as follows:

struct ibv_tm_cap {
	 uint32_t max_num_tags;   /* Matching list size */
	 uint32_t max_ops;	  /* Number of outstanding TM operations */
}

Similarly to XRC SRQs, a TM-SRQ has a dedicated CQ.

RC QPs are associated with the TM-SRQ just like standard SRQs. However, the
ownership of the QP's Send Queue is passed to the TM-SRQ, which uses it to
initiate rendezvous RDMA-Reads. Receive completions are reported to the
TM-SRQ's CQ.


Managing TM receive buffers
---------------------------

Untagged (unexpected) buffers are posted using the standard
ibv_post_srq_recv() Verb.

Tagged buffers are manipulated by a new ibv_post_srq_ops() Verb:

int ibv_post_srq_ops(struct ibv_srq *srq, struct ibv_ops_wr *wr,
		     struct ibv_ops_wr **bad_wr);

struct ibv_ops_wr {
	 uint64_t		 wr_id;    /* User defined WR ID */
	 /* Pointer to next WR in list, NULL if last WR */
	 struct ibv_ops_wr	*next;
	 enum ibv_ops_wr_opcode  opcode;   /* From enum ibv_ops_wr_opcode */
	 int			 flags;    /* From enum ibv_ops_flags */
	 struct {
		  /* Number of unexpected messages
		   * handled by SW */
		  uint32_t unexpected_cnt;
		  /* Input parameter for the DEL opcode
		   * and output parameter for the ADD opcode */
		  uint32_t handle;
		  struct {
			  /* WR ID for TM_RECV */
			  uint64_t		  recv_wr_id;
			  struct ibv_sge	 *sg_list;
			  int			  num_sge;
			  uint64_t		  tag;
			  uint64_t		  mask;
		  } add;
	 } tm;
};

The following opcodes are defined:

Opcode IBV_WR_TAG_ADD - add a tagged buffer entry to the tag matching list.
The input consists of an SGE list, a tag, a mask (matching parameters), and the
latest unexpected message count. A handle that uniquely identifies the entry is
returned upon return.

Opcode IBV_WR_TAG_DEL - delete a tag entry.
The input is an entry handle returned from a previous IBV_WR_TAG_ADD
operation, and the latest unexpected message count.

Note that the operation may fail if the associated tag was consumed by an
incoming message. In this case IBV_WC_TM_ERR status will be returned in WC.

Opcode IBV_WR_TAG_SYNC - report the number of unexpected messages handled by
SW.
The input comprises only the unexpected message count. To reduce explicit
synchronization to a minimum, all completions indicate when synchronization is
necessary by setting the IBV_WC_TM_SYNC_REQ flag.

ibv_post_srq_ops() operations are non-signaled by default. To request an
explicit completion for a given operation, the standard IBV_OPS_SIGNALED flag
must be set. The number of outstanding tag-manipulation operations must not
exceed the 'max_ops' capability.

While 'wr_id' identifies the tag manipulation operation itself, the
'recv_wr_id' field is used to identify the tagged buffer in receive
completions.


Sending TM messages
-------------------

TM messages are sent using standard RC Send operations. A TM message comprises
a Tag-Matching Header (TMH), an optional Rendezvous Header (RVH), and
payload.

TMH and RVH defined in infiniband/tm_types.h:

struct ibv_tmh {
	  uint8_t	  opcode;
	  uint8_t	  reserved[3];
	  __be32	  app_ctx;
	  __be64	  tag;
};

struct ibv_rvh {
	  __be64	  va;
	  __be32	  rkey;
	  __be32	  len;
};

The following opcodes are defined:

* IBV_TM_NO_TAG   - Send a message without a tag.
Such a message will always be treated as unexpected by the receiver TM-SRQ.
Any data following the opcode is ignored by the tag matching logic, and the
message is delivered in its entirety (including the opcode) to the standard
SRQ buffer.

* IBV_TM_OP_EAGER - Send an eager tagged message.
The message consists of a TMH followed by payload.

* IBV_TM_OP_RNDV  - Send a tagged rendezvous request.
The message consists of a TMH, an RVH, and optional additional data (which may
be inspected by receiver SW if the message is deemed unexpected). The RVH must
refer to a registered buffer containing the rendezvous payload. The total
rendezvous message size must not exceed the 'max_rndv_hdr_size' capability.
The Sender must consider the operation outstanding until a TM message with the
IBV_TM_OP_FIN opcode is received, after which the buffer may be deregistered
and freed.

* IBV_TM_OP_FIN   - Send a rendezvous completion indication.
The message consists of a copy of the original TMH and RVH of the rendezvous
request, apart the opcode. This message is sent after the receiver has
completed the transfer of the rendezvous payload by an RDMA-read operation. It
may be sent either by HW or SW, depending on whether the rendezvous request
was handled as expected or unexpected by the TM-SRQ.

TM completion processing
------------------------

There are 2 types of TM completions: tag-manipulation and receive completions.

Tag-manipulation operations generate the following completion opcodes:
* IBV_WC_TM_ADD - completion of a tag addition operation
* IBV_WC_TM_DEL - completion of a tag removal operation
* IBV_WC_TM_SYNC - completion of synchronization operation

These completions are complemented by the IBV_WC_TM_SYNC_REQ flag, which
indicates whether further HW synchronization is needed.

TM receive completions generate the following completion codes:
* IBV_WC_RECV - standard SRQ completion; used for unexpected messages
* IBV_WC_TM_NO_TAG - completion of a message sent with the IBV_TM_NO_TAG opcode.
* IBV_WC_TM_RECV - completion of a tag-matching operation

The IBV_WC_TM_RECV completion is complemented by the following completion flags:
-	IBV_WC_TM_MATCH - a match was performed
-	IBV_WC_TM_DATA_VALID - all data of the matched message has been
	delivered to memory

In single-packet eager messages, both flags are set. When larger messages or
rendezvous transfers are involved, matching and data transfer completion are
distinct events that generate 2 completion events for the same 'recv_wr_id'.
While data transfer completions may be arbitrarily delayed depending on
message size, matching completion is reported immediately and is always
serialized with respect to other matches and the completion of unexpected
messages.

In addition, IBV_WC_TM_RECV completions provide further information about the
matched message. This information is obtained using extended CQ processing via
the following extractor function:

static inline void ibv_wc_read_tm_info(struct ibv_cq_ex *cq,
				       struct ibv_wc_tm_info *tm_info);

struct ibv_wc_tm_info {
	  uint64_t		  tag;	   /* Tag information */
	  uint32_t		  priv;    /* Application context */
};

Finally, when a posted tagged buffer is insufficient to hold the data of a
rendezvous request, the HW completes the buffer with an
IBV_WC_TM_RNDV_INCOMPLETE status. In this case, the TMH and RVH headers are
scattered into the tagged buffer (tag-matching has still been completed!), and
message handling is resumed by SW.

Artemy Kovalyov (5):
  verbs: Expose tag matching capabilities
  verbs: Introduce tag matching SRQ
  verbs: Tag matching list manipulation interface
  verbs: Tag matching send interface
  verbs: Tag matching receive interface

 libibverbs/tm_types.h |  70 ++++++++++++++++++++++++++++++++
 libibverbs/verbs.h    | 110 +++++++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 174 insertions(+), 6 deletions(-)
 create mode 100644 libibverbs/tm_types.h

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC 1/5] verbs: Expose tag matching capabilities

[Artemy Kovalyov]

Expose tag matching capabilities and show them in ibv_devinfo.

* max_rndv_hdr_size - Max size of RNDV message
* max_num_tags - Max number of entries in tag matching list
* max_ops - Max number of outstanding list operations
* max_sge - Max number of SGE in a tag matching entry
* capability_flags - TM capabilities mask:
    - IBV_TM_CAP_RC - Support tag matching on RC transport

Change-Id: I6f15a6db209155be6406bb54c714796047979f68
Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/verbs.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index b27dfd1..43fa386 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -259,6 +259,23 @@ enum ibv_raw_packet_caps {
 	IBV_RAW_PACKET_CAP_IP_CSUM		= 1 << 2,
 };
 
+enum ibv_tm_cap_flags {
+	IBV_TM_CAP_RC		    = 1 << 0
+};
+
+struct ibv_tm_caps {
+	/* Max size for RNDV message */
+	uint32_t max_rndv_hdr_size;
+	/* Max number of entries in tag matching list */
+	uint32_t max_num_tags;
+	/* TM capabilities mask - enumerated in ibv_tm_cap_flags */
+	uint32_t capability_flags;
+	/* Max number of outstanding list operations */
+	uint32_t max_ops;
+	/* Max number of SGE in a tag matching entry */
+	uint32_t max_sge;
+};
+
 struct ibv_device_attr_ex {
 	struct ibv_device_attr	orig_attr;
 	uint32_t		comp_mask;
@@ -271,6 +288,7 @@ struct ibv_device_attr_ex {
 	uint32_t		max_wq_type_rq;
 	struct ibv_packet_pacing_caps packet_pacing_caps;
 	uint32_t		raw_packet_caps; /* Use ibv_raw_packet_caps */
+	struct ibv_tm_caps	tm_caps;
 };
 
 enum ibv_mtu {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC 2/5] verbs: Introduce tag matching SRQ

[Artemy Kovalyov]

Introducing tag matching SRQ (TM-SRQ), which retains basic semantic of
regular SRQ, reports completions to own CQ, and has additional tag
based message receiving mechanism.

Change-Id: Ic527680984f343608507ba2162147435be2f7aee
Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/verbs.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 43fa386..120471a 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -666,7 +666,8 @@ struct ibv_srq_init_attr {
 
 enum ibv_srq_type {
 	IBV_SRQT_BASIC,
-	IBV_SRQT_XRC
+	IBV_SRQT_XRC,
+	IBV_SRQT_TAG_MATCHING
 };
 
 enum ibv_srq_init_attr_mask {
@@ -674,7 +675,13 @@ enum ibv_srq_init_attr_mask {
 	IBV_SRQ_INIT_ATTR_PD		= 1 << 1,
 	IBV_SRQ_INIT_ATTR_XRCD		= 1 << 2,
 	IBV_SRQ_INIT_ATTR_CQ		= 1 << 3,
-	IBV_SRQ_INIT_ATTR_RESERVED	= 1 << 4
+	IBV_SRQ_INIT_ATTR_TAG_MATCHING  = 1 << 4,
+	IBV_SRQ_INIT_ATTR_RESERVED	= 1 << 5
+};
+
+struct ibv_tm_cap {
+	uint32_t		max_num_tags;
+	uint32_t		max_ops;
 };
 
 struct ibv_srq_init_attr_ex {
@@ -686,6 +693,7 @@ struct ibv_srq_init_attr_ex {
 	struct ibv_pd	       *pd;
 	struct ibv_xrcd	       *xrcd;
 	struct ibv_cq	       *cq;
+	struct ibv_tm_cap	tm_cap;
 };
 
 enum ibv_wq_type {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC 3/5] verbs: Tag matching list manipulation interface

[Artemy Kovalyov]

TM-SRQ shows demand of robust offload configuration primitive.
This patch introduced a new verb named ibv_post_srq_ops() to supply an
API to perform such operations like tag matching list manipulations.

Change-Id: I9fd8df2152b12f06b6b1545afec9ea1a259c0f58
Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/verbs.h | 58 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 55 insertions(+), 3 deletions(-)

diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 120471a..42d7967 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -420,7 +420,8 @@ enum ibv_wc_status {
 	IBV_WC_INV_EEC_STATE_ERR,
 	IBV_WC_FATAL_ERR,
 	IBV_WC_RESP_TIMEOUT_ERR,
-	IBV_WC_GENERAL_ERR
+	IBV_WC_GENERAL_ERR,
+	IBV_WC_TM_ERR
 };
 const char *ibv_wc_status_str(enum ibv_wc_status status);
 
@@ -438,7 +439,11 @@ enum ibv_wc_opcode {
  * receive by testing (opcode & IBV_WC_RECV).
  */
 	IBV_WC_RECV			= 1 << 7,
-	IBV_WC_RECV_RDMA_WITH_IMM
+	IBV_WC_RECV_RDMA_WITH_IMM,
+
+	IBV_WC_TM_ADD,
+	IBV_WC_TM_DEL,
+	IBV_WC_TM_SYNC
 };
 
 enum {
@@ -479,7 +484,8 @@ enum ibv_wc_flags {
 	IBV_WC_GRH		= 1 << 0,
 	IBV_WC_WITH_IMM		= 1 << 1,
 	IBV_WC_IP_CSUM_OK	= 1 << IBV_WC_IP_CSUM_OK_SHIFT,
-	IBV_WC_WITH_INV         = 1 << 3
+	IBV_WC_WITH_INV		= 1 << 3,
+	IBV_WC_TM_SYNC_REQ	= 1 << 4
 };
 
 struct ibv_wc {
@@ -1011,6 +1017,35 @@ struct ibv_recv_wr {
 	int			num_sge;
 };
 
+enum ibv_ops_wr_opcode {
+	IBV_WR_TAG_ADD,
+	IBV_WR_TAG_DEL,
+	IBV_WR_TAG_SYNC
+};
+
+enum ibv_ops_flags {
+	IBV_OPS_SIGNALED = 1 << 0,
+	IBV_OPS_TM_SYNC  = 1 << 1
+};
+
+struct ibv_ops_wr {
+	uint64_t				wr_id;
+	struct ibv_ops_wr		       *next;
+	enum ibv_ops_wr_opcode			opcode;
+	int					flags;
+	struct {
+		uint32_t			unexpected_cnt;
+		uint32_t			handle;
+		struct {
+			uint64_t		recv_wr_id;
+			struct ibv_sge	       *sg_list;
+			int			num_sge;
+			uint64_t		tag;
+			uint64_t		mask;
+		} add;
+	} tm;
+};
+
 struct ibv_mw_bind {
 	uint64_t		wr_id;
 	int			send_flags;
@@ -1540,6 +1575,9 @@ enum verbs_context_mask {
 
 struct verbs_context {
 	/*  "grows up" - new fields go here */
+	int (*post_srq_ops)(struct ibv_srq *srq,
+			    struct ibv_ops_wr *op,
+			    struct ibv_ops_wr **bad_op);
 	int (*destroy_rwq_ind_table)(struct ibv_rwq_ind_table *rwq_ind_table);
 	struct ibv_rwq_ind_table *(*create_rwq_ind_table)(struct ibv_context *context,
 							  struct ibv_rwq_ind_table_init_attr *init_attr);
@@ -2038,6 +2076,20 @@ static inline int ibv_post_srq_recv(struct ibv_srq *srq,
 	return srq->context->ops.post_srq_recv(srq, recv_wr, bad_recv_wr);
 }
 
+static inline int ibv_post_srq_ops(struct ibv_srq *srq,
+				   struct ibv_ops_wr *op,
+				   struct ibv_ops_wr **bad_op)
+{
+	struct verbs_context *vctx;
+
+	vctx = verbs_get_ctx_op(srq->context, post_srq_ops);
+	if (!vctx) {
+		*bad_op = op;
+		return ENOSYS;
+	}
+	return vctx->post_srq_ops(srq, op, bad_op);
+}
+
 /**
  * ibv_create_qp - Create a queue pair.
  */
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC 4/5] verbs: Tag matching send interface

[Artemy Kovalyov]

Add tag matching header (TMH) and rendezvous header (RVH) definitions
to infiniband/tm_types.h

Change-Id: I35a2088190ff4a1776f77b43a4564bb6f65ca83c
Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/tm_types.h | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100644 libibverbs/tm_types.h

diff --git a/libibverbs/tm_types.h b/libibverbs/tm_types.h
new file mode 100644
index 0000000..7626787
--- /dev/null
+++ b/libibverbs/tm_types.h
@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2017 Mellanox Technologies Ltd.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ *
+ */
+#ifndef _TM_TYPES_H
+#define _TM_TYPES_H
+
+#include <linux/types.h>
+#include <stdint.h>
+
+#ifdef __cplusplus
+#  define BEGIN_C_DECLS extern "C" {
+#  define END_C_DECLS   }
+#else				/* !__cplusplus */
+#  define BEGIN_C_DECLS
+#  define END_C_DECLS
+#endif				/* __cplusplus */
+
+BEGIN_C_DECLS
+
+enum ibv_tmh_op {
+	IBV_TMH_NO_TAG	      = 0,
+	IBV_TMH_RNDV	      = 1,
+	IBV_TMH_FIN	      = 2,
+	IBV_TMH_EAGER	      = 3
+};
+
+struct ibv_tmh {
+	uint8_t		opcode;      /* from enum ibv_tmh_op */
+	uint8_t		reserved[3]; /* must be zero */
+	__be32		app_ctx;
+	__be64		tag;
+};
+
+struct ibv_rvh {
+	__be64		va;
+	__be32		rkey;
+	__be32		len;
+};
+
+END_C_DECLS
+#endif				/* _TM_TYPES_H */
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC 5/5] verbs: Tag matching receive interface

[Artemy Kovalyov]

Extending poll_cq with tag matching support.
New read function named ibv_wc_read_tm_info was added to read
additional TM related information from CQE.

Change-Id: Ia5bae6732d03ce6d12d731fdd675a2668e258934
Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/verbs.h | 28 ++++++++++++++++++++++++----
 1 file changed, 24 insertions(+), 4 deletions(-)

diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 42d7967..0c5be05 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -421,7 +421,8 @@ enum ibv_wc_status {
 	IBV_WC_FATAL_ERR,
 	IBV_WC_RESP_TIMEOUT_ERR,
 	IBV_WC_GENERAL_ERR,
-	IBV_WC_TM_ERR
+	IBV_WC_TM_ERR,
+	IBV_WC_TM_RNDV_INCOMPLETE
 };
 const char *ibv_wc_status_str(enum ibv_wc_status status);
 
@@ -443,7 +444,9 @@ enum ibv_wc_opcode {
 
 	IBV_WC_TM_ADD,
 	IBV_WC_TM_DEL,
-	IBV_WC_TM_SYNC
+	IBV_WC_TM_SYNC,
+	IBV_WC_TM_RECV,
+	IBV_WC_TM_NO_TAG
 };
 
 enum {
@@ -461,6 +464,7 @@ enum ibv_create_cq_wc_flags {
 	IBV_WC_EX_WITH_COMPLETION_TIMESTAMP	= 1 << 7,
 	IBV_WC_EX_WITH_CVLAN		= 1 << 8,
 	IBV_WC_EX_WITH_FLOW_TAG		= 1 << 9,
+	IBV_WC_EX_WITH_TM_INFO		= 1 << 10
 };
 
 enum {
@@ -477,7 +481,8 @@ enum {
 	IBV_CREATE_CQ_SUP_WC_FLAGS = IBV_WC_STANDARD_FLAGS |
 				IBV_WC_EX_WITH_COMPLETION_TIMESTAMP |
 				IBV_WC_EX_WITH_CVLAN |
-				IBV_WC_EX_WITH_FLOW_TAG
+				IBV_WC_EX_WITH_FLOW_TAG |
+				IBV_WC_EX_WITH_TM_INFO
 };
 
 enum ibv_wc_flags {
@@ -485,7 +490,9 @@ enum ibv_wc_flags {
 	IBV_WC_WITH_IMM		= 1 << 1,
 	IBV_WC_IP_CSUM_OK	= 1 << IBV_WC_IP_CSUM_OK_SHIFT,
 	IBV_WC_WITH_INV		= 1 << 3,
-	IBV_WC_TM_SYNC_REQ	= 1 << 4
+	IBV_WC_TM_SYNC_REQ	= 1 << 4,
+	IBV_WC_TM_MATCH		= 1 << 5,
+	IBV_WC_TM_DATA_VALID	= 1 << 6
 };
 
 struct ibv_wc {
@@ -1132,6 +1139,11 @@ struct ibv_poll_cq_attr {
 	uint32_t comp_mask;
 };
 
+struct ibv_wc_tm_info {
+	uint64_t		tag;	 /* Tag information */
+	uint32_t		priv;	 /* Application context */
+};
+
 struct ibv_cq_ex {
 	struct ibv_context     *context;
 	struct ibv_comp_channel *channel;
@@ -1164,6 +1176,8 @@ struct ibv_cq_ex {
 	uint64_t (*read_completion_ts)(struct ibv_cq_ex *current);
 	uint16_t (*read_cvlan)(struct ibv_cq_ex *current);
 	uint32_t (*read_flow_tag)(struct ibv_cq_ex *current);
+	void (*read_tm_info)(struct ibv_cq_ex *current,
+			     struct ibv_wc_tm_info *tm_info);
 };
 
 static inline struct ibv_cq *ibv_cq_ex_to_cq(struct ibv_cq_ex *cq)
@@ -1252,6 +1266,12 @@ static inline uint32_t ibv_wc_read_flow_tag(struct ibv_cq_ex *cq)
 	return cq->read_flow_tag(cq);
 }
 
+static inline void ibv_wc_read_tm_info(struct ibv_cq_ex *cq,
+				       struct ibv_wc_tm_info *tm_info)
+{
+	cq->read_tm_info(cq, tm_info);
+}
+
 static inline int ibv_post_wq_recv(struct ibv_wq *wq,
 				   struct ibv_recv_wr *recv_wr,
 				   struct ibv_recv_wr **bad_recv_wr)
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Christoph Hellwig]

Artem, what firmware version / what specific hardware do I need for
this?  I've been working on prototyping a simpler subset of this
(1:1 tag matching) for use in NVMe or NFS over RDMA, so I'd like to
play around with your code to see if it matches my use case as well.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Artemy Kovalyov]

On 12/06/2017 10:40, Christoph Hellwig wrote:
> Artem, what firmware version / what specific hardware do I need for
> this?  I've been working on prototyping a simpler subset of this
> (1:1 tag matching) for use in NVMe or NFS over RDMA, so I'd like to
> play around with your code to see if it matches my use case as well.

Christoph, here is Contect-X5 firmware with Tag matching support:
  http://www.mellanox.com/page/firmware_table_ConnectX5IB
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

On Mon, Jun 12, 2017 at 12:41:12PM +0300, Artemy Kovalyov wrote:
> On 12/06/2017 10:40, Christoph Hellwig wrote:
> > Artem, what firmware version / what specific hardware do I need for
> > this?  I've been working on prototyping a simpler subset of this
> > (1:1 tag matching) for use in NVMe or NFS over RDMA, so I'd like to
> > play around with your code to see if it matches my use case as well.
>
> Christoph, here is Contect-X5 firmware with Tag matching support:
>  http://www.mellanox.com/page/firmware_table_ConnectX5IB

Latest available FW for ConnectX-5 supports Tag Matching.

Thanks

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC 0/5] Hardware tag matching

[Christoph Hellwig]

On Mon, Jun 12, 2017 at 01:46:00PM +0300, Leon Romanovsky wrote:
> On Mon, Jun 12, 2017 at 12:41:12PM +0300, Artemy Kovalyov wrote:
> > On 12/06/2017 10:40, Christoph Hellwig wrote:
> > > Artem, what firmware version / what specific hardware do I need for
> > > this?  I've been working on prototyping a simpler subset of this
> > > (1:1 tag matching) for use in NVMe or NFS over RDMA, so I'd like to
> > > play around with your code to see if it matches my use case as well.
> >
> > Christoph, here is Contect-X5 firmware with Tag matching support:
> >  http://www.mellanox.com/page/firmware_table_ConnectX5IB
> 
> Latest available FW for ConnectX-5 supports Tag Matching.

Is there support for ConnectX-4 as well?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1112 bytes --]

On Mon, Jun 12, 2017 at 04:05:07AM -0700, Christoph Hellwig wrote:
> On Mon, Jun 12, 2017 at 01:46:00PM +0300, Leon Romanovsky wrote:
> > On Mon, Jun 12, 2017 at 12:41:12PM +0300, Artemy Kovalyov wrote:
> > > On 12/06/2017 10:40, Christoph Hellwig wrote:
> > > > Artem, what firmware version / what specific hardware do I need for
> > > > this?  I've been working on prototyping a simpler subset of this
> > > > (1:1 tag matching) for use in NVMe or NFS over RDMA, so I'd like to
> > > > play around with your code to see if it matches my use case as well.
> > >
> > > Christoph, here is Contect-X5 firmware with Tag matching support:
> > >  http://www.mellanox.com/page/firmware_table_ConnectX5IB
> >
> > Latest available FW for ConnectX-5 supports Tag Matching.
>
> Is there support for ConnectX-4 as well?

Right now - no.
AFAIK, there is need HW support and CX-4 lacks it.

Thanks

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Jason Gunthorpe]

On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
> Introducing tag matching SRQ (TM-SRQ), which retains basic semantic of
> regular SRQ, reports completions to own CQ, and has additional tag
> based message receiving mechanism.
> 
> Change-Id: Ic527680984f343608507ba2162147435be2f7aee

Drop change-id from all your patches

>  enum ibv_srq_type {
>  	IBV_SRQT_BASIC,
> -	IBV_SRQT_XRC
> +	IBV_SRQT_XRC,
> +	IBV_SRQT_TAG_MATCHING
>  };

Please put trailing commas on all these enum changes in all patches

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Jason Gunthorpe]

On Mon, Jun 12, 2017 at 09:42:42AM +0300, Artemy Kovalyov wrote:
> Introduction
> ============

This entire text should go into a documentation/ file in rdma-core, or
perhaps a man page.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 4/5] verbs: Tag matching send interface

[Jason Gunthorpe]

On Mon, Jun 12, 2017 at 09:42:46AM +0300, Artemy Kovalyov wrote:
> Add tag matching header (TMH) and rendezvous header (RVH) definitions
> to infiniband/tm_types.h
> 
> Change-Id: I35a2088190ff4a1776f77b43a4564bb6f65ca83c
> Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>  libibverbs/tm_types.h | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 70 insertions(+)
>  create mode 100644 libibverbs/tm_types.h
> 
> diff --git a/libibverbs/tm_types.h b/libibverbs/tm_types.h
> new file mode 100644
> index 0000000..7626787
> +++ b/libibverbs/tm_types.h

There is no cmake stuff for this file, and it is never used in this
series - what is the intention here?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 5/5] verbs: Tag matching receive interface

[Jason Gunthorpe]

On Mon, Jun 12, 2017 at 09:42:47AM +0300, Artemy Kovalyov wrote:
> Extending poll_cq with tag matching support.

> +static inline void ibv_wc_read_tm_info(struct ibv_cq_ex *cq,
> +				       struct ibv_wc_tm_info *tm_info)
> +{
> +	cq->read_tm_info(cq, tm_info);
> +}

This series has no man pages.

Every single new public API needs corresponding man page updates.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Artemy Kovalyov]

On 12/06/2017 19:38, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
>>  enum ibv_srq_type {
>>  	IBV_SRQT_BASIC,
>> -	IBV_SRQT_XRC
>> +	IBV_SRQT_XRC,
>> +	IBV_SRQT_TAG_MATCHING
>>  };
>
> Please put trailing commas on all these enum changes in all patches

So do we drop overboard all -Wpedantic users?
>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 0/5] Hardware tag matching

[Artemy Kovalyov]

On 12/06/2017 19:40, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 09:42:42AM +0300, Artemy Kovalyov wrote:
>> Introduction
>> ============
>
> This entire text should go into a documentation/ file in rdma-core, or
> perhaps a man page.
Sure.

This is only request for comments.
For the beginning I send only conceptual things - proposed API changes
and documentation to explain it.

Once it will get acceptance I'll send patch series, with all technical
stuff: this text in Documentation/tag_matching.md, CMakeLists.txt
update, man pages updates, examples updates and provider/mlx5
implementation.

>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Jason Gunthorpe]

On Mon, Jun 12, 2017 at 11:38:09PM +0300, Artemy Kovalyov wrote:
> 
> 
> On 12/06/2017 19:38, Jason Gunthorpe wrote:
> >On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
> >> enum ibv_srq_type {
> >> 	IBV_SRQT_BASIC,
> >>-	IBV_SRQT_XRC
> >>+	IBV_SRQT_XRC,
> >>+	IBV_SRQT_TAG_MATCHING
> >> };
> >
> >Please put trailing commas on all these enum changes in all patches
> 
> So do we drop overboard all -Wpedantic users?

Eh? Try it.

You only get warnings if your compiler is running in c89 or ansi mode,
which went 'overboard' a long time ago.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Artemy Kovalyov]

On 12/06/2017 23:50, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 11:38:09PM +0300, Artemy Kovalyov wrote:
>>
>>
>> On 12/06/2017 19:38, Jason Gunthorpe wrote:
>>> On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
>>>> enum ibv_srq_type {
>>>> 	IBV_SRQT_BASIC,
>>>> -	IBV_SRQT_XRC
>>>> +	IBV_SRQT_XRC,
>>>> +	IBV_SRQT_TAG_MATCHING
>>>> };
>>>
>>> Please put trailing commas on all these enum changes in all patches
>>
>> So do we drop overboard all -Wpedantic users?
>
> Eh? Try it.
>
> You only get warnings if your compiler is running in c89 or ansi mode,
> which went 'overboard' a long time ago.
Bare -Wpedantic -Werror fails with this.

>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Jason Gunthorpe]

On Tue, Jun 13, 2017 at 12:09:29AM +0300, Artemy Kovalyov wrote:
> 
> 
> On 12/06/2017 23:50, Jason Gunthorpe wrote:
> >On Mon, Jun 12, 2017 at 11:38:09PM +0300, Artemy Kovalyov wrote:
> >>
> >>
> >>On 12/06/2017 19:38, Jason Gunthorpe wrote:
> >>>On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
> >>>>enum ibv_srq_type {
> >>>>	IBV_SRQT_BASIC,
> >>>>-	IBV_SRQT_XRC
> >>>>+	IBV_SRQT_XRC,
> >>>>+	IBV_SRQT_TAG_MATCHING
> >>>>};
> >>>
> >>>Please put trailing commas on all these enum changes in all patches
> >>
> >>So do we drop overboard all -Wpedantic users?
> >
> >Eh? Try it.
> >
> >You only get warnings if your compiler is running in c89 or ansi mode,
> >which went 'overboard' a long time ago.
> Bare -Wpedantic -Werror fails with this.

You need -std=c99 on older compilers.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Artemy Kovalyov]

On 13/06/2017 00:18, Jason Gunthorpe wrote:
> On Tue, Jun 13, 2017 at 12:09:29AM +0300, Artemy Kovalyov wrote:
>>
>>
>> On 12/06/2017 23:50, Jason Gunthorpe wrote:
>>> On Mon, Jun 12, 2017 at 11:38:09PM +0300, Artemy Kovalyov wrote:
>>>>
>>>>
>>>> On 12/06/2017 19:38, Jason Gunthorpe wrote:
>>>>> On Mon, Jun 12, 2017 at 09:42:44AM +0300, Artemy Kovalyov wrote:
>>>>>> enum ibv_srq_type {
>>>>>> 	IBV_SRQT_BASIC,
>>>>>> -	IBV_SRQT_XRC
>>>>>> +	IBV_SRQT_XRC,
>>>>>> +	IBV_SRQT_TAG_MATCHING
>>>>>> };
>>>>>
>>>>> Please put trailing commas on all these enum changes in all patches
>>>>
>>>> So do we drop overboard all -Wpedantic users?
>>>
>>> Eh? Try it.
>>>
>>> You only get warnings if your compiler is running in c89 or ansi mode,
>>> which went 'overboard' a long time ago.
>> Bare -Wpedantic -Werror fails with this.
>
> You need -std=c99 on older compilers.

That was my point - implying new standards' ubiquity we narrowing 
compatibility.
I not insist, just want to clarify whether this code standard was chosen 
for the project.

>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Jason Gunthorpe]

On Tue, Jun 13, 2017 at 12:31:11AM +0300, Artemy Kovalyov wrote:

> I not insist, just want to clarify whether this code standard was
> chosen for the project.

It is, and we already have things that are not C89 in the public
headers for some time.

If someone wishes to support C89 then they will need to make a patch
and travis support, otherwise please continue to follow the existing
examples of adding the trailing, as other new cases do.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 2/5] verbs: Introduce tag matching SRQ

[Artemy Kovalyov]

On 13/06/2017 00:40, Jason Gunthorpe wrote:
> On Tue, Jun 13, 2017 at 12:31:11AM +0300, Artemy Kovalyov wrote:
>
>> I not insist, just want to clarify whether this code standard was
>> chosen for the project.
>
> It is, and we already have things that are not C89 in the public
> headers for some time.
>
> If someone wishes to support C89 then they will need to make a patch
> and travis support, otherwise please continue to follow the existing
> examples of adding the trailing, as other new cases do.

Legit.

>
> Jason
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC 4/5] verbs: Tag matching send interface

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

On Mon, Jun 12, 2017 at 10:41:40AM -0600, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 09:42:46AM +0300, Artemy Kovalyov wrote:
> > Add tag matching header (TMH) and rendezvous header (RVH) definitions
> > to infiniband/tm_types.h
> >
> > Change-Id: I35a2088190ff4a1776f77b43a4564bb6f65ca83c
> > Signed-off-by: Artemy Kovalyov <artemyko-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >  libibverbs/tm_types.h | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 70 insertions(+)
> >  create mode 100644 libibverbs/tm_types.h
> >
> > diff --git a/libibverbs/tm_types.h b/libibverbs/tm_types.h
> > new file mode 100644
> > index 0000000..7626787
> > +++ b/libibverbs/tm_types.h
>
> There is no cmake stuff for this file, and it is never used in this
> series - what is the intention here?

Jason,

It is RFC, the intention is to present headers and APIs in most clear way
without distraction of error handling, manuals and Cmake changes.

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC 5/5] verbs: Tag matching receive interface

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Mon, Jun 12, 2017 at 10:42:06AM -0600, Jason Gunthorpe wrote:
> On Mon, Jun 12, 2017 at 09:42:47AM +0300, Artemy Kovalyov wrote:
> > Extending poll_cq with tag matching support.
>
> > +static inline void ibv_wc_read_tm_info(struct ibv_cq_ex *cq,
> > +				       struct ibv_wc_tm_info *tm_info)
> > +{
> > +	cq->read_tm_info(cq, tm_info);
> > +}
>
> This series has no man pages.
>
> Every single new public API needs corresponding man page updates.

It will.

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC 0/5] Hardware tag matching

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

On Mon, Jun 12, 2017 at 09:42:42AM +0300, Artemy Kovalyov wrote:

<snip>

Hi All,

Next OFVWG meeting will be dedicated to discussion about TAG matching.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]