[PATCH rdma-next 0/6] Expose CQ moderation to user space

[PATCH rdma-next 0/6] Expose CQ moderation to user space

[Leon Romanovsky]

This patch set exposes CQ moderation. This will allow to moderate
number of CQEs needed to create an event. Such change brings performance
improvement by reducing pressure on application to receive event per-CQE.

The proposed semantics follows the well-established kernel semantics.

 * cq_max_count - defines the number of cookies needed to create an event.
 * cq_period 	- defines the timeout between last event and a new one
		  that will occur even if cq_max_count was not reached

The patches are available in the git repository at:
  git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-19-2

	Thanks
---------------------------------------

Yonatan Cohen (6):
  IB/uverbs: Allow CQ moderation with modify CQ
  IB/mlx4: Exposing modify CQ callback to uverbs layer
  IB/mlx5: Exposing modify CQ callback to uverbs layer
  IB/uverbs: Add CQ moderation capability to query_device
  IB/mlx4: Add CQ moderation capability to query_device
  IB/mlx5: Add CQ moderation capability to query_device

 drivers/infiniband/core/uverbs.h      |  1 +
 drivers/infiniband/core/uverbs_cmd.c  | 45 +++++++++++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |  1 +
 drivers/infiniband/hw/mlx4/main.c     |  6 +++++
 drivers/infiniband/hw/mlx4/mlx4_ib.h  |  1 +
 drivers/infiniband/hw/mlx5/cq.c       |  3 +++
 drivers/infiniband/hw/mlx5/main.c     | 10 +++++++-
 include/linux/mlx4/cq.h               |  3 +++
 include/linux/mlx5/cq.h               |  3 +++
 include/rdma/ib_verbs.h               | 10 ++++++++
 include/uapi/rdma/ib_user_verbs.h     | 18 +++++++++++++-
 11 files changed, 99 insertions(+), 2 deletions(-)

-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 1/6] IB/uverbs: Allow CQ moderation with modify CQ

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Uverbs support in modify_cq for CQ moderation only.
Gives ability to change cq_max_count and cq_period.
CQ moderation enhance performance by moderating the number
of cookies needed to create an event instead of application
having to suffer from event per cookie.
To achieve CQ moderation the application needs to set cq_max_count
and cq_period.
cq_max_count - defines the number of cookies
               needed to create an event.
cq_period - defines the timeout (micro seconds) between last
            event and a new one that will occur even if
	    cq_max_count was not satisfied

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/core/uverbs.h      |  1 +
 drivers/infiniband/core/uverbs_cmd.c  | 36 +++++++++++++++++++++++++++++++++++
 drivers/infiniband/core/uverbs_main.c |  1 +
 include/rdma/ib_verbs.h               |  4 ++++
 include/uapi/rdma/ib_user_verbs.h     | 11 ++++++++++-
 5 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
index ee2739ae4305..deccefb71a6b 100644
--- a/drivers/infiniband/core/uverbs.h
+++ b/drivers/infiniband/core/uverbs.h
@@ -306,5 +306,6 @@ IB_UVERBS_DECLARE_EX_CMD(destroy_wq);
 IB_UVERBS_DECLARE_EX_CMD(create_rwq_ind_table);
 IB_UVERBS_DECLARE_EX_CMD(destroy_rwq_ind_table);
 IB_UVERBS_DECLARE_EX_CMD(modify_qp);
+IB_UVERBS_DECLARE_EX_CMD(modify_cq);
 
 #endif /* UVERBS_H */
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index ab29a0327831..c7819e5540f1 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3856,3 +3856,39 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	return err;
 }
+
+int ib_uverbs_ex_modify_cq(struct ib_uverbs_file *file,
+			   struct ib_device *ib_dev,
+			   struct ib_udata *ucore,
+			   struct ib_udata *uhw)
+{
+	struct ib_uverbs_ex_modify_cq cmd = {};
+	struct ib_cq *cq;
+	int ret;
+
+	/* sanity checks */
+	if (ucore->inlen > sizeof(cmd) &&
+	    !ib_is_udata_cleared(ucore, sizeof(cmd),
+				 ucore->inlen - sizeof(cmd)))
+		return -EOPNOTSUPP;
+
+	ret = ib_copy_from_udata(&cmd, ucore, min(sizeof(cmd), ucore->inlen));
+	if (ret)
+		return ret;
+
+	if (cmd.attr_mask != IB_CQ_MODERATE)
+		return -EINVAL;
+
+	if (!cmd.comp_mask)
+		return -EINVAL;
+
+	cq = uobj_get_obj_read(cq, cmd.cq_handle, file->ucontext);
+	if (!cq)
+		return -EINVAL;
+
+	ret = ib_modify_cq(cq, cmd.cq_count, cmd.cq_period);
+
+	uobj_put_obj_read(cq);
+
+	return ret;
+}
diff --git a/drivers/infiniband/core/uverbs_main.c b/drivers/infiniband/core/uverbs_main.c
index b5febfd84ee5..381fd9c096ae 100644
--- a/drivers/infiniband/core/uverbs_main.c
+++ b/drivers/infiniband/core/uverbs_main.c
@@ -128,6 +128,7 @@ static int (*uverbs_ex_cmd_table[])(struct ib_uverbs_file *file,
 	[IB_USER_VERBS_EX_CMD_CREATE_RWQ_IND_TBL] = ib_uverbs_ex_create_rwq_ind_table,
 	[IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL] = ib_uverbs_ex_destroy_rwq_ind_table,
 	[IB_USER_VERBS_EX_CMD_MODIFY_QP]        = ib_uverbs_ex_modify_qp,
+	[IB_USER_VERBS_EX_CMD_MODIFY_CQ]        = ib_uverbs_ex_modify_cq,
 };
 
 static void ib_uverbs_add_one(struct ib_device *device);
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 4c0a539cd2a2..ec80fec2d113 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -311,6 +311,10 @@ struct ib_cq_init_attr {
 	u32		flags;
 };
 
+enum ib_cq_attr_mask {
+	IB_CQ_MODERATE = 1 << 0,
+};
+
 struct ib_device_attr {
 	u64			fw_ver;
 	__be64			sys_image_guid;
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index d4e0b53bfc75..854aaafc60d0 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -100,7 +100,8 @@ enum {
 	IB_USER_VERBS_EX_CMD_MODIFY_WQ,
 	IB_USER_VERBS_EX_CMD_DESTROY_WQ,
 	IB_USER_VERBS_EX_CMD_CREATE_RWQ_IND_TBL,
-	IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL
+	IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL,
+	IB_USER_VERBS_EX_CMD_MODIFY_CQ
 };
 
 /*
@@ -1150,6 +1151,14 @@ struct ib_uverbs_ex_destroy_rwq_ind_table  {
 	__u32 ind_tbl_handle;
 };
 
+struct ib_uverbs_ex_modify_cq {
+	__u32 cq_handle;
+	__u32 attr_mask;
+	__u16 cq_count;
+	__u16 cq_period;
+	__u32 comp_mask;
+};
+
 #define IB_DEVICE_NAME_MAX 64
 
 #endif /* IB_USER_VERBS_H */
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 1/6] IB/uverbs: Allow CQ moderation with modify CQ

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3112 bytes --]

On Thu, Oct 19, 2017 at 08:35:19AM +0300, Leon Romanovsky wrote:
> From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>
> Uverbs support in modify_cq for CQ moderation only.
> Gives ability to change cq_max_count and cq_period.
> CQ moderation enhance performance by moderating the number
> of cookies needed to create an event instead of application
> having to suffer from event per cookie.
> To achieve CQ moderation the application needs to set cq_max_count
> and cq_period.
> cq_max_count - defines the number of cookies
>                needed to create an event.
> cq_period - defines the timeout (micro seconds) between last
>             event and a new one that will occur even if
> 	    cq_max_count was not satisfied
>
> Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> ---
>  drivers/infiniband/core/uverbs.h      |  1 +
>  drivers/infiniband/core/uverbs_cmd.c  | 36 +++++++++++++++++++++++++++++++++++
>  drivers/infiniband/core/uverbs_main.c |  1 +
>  include/rdma/ib_verbs.h               |  4 ++++
>  include/uapi/rdma/ib_user_verbs.h     | 11 ++++++++++-
>  5 files changed, 52 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h
> index ee2739ae4305..deccefb71a6b 100644
> --- a/drivers/infiniband/core/uverbs.h
> +++ b/drivers/infiniband/core/uverbs.h
> @@ -306,5 +306,6 @@ IB_UVERBS_DECLARE_EX_CMD(destroy_wq);
>  IB_UVERBS_DECLARE_EX_CMD(create_rwq_ind_table);
>  IB_UVERBS_DECLARE_EX_CMD(destroy_rwq_ind_table);
>  IB_UVERBS_DECLARE_EX_CMD(modify_qp);
> +IB_UVERBS_DECLARE_EX_CMD(modify_cq);
>
>  #endif /* UVERBS_H */
> diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
> index ab29a0327831..c7819e5540f1 100644
> --- a/drivers/infiniband/core/uverbs_cmd.c
> +++ b/drivers/infiniband/core/uverbs_cmd.c
> @@ -3856,3 +3856,39 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
>  	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
>  	return err;
>  }
> +
> +int ib_uverbs_ex_modify_cq(struct ib_uverbs_file *file,
> +			   struct ib_device *ib_dev,
> +			   struct ib_udata *ucore,
> +			   struct ib_udata *uhw)
> +{
> +	struct ib_uverbs_ex_modify_cq cmd = {};
> +	struct ib_cq *cq;
> +	int ret;
> +
> +	/* sanity checks */
> +	if (ucore->inlen > sizeof(cmd) &&
> +	    !ib_is_udata_cleared(ucore, sizeof(cmd),
> +				 ucore->inlen - sizeof(cmd)))
> +		return -EOPNOTSUPP;
> +
> +	ret = ib_copy_from_udata(&cmd, ucore, min(sizeof(cmd), ucore->inlen));
> +	if (ret)
> +		return ret;
> +
> +	if (cmd.attr_mask != IB_CQ_MODERATE)
> +		return -EINVAL;
> +
> +	if (!cmd.comp_mask)
> +		return -EINVAL;

No way that it worked, comp_mask is never set, so it is supposed to be
zero and it is always returning EINVAL.

Additionally, the "attr_mask" is already carrying the same information as
"comp_mask" will handle, hence the "comp_mask" should be dropped.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 1/6] IB/uverbs: Allow CQ moderation with modify CQ

[Doug Ledford]

On Thu, 2017-10-19 at 08:35 +0300, Leon Romanovsky wrote:
> From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> 
> Uverbs support in modify_cq for CQ moderation only.
> Gives ability to change cq_max_count and cq_period.
> CQ moderation enhance performance by moderating the number
> of cookies
     ^ Please use non-slang in your commit messages.  People not
familiar with RDMA will not know what cookies means.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 1/6] IB/uverbs: Allow CQ moderation with modify CQ

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

On Wed, Oct 25, 2017 at 02:28:04PM -0400, Doug Ledford wrote:
> On Thu, 2017-10-19 at 08:35 +0300, Leon Romanovsky wrote:
> > From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >
> > Uverbs support in modify_cq for CQ moderation only.
> > Gives ability to change cq_max_count and cq_period.
> > CQ moderation enhance performance by moderating the number
> > of cookies
>      ^ Please use non-slang in your commit messages.  People not
> familiar with RDMA will not know what cookies means.

I'll do my best.

Thanks

>
> --
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>     GPG KeyID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH rdma-next 2/6] IB/mlx4: Exposing modify CQ callback to uverbs layer

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Exposed mlx4_ib_modify_cq to be called from ib device
verb list.

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/main.c    | 3 +++
 drivers/infiniband/hw/mlx4/mlx4_ib.h | 1 +
 include/linux/mlx4/cq.h              | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index c636842c5be0..58845e67dad1 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -2733,6 +2733,9 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
 	ibdev->ib_dev.get_dev_fw_str    = get_fw_ver_str;
 	ibdev->ib_dev.disassociate_ucontext = mlx4_ib_disassociate_ucontext;
 
+	ibdev->ib_dev.uverbs_ex_cmd_mask |=
+		(1ull << IB_USER_VERBS_EX_CMD_MODIFY_CQ);
+
 	if ((dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_RSS) &&
 	    ((mlx4_ib_port_link_layer(&ibdev->ib_dev, 1) ==
 	    IB_LINK_LAYER_ETHERNET) ||
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 1fa19820355a..9a55064ed69a 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -47,6 +47,7 @@
 #include <linux/mlx4/device.h>
 #include <linux/mlx4/doorbell.h>
 #include <linux/mlx4/qp.h>
+#include <linux/mlx4/cq.h>
 
 #define MLX4_IB_DRV_NAME	"mlx4_ib"
 
diff --git a/include/linux/mlx4/cq.h b/include/linux/mlx4/cq.h
index 09cebe528488..508e8cc5ee86 100644
--- a/include/linux/mlx4/cq.h
+++ b/include/linux/mlx4/cq.h
@@ -136,6 +136,9 @@ enum {
 	MLX4_CQE_BAD_FCS                 = 1 << 4,
 };
 
+#define MLX4_MAX_CQ_PERIOD (BIT(16) - 1)
+#define MLX4_MAX_CQ_COUNT (BIT(16) - 1)
+
 static inline void mlx4_cq_arm(struct mlx4_cq *cq, u32 cmd,
 			       void __iomem *uar_page,
 			       spinlock_t *doorbell_lock)
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 3/6] IB/mlx5: Exposing modify CQ callback to uverbs layer

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Exposed mlx5_ib_modify_cq to be called from ib device
verb list.

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/cq.c   | 3 +++
 drivers/infiniband/hw/mlx5/main.c | 3 ++-
 include/linux/mlx5/cq.h           | 3 +++
 3 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 01b218a3c277..18705cbcdc8c 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -1149,6 +1149,9 @@ int mlx5_ib_modify_cq(struct ib_cq *cq, u16 cq_count, u16 cq_period)
 	if (!MLX5_CAP_GEN(dev->mdev, cq_moderation))
 		return -ENOSYS;
 
+	if (cq_period > MLX5_MAX_CQ_PERIOD)
+		return -EINVAL;
+
 	err = mlx5_core_modify_cq_moderation(dev->mdev, &mcq->mcq,
 					     cq_period, cq_count);
 	if (err)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index c44dd0896f80..da4c577d653c 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -4020,7 +4020,8 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)
 		(1ull << IB_USER_VERBS_EX_CMD_QUERY_DEVICE)	|
 		(1ull << IB_USER_VERBS_EX_CMD_CREATE_CQ)	|
 		(1ull << IB_USER_VERBS_EX_CMD_CREATE_QP)	|
-		(1ull << IB_USER_VERBS_EX_CMD_MODIFY_QP);
+		(1ull << IB_USER_VERBS_EX_CMD_MODIFY_QP)	|
+		(1ull << IB_USER_VERBS_EX_CMD_MODIFY_CQ);
 
 	dev->ib_dev.query_device	= mlx5_ib_query_device;
 	dev->ib_dev.query_port		= mlx5_ib_query_port;
diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
index cc718e245b1e..6be357b219ec 100644
--- a/include/linux/mlx5/cq.h
+++ b/include/linux/mlx5/cq.h
@@ -128,6 +128,9 @@ enum {
 	CQE_SIZE_128_PAD = 2,
 };
 
+#define MLX5_MAX_CQ_PERIOD (BIT(__mlx5_bit_sz(cqc, cq_period)) - 1)
+#define MLX5_MAX_CQ_COUNT (BIT(__mlx5_bit_sz(cqc, cq_max_count)) - 1)
+
 static inline int cqe_sz_to_mlx_sz(u8 size, int padding_128_en)
 {
 	return padding_128_en ? CQE_SIZE_128_PAD :
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 4/6] IB/uverbs: Add CQ moderation capability to query_device

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

The query_device function can now obtain the maximum values for
cq_max_count and cq_period, needed for CQ moderation.
cq_max_count is a 16 bits number that determines the number
of cookies to accumulate before generating an event.
cq_period is a 16 bits number that detemines the timeout in micro
seconds from the last event generated, upon which a new event will
be generated even if cq_max_count was not reached.

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/core/uverbs_cmd.c | 9 +++++++++
 include/rdma/ib_verbs.h              | 6 ++++++
 include/uapi/rdma/ib_user_verbs.h    | 7 +++++++
 3 files changed, 22 insertions(+)

diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index c7819e5540f1..ce969cc84494 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3852,6 +3852,15 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 	resp.tm_caps.max_sge		= attr.tm_caps.max_sge;
 	resp.tm_caps.flags		= attr.tm_caps.flags;
 	resp.response_length += sizeof(resp.tm_caps);
+
+	if (ucore->outlen < resp.response_length + sizeof(resp.cq_moderation_caps))
+		goto end;
+
+	resp.cq_moderation_caps.max_cq_moderation_count  =
+		attr.cq_caps.max_cq_moderation_count;
+	resp.cq_moderation_caps.max_cq_moderation_period =
+		attr.cq_caps.max_cq_moderation_period;
+	resp.response_length += sizeof(resp.cq_moderation_caps);
 end:
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	return err;
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index ec80fec2d113..e1d027acddbd 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -315,6 +315,11 @@ enum ib_cq_attr_mask {
 	IB_CQ_MODERATE = 1 << 0,
 };
 
+struct ib_cq_caps {
+	u16     max_cq_moderation_count;
+	u16     max_cq_moderation_period;
+};
+
 struct ib_device_attr {
 	u64			fw_ver;
 	__be64			sys_image_guid;
@@ -365,6 +370,7 @@ struct ib_device_attr {
 	u32			max_wq_type_rq;
 	u32			raw_packet_caps; /* Use ib_raw_packet_caps enum */
 	struct ib_tm_caps	tm_caps;
+	struct ib_cq_caps       cq_caps;
 };
 
 enum ib_mtu {
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 854aaafc60d0..7745a0eea57c 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -125,6 +125,12 @@ struct ib_uverbs_comp_event_desc {
 	__u64 cq_handle;
 };
 
+struct ib_uverbs_cq_caps {
+	__u16     max_cq_moderation_count;
+	__u16     max_cq_moderation_period;
+	__u32     reserved;
+};
+
 /*
  * All commands from userspace should start with a __u32 command field
  * followed by __u16 in_words and out_words fields (which give the
@@ -263,6 +269,7 @@ struct ib_uverbs_ex_query_device_resp {
 	__u32  max_wq_type_rq;
 	__u32 raw_packet_caps;
 	struct ib_uverbs_tm_caps tm_caps;
+	struct ib_uverbs_cq_caps cq_moderation_caps;
 };
 
 struct ib_uverbs_query_port {
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 5/6] IB/mlx4: Add CQ moderation capability to query_device

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

query_device can now obtain the maximum values for
cq_max_count and cq_period, needed for cq moderation.

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 58845e67dad1..c6420e26f2d8 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -563,6 +563,9 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 		props->max_wq_type_rq = props->max_qp;
 	}
 
+	props->cq_caps.max_cq_moderation_count = MLX4_MAX_CQ_COUNT;
+	props->cq_caps.max_cq_moderation_period = MLX4_MAX_CQ_PERIOD;
+
 	if (!mlx4_is_slave(dev->dev))
 		err = mlx4_get_internal_clock_params(dev->dev, &clock_params);
 
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 6/6] IB/mlx5: Add CQ moderation capability to query_device

[Leon Romanovsky]

From: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

query_device can now obtain the maximum values for
cq_max_count and cq_period, needed for cq moderation.

Signed-off-by: Yonatan Cohen <yonatanc-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/hw/mlx5/main.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index da4c577d653c..61ce3ca503b8 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -790,6 +790,13 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 		props->tm_caps.max_sge = MLX5_TM_MAX_SGE;
 	}
 
+	if (MLX5_CAP_GEN(dev->mdev, cq_moderation)) {
+		props->cq_caps.max_cq_moderation_count =
+						MLX5_MAX_CQ_COUNT;
+		props->cq_caps.max_cq_moderation_period =
+						MLX5_MAX_CQ_PERIOD;
+	}
+
 	if (field_avail(typeof(resp), cqe_comp_caps, uhw->outlen)) {
 		resp.cqe_comp_caps.max_num =
 			MLX5_CAP_GEN(dev->mdev, cqe_compression) ?
-- 
2.14.2

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Thu, Oct 19, 2017 at 08:35:18AM +0300, Leon Romanovsky wrote:
> This patch set exposes CQ moderation. This will allow to moderate
> number of CQEs needed to create an event. Such change brings performance
> improvement by reducing pressure on application to receive event per-CQE.
>
> The proposed semantics follows the well-established kernel semantics.
>
>  * cq_max_count - defines the number of cookies needed to create an event.
>  * cq_period 	- defines the timeout between last event and a new one
> 		  that will occur even if cq_max_count was not reached
>
> The patches are available in the git repository at:
>   git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-19-2
>
> 	Thanks
> ---------------------------------------
>

Doug,

Please drop this series.

I'll resubmit it once user space part will be finished.

Mellanox,
I would like to use this opportunity and to ask publicly, please submit kernel
and user spaces altogether.

It causes for too much errors and it is actually wrong to submit it separately.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Shiraz Saleem]

On Mon, Oct 23, 2017 at 12:47:44PM +0300, Leon Romanovsky wrote:
> On Thu, Oct 19, 2017 at 08:35:18AM +0300, Leon Romanovsky wrote:
> > This patch set exposes CQ moderation. This will allow to moderate
> > number of CQEs needed to create an event. Such change brings performance
> > improvement by reducing pressure on application to receive event per-CQE.
> >
> > The proposed semantics follows the well-established kernel semantics.
> >
> >  * cq_max_count - defines the number of cookies needed to create an event.
> >  * cq_period 	- defines the timeout between last event and a new one
> > 		  that will occur even if cq_max_count was not reached
> >
> > The patches are available in the git repository at:
> >   git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-19-2
> >
> > 	Thanks
> > ---------------------------------------
> >
> 
> Doug,
> 
> Please drop this series.
> 
> I'll resubmit it once user space part will be finished.
> 
> Mellanox,
> I would like to use this opportunity and to ask publicly, please submit kernel
> and user spaces altogether.
> 
> It causes for too much errors and it is actually wrong to submit it separately.
> 

Hi Leon,

I have a generic question. And I apologize if this is redundant and 
has been answered before. But what is the process for user-space submissions?

For libi40iw we make all submissions to this mailing list _only_ and
this is all we are aware of. Are there alternate submission paths?  

Shiraz


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Jason Gunthorpe]

On Mon, Oct 23, 2017 at 08:08:04AM -0500, Shiraz Saleem wrote:

> For libi40iw we make all submissions to this mailing list _only_ and
> this is all we are aware of. Are there alternate submission paths?  

It works best if you send a github pull request to

https://github.com/linux-rdma/rdma-core

At the same time you make your mailing list submission, then we can
see that the patches pass CI and it is less work for the maintainers.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Shiraz Saleem]

On Mon, Oct 23, 2017 at 09:29:19AM -0600, Jason Gunthorpe wrote:
> On Mon, Oct 23, 2017 at 08:08:04AM -0500, Shiraz Saleem wrote:
> 
> > For libi40iw we make all submissions to this mailing list _only_ and
> > this is all we are aware of. Are there alternate submission paths?  
> 
> It works best if you send a github pull request to
> 
> https://github.com/linux-rdma/rdma-core
> 
> At the same time you make your mailing list submission, then we can
> see that the patches pass CI and it is less work for the maintainers.
> 
> Jason

OK. Thanks!

Shiraz
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

On Mon, Oct 23, 2017 at 08:08:04AM -0500, Shiraz Saleem wrote:
> On Mon, Oct 23, 2017 at 12:47:44PM +0300, Leon Romanovsky wrote:
> > On Thu, Oct 19, 2017 at 08:35:18AM +0300, Leon Romanovsky wrote:
> > > This patch set exposes CQ moderation. This will allow to moderate
> > > number of CQEs needed to create an event. Such change brings performance
> > > improvement by reducing pressure on application to receive event per-CQE.
> > >
> > > The proposed semantics follows the well-established kernel semantics.
> > >
> > >  * cq_max_count - defines the number of cookies needed to create an event.
> > >  * cq_period 	- defines the timeout between last event and a new one
> > > 		  that will occur even if cq_max_count was not reached
> > >
> > > The patches are available in the git repository at:
> > >   git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-19-2
> > >
> > > 	Thanks
> > > ---------------------------------------
> > >
> >
> > Doug,
> >
> > Please drop this series.
> >
> > I'll resubmit it once user space part will be finished.
> >
> > Mellanox,
> > I would like to use this opportunity and to ask publicly, please submit kernel
> > and user spaces altogether.
> >
> > It causes for too much errors and it is actually wrong to submit it separately.
> >
>
> Hi Leon,
>
> I have a generic question. And I apologize if this is redundant and
> has been answered before. But what is the process for user-space submissions?

On top of Jason's answer. Sometimes, you can skip mailing list, usually it is
spelling fixes or very local and small fixes to your code.

>
> For libi40iw we make all submissions to this mailing list _only_ and
> this is all we are aware of. Are there alternate submission paths?
>
> Shiraz
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Jason Gunthorpe]

On Mon, Oct 23, 2017 at 12:47:44PM +0300, Leon Romanovsky wrote:

> Mellanox,
> I would like to use this opportunity and to ask publicly, please submit kernel
> and user spaces altogether.

+1

I was looking at the flow counters patches and they make no sense
without also seeing the man pages that describe what the heck this
stuff is.

At the very least put a rdma-core branch someplace and reference it in
your kernel patch series, and it better have really good man pages for
the verbs features being added.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Mon, Oct 23, 2017 at 09:26:38AM -0600, Jason Gunthorpe wrote:
> On Mon, Oct 23, 2017 at 12:47:44PM +0300, Leon Romanovsky wrote:
>
> > Mellanox,
> > I would like to use this opportunity and to ask publicly, please submit kernel
> > and user spaces altogether.
>
> +1
>
> I was looking at the flow counters patches and they make no sense
> without also seeing the man pages that describe what the heck this
> stuff is.

+ kernel and user are one package, it makes no sense to split submission
and to cause artificial delays between them.

>
> At the very least put a rdma-core branch someplace and reference it in
> your kernel patch series, and it better have really good man pages for
> the verbs features being added.
>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Expose CQ moderation to user space

[Doug Ledford]

On Mon, 2017-10-23 at 12:47 +0300, Leon Romanovsky wrote:
> Doug,
> 
> Please drop this series.

Done.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html