[PATCH 0/3] Enable checksum offload capability reporting

[PATCH 0/3] Enable checksum offload capability reporting

[Bodong Wang]

The checksum offload capability reporting is enabled based on extended verbs.
The capability field has sub-fields for every link layer, and depends on device
cap, each link layer will support specific QP types. These will be reported to
user space.

I'm new to uverbs extensions and looking forward for review comments on that
aspect of the patches.

Bodong Wang (3):
  IB/core: Add support of checksum capability reporting in ib verbs
  IB/uverbs: Add support for checksum capability reporting in user verbs
  IB/mlx4: Report checksum offload cap when query device

 drivers/infiniband/core/uverbs_cmd.c |  7 +++++++
 drivers/infiniband/hw/mlx4/main.c    |  3 +++
 include/rdma/ib_verbs.h              | 10 ++++++++++
 include/uapi/rdma/ib_user_verbs.h    |  6 ++++++
 4 files changed, 26 insertions(+)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 1/3] IB/core: Add support of checksum capability reporting in ib verbs

[Bodong Wang]

A new filed csum_cap is added to both ib_query_device. It contains two members:
eth_csum_cap and ib_csum_cap, indicates checksum capability of Ethernet and
Infiniband link layer respectively for different QP types.

Current checksum caps use the following enum members:
- IB_CSUM_SUPPORT_UD: device supports validation/calculation of csum for UD QP.
- IB_CSUM_SUPPORT_RAW: device supports validation/calculation of csum for raw QP.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 include/rdma/ib_verbs.h | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index b0f898e..94dbaee 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -183,6 +183,11 @@ struct ib_cq_init_attr {
 	u32		flags;
 };
 
+struct ib_csum_cap_per_link {
+	uint32_t  eth_csum_cap;
+	uint32_t  ib_csum_cap;
+};
+
 struct ib_device_attr {
 	u64			fw_ver;
 	__be64			sys_image_guid;
@@ -229,6 +234,7 @@ struct ib_device_attr {
 	struct ib_odp_caps	odp_caps;
 	uint64_t		timestamp_mask;
 	uint64_t		hca_core_clock; /* in KHZ */
+	struct ib_csum_cap_per_link csum_cap;
 };
 
 enum ib_mtu {
@@ -868,6 +874,10 @@ enum ib_qp_create_flags {
 	IB_QP_CREATE_RESERVED_END		= 1 << 31,
 };
 
+enum ib_csum_cap_flags {
+	IB_CSUM_SUPPORT_UD	= (1 << IB_QPT_UD),
+	IB_CSUM_SUPPORT_RAW	= (1 << IB_QPT_RAW_PACKET),
+};
 
 /*
  * Note: users may not call ib_close_qp or ib_destroy_qp from the event_handler
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] IB/core: Add support of checksum capability reporting in ib verbs

[Christoph Lameter]

On Wed, 16 Sep 2015, Bodong Wang wrote:

> A new filed csum_cap is added to both ib_query_device. It contains two members:
> eth_csum_cap and ib_csum_cap, indicates checksum capability of Ethernet and
> Infiniband link layer respectively for different QP types.
>
> Current checksum caps use the following enum members:
> - IB_CSUM_SUPPORT_UD: device supports validation/calculation of csum for UD QP.
> - IB_CSUM_SUPPORT_RAW: device supports validation/calculation of csum for raw QP.

A combination? Is it possible then to also support calculation without
validation? Maybe we want to receive packets that do have invalid
checksums.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH 1/3] IB/core: Add support of checksum capability reporting in ib verbs

[Bodong Wang]

For RX: if corresponding QP is not supported, it will not validate the csum, but packets are still received normally. 
For TX: if corresponding QP is not supported for csum calculation and user application sets the IBV_SEND_IP_CSUM flag, it will return error.

-----Original Message-----
From: Christoph Lameter [mailto:cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org] 
Sent: Wednesday, September 16, 2015 12:07 PM
To: Bodong Wang
Cc: dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Bodong Wang; Or Gerlitz; jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org; Moshe Lazer; Haggai Eran; Matan Barak
Subject: Re: [PATCH 1/3] IB/core: Add support of checksum capability reporting in ib verbs

On Wed, 16 Sep 2015, Bodong Wang wrote:

> A new filed csum_cap is added to both ib_query_device. It contains two members:
> eth_csum_cap and ib_csum_cap, indicates checksum capability of 
> Ethernet and Infiniband link layer respectively for different QP types.
>
> Current checksum caps use the following enum members:
> - IB_CSUM_SUPPORT_UD: device supports validation/calculation of csum for UD QP.
> - IB_CSUM_SUPPORT_RAW: device supports validation/calculation of csum for raw QP.

A combination? Is it possible then to also support calculation without validation? Maybe we want to receive packets that do have invalid checksums.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 1/3] IB/core: Add support of checksum capability reporting in ib verbs

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2547 bytes --]

On 09/16/2015 11:56 AM, Bodong Wang wrote:
> A new filed csum_cap is added to both ib_query_device. It contains two members:
> eth_csum_cap and ib_csum_cap, indicates checksum capability of Ethernet and
> Infiniband link layer respectively for different QP types.
> 
> Current checksum caps use the following enum members:
> - IB_CSUM_SUPPORT_UD: device supports validation/calculation of csum for UD QP.
> - IB_CSUM_SUPPORT_RAW: device supports validation/calculation of csum for raw QP.
> 
> Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  include/rdma/ib_verbs.h | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index b0f898e..94dbaee 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -183,6 +183,11 @@ struct ib_cq_init_attr {
>  	u32		flags;
>  };
>  
> +struct ib_csum_cap_per_link {
> +	uint32_t  eth_csum_cap;
> +	uint32_t  ib_csum_cap;
> +};
> +

I generally don't like to waste this many bits on this little
information.  64 bits total for what only uses 4 bits right now, and
even on the high side would probably only ever use 8 or 12 bits, is
excessive.

That said, it's cleaner and easier to read than something like a double
shift where ib is lower 16 bits, eth is upper 16 bits, so I won't
request you change it, just register my eyebrow raise over the number of
bits used to record so little information.  (In fairness, I thought
about make you shrink it down, but the area of the struct you are adding
this to is currently 64bit aligned and it is reasonably likely that the
next item will need 64bit alignment, so saving bits only to possibly
loose them to alignment is a exercise in futility)

>  struct ib_device_attr {
>  	u64			fw_ver;
>  	__be64			sys_image_guid;
> @@ -229,6 +234,7 @@ struct ib_device_attr {
>  	struct ib_odp_caps	odp_caps;
>  	uint64_t		timestamp_mask;
>  	uint64_t		hca_core_clock; /* in KHZ */
> +	struct ib_csum_cap_per_link csum_cap;
>  };
>  
>  enum ib_mtu {
> @@ -868,6 +874,10 @@ enum ib_qp_create_flags {
>  	IB_QP_CREATE_RESERVED_END		= 1 << 31,
>  };
>  
> +enum ib_csum_cap_flags {
> +	IB_CSUM_SUPPORT_UD	= (1 << IB_QPT_UD),
> +	IB_CSUM_SUPPORT_RAW	= (1 << IB_QPT_RAW_PACKET),
> +};
>  
>  /*
>   * Note: users may not call ib_close_qp or ib_destroy_qp from the event_handler
> 


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

[PATCH 2/3] IB/uverbs: Add support for checksum capability reporting in user verbs

[Bodong Wang]

New field csum_cap is added to respective uverbs counterpart according
to ib_verbs.

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/core/uverbs_cmd.c | 7 +++++++
 include/uapi/rdma/ib_user_verbs.h    | 6 ++++++
 2 files changed, 13 insertions(+)

diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index bbb02ff..9d5deec 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3464,6 +3464,13 @@ int ib_uverbs_ex_query_device(struct ib_uverbs_file *file,
 	resp.hca_core_clock = attr.hca_core_clock;
 	resp.response_length += sizeof(resp.hca_core_clock);
 
+	if (ucore->outlen < resp.response_length + sizeof(resp.csum_cap))
+		goto end;
+
+	resp.csum_cap.eth_csum_cap = attr.csum_cap.eth_csum_cap;
+	resp.csum_cap.ib_csum_cap = attr.csum_cap.ib_csum_cap;
+	resp.response_length += sizeof(resp.csum_cap);
+
 end:
 	err = ib_copy_to_udata(ucore, &resp, resp.response_length);
 	if (err)
diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_verbs.h
index 978841e..9d69546 100644
--- a/include/uapi/rdma/ib_user_verbs.h
+++ b/include/uapi/rdma/ib_user_verbs.h
@@ -218,6 +218,11 @@ struct ib_uverbs_odp_caps {
 	__u32 reserved;
 };
 
+struct ib_uverbs_csum_cap_per_link {
+	__u32 eth_csum_cap;
+	__u32 ib_csum_cap;
+};
+
 struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_query_device_resp base;
 	__u32 comp_mask;
@@ -225,6 +230,7 @@ struct ib_uverbs_ex_query_device_resp {
 	struct ib_uverbs_odp_caps odp_caps;
 	__u64 timestamp_mask;
 	__u64 hca_core_clock; /* in KHZ */
+	struct ib_uverbs_csum_cap_per_link csum_cap;
 };
 
 struct ib_uverbs_query_port {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 3/3] IB/mlx4: Report checksum offload cap when query device

[Bodong Wang]

Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 drivers/infiniband/hw/mlx4/main.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 8be6db8..a70ca6a 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -217,6 +217,9 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
 		props->device_cap_flags |= IB_DEVICE_MANAGED_FLOW_STEERING;
 	}
 
+	props->csum_cap.eth_csum_cap |= IB_CSUM_SUPPORT_RAW;
+	props->csum_cap.ib_csum_cap |= IB_CSUM_SUPPORT_UD;
+
 	props->vendor_id	   = be32_to_cpup((__be32 *) (out_mad->data + 36)) &
 		0xffffff;
 	props->vendor_part_id	   = dev->dev->persist->pdev->device;
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/3] IB/mlx4: Report checksum offload cap when query device

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1910 bytes --]

On 09/16/2015 11:56 AM, Bodong Wang wrote:
> Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
>  drivers/infiniband/hw/mlx4/main.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
> index 8be6db8..a70ca6a 100644
> --- a/drivers/infiniband/hw/mlx4/main.c
> +++ b/drivers/infiniband/hw/mlx4/main.c
> @@ -217,6 +217,9 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
>  		props->device_cap_flags |= IB_DEVICE_MANAGED_FLOW_STEERING;
>  	}
>  
> +	props->csum_cap.eth_csum_cap |= IB_CSUM_SUPPORT_RAW;
> +	props->csum_cap.ib_csum_cap |= IB_CSUM_SUPPORT_UD;
> +
>  	props->vendor_id	   = be32_to_cpup((__be32 *) (out_mad->data + 36)) &
>  		0xffffff;
>  	props->vendor_part_id	   = dev->dev->persist->pdev->device;
> 

This patch highlights something I didn't think about on the previous
patch.  Why separate eth/ib if you have per QP flags?  The QP denotes
the ib/eth relationship without the need to separate it into two
different caps.  In other words, you can never have an IB qp type on eth
because the only eth QP types we support other than RAW are all RDMA and
not IP.  Really, there's enough spare bits in ib_device_cap_flags that
you could do away with the new caps entirely.  Right now, we support UD
(which we already have a flag for), we can add two flags (for RAW and
RC) and that should cover all of the foreseeable options as that would
allow us to extend IP CSUM support to cover connected mode and cover all
of the current options.  I don't see us doing IP traffic in any other
situation, so I thing that should suffice.  Bits 25 and 26 could be used
for the two new bits.  Then you just need to extend the bits to user space.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH 3/3] IB/mlx4: Report checksum offload cap when query device

[Or Gerlitz]

On Fri, Sep 18, 2015 at 5:46 AM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> On 09/16/2015 11:56 AM, Bodong Wang wrote:
>> Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> ---
>>  drivers/infiniband/hw/mlx4/main.c | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
>> index 8be6db8..a70ca6a 100644
>> --- a/drivers/infiniband/hw/mlx4/main.c
>> +++ b/drivers/infiniband/hw/mlx4/main.c
>> @@ -217,6 +217,9 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
>>               props->device_cap_flags |= IB_DEVICE_MANAGED_FLOW_STEERING;
>>       }
>>
>> +     props->csum_cap.eth_csum_cap |= IB_CSUM_SUPPORT_RAW;
>> +     props->csum_cap.ib_csum_cap |= IB_CSUM_SUPPORT_UD;
>> +

> This patch highlights something I didn't think about on the previous
> patch.  Why separate eth/ib if you have per QP flags?  The QP denotes
> the ib/eth relationship without the need to separate it into two
> different caps.  In other words, you can never have an IB qp type on eth
> because the only eth QP types we support other than RAW are all RDMA and
> not IP.  Really, there's enough spare bits in ib_device_cap_flags that
> you could do away with the new caps entirely.  Right now, we support UD
> (which we already have a flag for), we can add two flags (for RAW and
> RC) and that should cover all of the foreseeable options as that would
> allow us to extend IP CSUM support to cover connected mode and cover all
> of the current options.  I don't see us doing IP traffic in any other
> situation, so I thing that should suffice.  Bits 25 and 26 could be used
> for the two new bits.  Then you just need to extend the bits to user space.

Doug,

The vendor may support the offload for a certain QP type only over
certain link. E.g mlx4 support checksum for UD QPs only over IB but
not over Eth,  checksum for RC QPs isn't supported, and RAW_PACKET QPs
are available anyway only for Eth links. But if this is what you think
needs to be done, I guess we can do that.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/3] IB/mlx4: Report checksum offload cap when query device

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 3272 bytes --]

On 09/21/2015 05:41 PM, Or Gerlitz wrote:
> On Fri, Sep 18, 2015 at 5:46 AM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>> On 09/16/2015 11:56 AM, Bodong Wang wrote:
>>> Signed-off-by: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>>> ---
>>>  drivers/infiniband/hw/mlx4/main.c | 3 +++
>>>  1 file changed, 3 insertions(+)
>>>
>>> diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
>>> index 8be6db8..a70ca6a 100644
>>> --- a/drivers/infiniband/hw/mlx4/main.c
>>> +++ b/drivers/infiniband/hw/mlx4/main.c
>>> @@ -217,6 +217,9 @@ static int mlx4_ib_query_device(struct ib_device *ibdev,
>>>               props->device_cap_flags |= IB_DEVICE_MANAGED_FLOW_STEERING;
>>>       }
>>>
>>> +     props->csum_cap.eth_csum_cap |= IB_CSUM_SUPPORT_RAW;
>>> +     props->csum_cap.ib_csum_cap |= IB_CSUM_SUPPORT_UD;
>>> +
> 
>> This patch highlights something I didn't think about on the previous
>> patch.  Why separate eth/ib if you have per QP flags?  The QP denotes
>> the ib/eth relationship without the need to separate it into two
>> different caps.  In other words, you can never have an IB qp type on eth
>> because the only eth QP types we support other than RAW are all RDMA and
>> not IP.  Really, there's enough spare bits in ib_device_cap_flags that
>> you could do away with the new caps entirely.  Right now, we support UD
>> (which we already have a flag for), we can add two flags (for RAW and
>> RC) and that should cover all of the foreseeable options as that would
>> allow us to extend IP CSUM support to cover connected mode and cover all
>> of the current options.  I don't see us doing IP traffic in any other
>> situation, so I thing that should suffice.  Bits 25 and 26 could be used
>> for the two new bits.  Then you just need to extend the bits to user space.
> 
> Doug,
> 
> The vendor may support the offload for a certain QP type only over
> certain link.

Exactly my point.

> E.g mlx4 support checksum for UD QPs only over IB but
> not over Eth,

As it should be.  We are, after all, talking about IP embedded in UD
RDMA.  Over IB that makes sense.  Over Eth it makes no sense.  If you
are going to do IP on Eth, then just do IP, don't do IP in UD.  I see no
reason to support this construct.

>  checksum for RC QPs isn't supported,

But could be for IB.  The fact that it isn't is why there is an ongoing
effort to work around this issue.

> and RAW_PACKET QPs
> are available anyway only for Eth links.

Correct.  And this will never be available on IB.

> But if this is what you think
> needs to be done, I guess we can do that.

Here's the only matrix of IP checksumming that makes sense:

1) UD over IB (because it is one of the supported IPoIB types)
2) RC over IB (same as above)
3) Raw ETH over Eth (because IP over Eth makes sense and is a common
type of packet to send on Raw Eth, but Raw Eth will never be sent on IB
as it isn't supported there at all)

Anything else would require adding more Raw ETH QPs elsewhere, or
expanding the IPoIB spec to include more connection types.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH 3/3] IB/mlx4: Report checksum offload cap when query device

[Or Gerlitz]

On Tue, Sep 22, 2015 at 12:59 AM, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> Here's the only matrix of IP checksumming that makes sense:
>
> 1) UD over IB (because it is one of the supported IPoIB types)
> 2) RC over IB (same as above)
> 3) Raw ETH over Eth (because IP over Eth makes sense and is a common
> type of packet to send on Raw Eth, but Raw Eth will never be sent on IB
> as it isn't supported there at all)
>
> Anything else would require adding more Raw ETH QPs elsewhere, or
> expanding the IPoIB spec to include more connection types.

OK, I'm good with that, so you want two news bits of device caps and
not the too fancy
matrix of checksum-caps-for-qp-type-and-link-type.

>> Really, there's enough spare bits in ib_device_cap_flags that
>> you could do away with the new caps entirely.

Yes, Bodong, please go ahead and use bits 26,27 of
include/rdma/ib_verbs.h :: enum ib_device_cap_flags
for the two new caps Doug asked.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html