[PATCH rdma-core 0/2] PCI write end padding

[PATCH rdma-core 0/2] PCI write end padding

[Yishai Hadas]

This series from Noa is the supplementary part of the kernel series that was
merged into 4.15 'for-next' branch by Doug few days ago.

The series exposes the "PCI write end padding" device capability and enables QP
and WQ creation with that option.

PR was sent:
https://github.com/linux-rdma/rdma-core/pull/250

Note:
The Travis issue that was reported in GITHUB is not related to this series but
to some 'vmw_pvrdma' patches that were previously taken.

Noa Osherovich (2):
  verbs: Report the device's PCI write end paddding capability
  verbs: Enable QP and WQ creation with PCI write end padding option

 libibverbs/cmd.c                  | 3 ++-
 libibverbs/examples/devinfo.c     | 5 ++++-
 libibverbs/man/ibv_create_qp_ex.3 | 1 +
 libibverbs/man/ibv_create_wq.3    | 3 ++-
 libibverbs/verbs.h                | 5 ++++-
 5 files changed, 13 insertions(+), 4 deletions(-)

-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Yishai Hadas]

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

There are PCIe root complex that are able to optimize their
performance when incoming data is multiple full cache lines.
Expose the device capability to report whether the device supports
padding the ending of incoming packets to full cache line, such that
the last upstream write generated by the incoming packet will be a
full cache line.

User should consider several factors before activating this feature:
- In case of high CPU memory load (which may cause PCI backpressure in
  turn), if a large percent of the writes are partial cache line, this
  feature should be checked as an optional solution.
- This feature might reduce performance if most packets are between
  one and two cache lines and PCIe throughput has reached its maximum
  capacity. E.g. 65B packet from the network port will lead to 128B
  write on PCIe, which may cause trafiic on PCIe to reach high
  throughput.

Signed-off-by: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/examples/devinfo.c | 5 ++++-
 libibverbs/verbs.h            | 1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/libibverbs/examples/devinfo.c b/libibverbs/examples/devinfo.c
index 169da2e..d02952e 100644
--- a/libibverbs/examples/devinfo.c
+++ b/libibverbs/examples/devinfo.c
@@ -331,10 +331,13 @@ static void print_odp_caps(const struct ibv_odp_caps *caps)
 static void print_device_cap_flags_ex(uint64_t device_cap_flags_ex)
 {
 	uint64_t ex_flags = device_cap_flags_ex & 0xffffffff00000000ULL;
-	uint64_t unknown_flags = ~(IBV_DEVICE_RAW_SCATTER_FCS);
+	uint64_t unknown_flags = ~(IBV_DEVICE_RAW_SCATTER_FCS |
+				   IBV_DEVICE_PCI_WRITE_END_PADDING);
 
 	if (ex_flags & IBV_DEVICE_RAW_SCATTER_FCS)
 		printf("\t\t\t\t\tRAW_SCATTER_FCS\n");
+	if (ex_flags & IBV_DEVICE_PCI_WRITE_END_PADDING)
+		printf("\t\t\t\t\tPCI_WRITE_END_PADDING\n");
 	if (ex_flags & unknown_flags)
 		printf("\t\t\t\t\tUnknown flags: 0x%" PRIX64 "\n",
 		       ex_flags & unknown_flags);
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 3e543cb..025e321 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -142,6 +142,7 @@ enum ibv_device_cap_flags {
  * enum range is limited to 4 bytes.
  */
 #define IBV_DEVICE_RAW_SCATTER_FCS (1ULL << 34)
+#define IBV_DEVICE_PCI_WRITE_END_PADDING (1ULL << 36)
 
 enum ibv_atomic_cap {
 	IBV_ATOMIC_NONE,
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Jason Gunthorpe]

On Tue, Nov 14, 2017 at 03:47:07PM +0200, Yishai Hadas wrote:
> From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> 
> There are PCIe root complex that are able to optimize their
> performance when incoming data is multiple full cache lines.
> Expose the device capability to report whether the device supports
> padding the ending of incoming packets to full cache line, such that
> the last upstream write generated by the incoming packet will be a
> full cache line.
> 
> User should consider several factors before activating this feature:
> - In case of high CPU memory load (which may cause PCI backpressure in
>   turn), if a large percent of the writes are partial cache line, this
>   feature should be checked as an optional solution.
> - This feature might reduce performance if most packets are between
>   one and two cache lines and PCIe throughput has reached its maximum
>   capacity. E.g. 65B packet from the network port will lead to 128B
>   write on PCIe, which may cause trafiic on PCIe to reach high
>   throughput.

This commit message would make a far better man page revision than
what was provided :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Jason Gunthorpe]

On Tue, Nov 14, 2017 at 03:47:07PM +0200, Yishai Hadas wrote:

>   * enum range is limited to 4 bytes.
>   */
>  #define IBV_DEVICE_RAW_SCATTER_FCS (1ULL << 34)
> +#define IBV_DEVICE_PCI_WRITE_END_PADDING (1ULL << 36)

Man page?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Yishai Hadas]

On 11/14/2017 10:33 PM, Jason Gunthorpe wrote:
> On Tue, Nov 14, 2017 at 03:47:07PM +0200, Yishai Hadas wrote:
> 
>>    * enum range is limited to 4 bytes.
>>    */
>>   #define IBV_DEVICE_RAW_SCATTER_FCS (1ULL << 34)
>> +#define IBV_DEVICE_PCI_WRITE_END_PADDING (1ULL << 36)
> 
> Man page?
> 

The man page was updated with a detailed description on the above 
capability, see PR:
https://github.com/linux-rdma/rdma-core/pull/250
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Jason Gunthorpe]

On Wed, Nov 15, 2017 at 03:10:30PM +0200, Yishai Hadas wrote:

> The man page was updated with a detailed description on the above
> capability, see PR:
> https://github.com/linux-rdma/rdma-core/pull/250

Yah, that is nicer

Here is some copy-editing:

Extended device capability flags (device_cap_flags_ex):
.br
.TP 7
IBV_DEVICE_PCI_WRITE_END_PADDING

Indicates the device has support for padding PCI writes to a full cache line.

Padding packets to full cache lines reduces the amount of traffic required at
the memory controller at the expense of creating more traffic on the PCI-E
port.

Workloads that have a high CPU memory load and low PCI-E utilization will
benefit from this feature, while workloads that have a high PCI-E utilization
and small packets will be harmed.

For instance, with a 128 byte cache line size, the transfer of any packets
less than 128 bytes will require a full 128 transfer on PCI, pontentially
doubling the required PCI-E bandwith.

This feature can be enabled on a QP or WQ basis via the
IBV_QP_CREATE_PCI_WRITE_END_PADDING or IBV_WQ_FLAGS_PCI_WRITE_END_PADDING
flags.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-core 1/2] verbs: Report the device's PCI write end paddding capability

[Yishai Hadas]

On 11/15/2017 5:15 PM, Jason Gunthorpe wrote:
> On Wed, Nov 15, 2017 at 03:10:30PM +0200, Yishai Hadas wrote:
> 
>> The man page was updated with a detailed description on the above
>> capability, see PR:
>> https://github.com/linux-rdma/rdma-core/pull/250
> 
> Yah, that is nicer
> 
> Here is some copy-editing:
> 
> Extended device capability flags (device_cap_flags_ex):
> .br
> .TP 7
> IBV_DEVICE_PCI_WRITE_END_PADDING
> 
> Indicates the device has support for padding PCI writes to a full cache line.
> 
> Padding packets to full cache lines reduces the amount of traffic required at
> the memory controller at the expense of creating more traffic on the PCI-E
> port.
> 
> Workloads that have a high CPU memory load and low PCI-E utilization will
> benefit from this feature, while workloads that have a high PCI-E utilization
> and small packets will be harmed.
> 
> For instance, with a 128 byte cache line size, the transfer of any packets
> less than 128 bytes will require a full 128 transfer on PCI, pontentially
> doubling the required PCI-E bandwith.
> 
> This feature can be enabled on a QP or WQ basis via the
> IBV_QP_CREATE_PCI_WRITE_END_PADDING or IBV_WQ_FLAGS_PCI_WRITE_END_PADDING
> flags.
> 

OK, PR was updated accordingly.
https://github.com/linux-rdma/rdma-core/pull/250
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-core 2/2] verbs: Enable QP and WQ creation with PCI write end padding option

[Yishai Hadas]

From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Update QP and WQ creation flags to allow setting the PCI write end
padding capability during QP/WQ creation.

Man pages were updated accordingly.

Signed-off-by: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Reviewed-by: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---
 libibverbs/cmd.c                  | 3 ++-
 libibverbs/man/ibv_create_qp_ex.3 | 1 +
 libibverbs/man/ibv_create_wq.3    | 3 ++-
 libibverbs/verbs.h                | 4 +++-
 4 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/libibverbs/cmd.c b/libibverbs/cmd.c
index eb5f263..62fcd0e 100644
--- a/libibverbs/cmd.c
+++ b/libibverbs/cmd.c
@@ -966,7 +966,8 @@ enum {
 	CREATE_QP_EX2_SUP_CREATE_FLAGS = IBV_QP_CREATE_BLOCK_SELF_MCAST_LB |
 					 IBV_QP_CREATE_SCATTER_FCS |
 					 IBV_QP_CREATE_CVLAN_STRIPPING |
-					 IBV_QP_CREATE_SOURCE_QPN,
+					 IBV_QP_CREATE_SOURCE_QPN |
+					 IBV_QP_CREATE_PCI_WRITE_END_PADDING,
 };
 
 int ibv_cmd_create_qp_ex2(struct ibv_context *context,
diff --git a/libibverbs/man/ibv_create_qp_ex.3 b/libibverbs/man/ibv_create_qp_ex.3
index fac0e03..865b5fe 100644
--- a/libibverbs/man/ibv_create_qp_ex.3
+++ b/libibverbs/man/ibv_create_qp_ex.3
@@ -59,6 +59,7 @@ IBV_QP_CREATE_BLOCK_SELF_MCAST_LB       = 1 << 1, /* Prevent self multicast loop
 IBV_QP_CREATE_SCATTER_FCS               = 1 << 8, /* FCS field will be scattered to host memory */
 IBV_QP_CREATE_CVLAN_STRIPPING           = 1 << 9, /* CVLAN field will be stripped from incoming packets */
 IBV_QP_CREATE_SOURCE_QPN                = 1 << 10, /* The created QP will use the source_qpn as its wire QP number */
+IBV_QP_CREATE_PCI_WRITE_END_PADDING     = 1 << 11, /* Incoming packets will be padded to cacheline size */
 .in -8
 };
 .nf
diff --git a/libibverbs/man/ibv_create_wq.3 b/libibverbs/man/ibv_create_wq.3
index a540130..10fe965 100644
--- a/libibverbs/man/ibv_create_wq.3
+++ b/libibverbs/man/ibv_create_wq.3
@@ -42,7 +42,8 @@ enum ibv_wq_flags {
 IBV_WQ_FLAGS_CVLAN_STRIPPING            = 1 << 0, /* CVLAN field will be stripped from incoming packets */
 IBV_WQ_FLAGS_SCATTER_FCS                = 1 << 1, /* FCS field will be scattered to host memory */
 IBV_WQ_FLAGS_DELAY_DROP                 = 1 << 2, /* Packets won't be dropped immediately if no receive WQEs */
-IBV_WQ_FLAGS_RESERVED                   = 1 << 3,
+IBV_WQ_FLAGS_PCI_WRITE_END_PADDING      = 1 << 3, /* Incoming packets will be padded to cacheline size */
+IBV_WQ_FLAGS_RESERVED                   = 1 << 4,
 .in -8
 };
 .nf
diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h
index 025e321..b39dc30 100644
--- a/libibverbs/verbs.h
+++ b/libibverbs/verbs.h
@@ -739,7 +739,8 @@ enum ibv_wq_flags {
 	IBV_WQ_FLAGS_CVLAN_STRIPPING		= 1 << 0,
 	IBV_WQ_FLAGS_SCATTER_FCS		= 1 << 1,
 	IBV_WQ_FLAGS_DELAY_DROP			= 1 << 2,
-	IBV_WQ_FLAGS_RESERVED			= 1 << 3,
+	IBV_WQ_FLAGS_PCI_WRITE_END_PADDING	= 1 << 3,
+	IBV_WQ_FLAGS_RESERVED			= 1 << 4,
 };
 
 struct ibv_wq_init_attr {
@@ -848,6 +849,7 @@ enum ibv_qp_create_flags {
 	IBV_QP_CREATE_SCATTER_FCS		= 1 << 8,
 	IBV_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
 	IBV_QP_CREATE_SOURCE_QPN		= 1 << 10,
+	IBV_QP_CREATE_PCI_WRITE_END_PADDING	= 1 << 11,
 };
 
 struct ibv_rx_hash_conf {
-- 
1.8.3.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html