* [PATCH net-next 1/3] net/mlx4_en: Replace netdev parameter with priv in XDP xmit function
2017-10-11 10:17 [PATCH net-next 0/3] mlx4_en XDP TX improvements Tariq Toukan
@ 2017-10-11 10:17 ` Tariq Toukan
2017-10-11 10:17 ` [PATCH net-next 2/3] net/mlx4_en: Obsolete call to generic write_desc in XDP xmit flow Tariq Toukan
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Tariq Toukan @ 2017-10-11 10:17 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Eran Ben Elisha, kernel-team, Tariq Toukan
The struct net_device parameter was passed only to extract
struct mlx4_en_priv out of it.
Here we pass the priv parameter directly.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx4/en_rx.c | 2 +-
drivers/net/ethernet/mellanox/mlx4/en_tx.c | 3 +--
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +-
3 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
index a7866954d106..92aec17f4b4d 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
@@ -778,7 +778,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud
case XDP_PASS:
break;
case XDP_TX:
- if (likely(!mlx4_en_xmit_frame(ring, frags, dev,
+ if (likely(!mlx4_en_xmit_frame(ring, frags, priv,
length, cq_ring,
&doorbell_pending))) {
frags[0].page = NULL;
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index 2cc82dc07397..f16774c9c347 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
@@ -1087,10 +1087,9 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
struct mlx4_en_rx_alloc *frame,
- struct net_device *dev, unsigned int length,
+ struct mlx4_en_priv *priv, unsigned int length,
int tx_ind, bool *doorbell_pending)
{
- struct mlx4_en_priv *priv = netdev_priv(dev);
union mlx4_wqe_qpn_vlan qpn_vlan = {};
struct mlx4_en_tx_desc *tx_desc;
struct mlx4_en_tx_info *tx_info;
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 245e9ea09ab2..8cad9b4f1936 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -693,7 +693,7 @@ u16 mlx4_en_select_queue(struct net_device *dev, struct sk_buff *skb,
netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev);
netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
struct mlx4_en_rx_alloc *frame,
- struct net_device *dev, unsigned int length,
+ struct mlx4_en_priv *priv, unsigned int length,
int tx_ind, bool *doorbell_pending);
void mlx4_en_xmit_doorbell(struct mlx4_en_tx_ring *ring);
bool mlx4_en_rx_recycle(struct mlx4_en_rx_ring *ring,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH net-next 2/3] net/mlx4_en: Obsolete call to generic write_desc in XDP xmit flow
2017-10-11 10:17 [PATCH net-next 0/3] mlx4_en XDP TX improvements Tariq Toukan
2017-10-11 10:17 ` [PATCH net-next 1/3] net/mlx4_en: Replace netdev parameter with priv in XDP xmit function Tariq Toukan
@ 2017-10-11 10:17 ` Tariq Toukan
2017-10-11 10:17 ` [PATCH net-next 3/3] net/mlx4_en: XDP_TX, assign constant values of TX descs on ring creaion Tariq Toukan
2017-10-12 3:21 ` [PATCH net-next 0/3] mlx4_en XDP TX improvements David Miller
3 siblings, 0 replies; 5+ messages in thread
From: Tariq Toukan @ 2017-10-11 10:17 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Eran Ben Elisha, kernel-team, Tariq Toukan
Function mlx4_en_tx_write_desc() is not optimized to use of XDP xmit.
Use the relevant parts inline instead.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx4/en_tx.c | 17 ++++++++++++-----
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index f16774c9c347..ac7254e3f909 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
@@ -1090,7 +1090,9 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
struct mlx4_en_priv *priv, unsigned int length,
int tx_ind, bool *doorbell_pending)
{
- union mlx4_wqe_qpn_vlan qpn_vlan = {};
+ union mlx4_wqe_qpn_vlan qpn_vlan = {
+ .fence_size = MLX4_EN_XDP_TX_REAL_SZ,
+ };
struct mlx4_en_tx_desc *tx_desc;
struct mlx4_en_tx_info *tx_info;
struct mlx4_wqe_data_seg *data;
@@ -1140,7 +1142,6 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
data->byte_count = cpu_to_be32(length);
/* tx completion can avoid cache line miss for common cases */
- tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
op_own = cpu_to_be32(MLX4_OPCODE_SEND) |
((ring->prod & ring->size) ?
@@ -1151,10 +1152,16 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
ring->prod += MLX4_EN_XDP_TX_NRTXBB;
- qpn_vlan.fence_size = MLX4_EN_XDP_TX_REAL_SZ;
+ tx_desc->ctrl.qpn_vlan = qpn_vlan;
+ tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
+
+ /* Ensure new descriptor hits memory
+ * before setting ownership of this descriptor to HW
+ */
+ dma_wmb();
+ tx_desc->ctrl.owner_opcode = op_own;
+ ring->xmit_more++;
- mlx4_en_tx_write_desc(ring, tx_desc, qpn_vlan, TXBB_SIZE, 0,
- op_own, false, false);
*doorbell_pending = true;
return NETDEV_TX_OK;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH net-next 3/3] net/mlx4_en: XDP_TX, assign constant values of TX descs on ring creaion
2017-10-11 10:17 [PATCH net-next 0/3] mlx4_en XDP TX improvements Tariq Toukan
2017-10-11 10:17 ` [PATCH net-next 1/3] net/mlx4_en: Replace netdev parameter with priv in XDP xmit function Tariq Toukan
2017-10-11 10:17 ` [PATCH net-next 2/3] net/mlx4_en: Obsolete call to generic write_desc in XDP xmit flow Tariq Toukan
@ 2017-10-11 10:17 ` Tariq Toukan
2017-10-12 3:21 ` [PATCH net-next 0/3] mlx4_en XDP TX improvements David Miller
3 siblings, 0 replies; 5+ messages in thread
From: Tariq Toukan @ 2017-10-11 10:17 UTC (permalink / raw)
To: David S. Miller; +Cc: netdev, Eran Ben Elisha, kernel-team, Tariq Toukan
In XDP_TX, some fields in tx_info and tx_desc are constants across
all entries of the different XDP_TX rings.
Assign values to these fields on ring creation time, rather than in
data-path.
Patchset performance tests:
Tested on ConnectX3Pro, Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Single queue no-RSS optimization ON.
XDP_TX packet rate:
------------------------------
Before | After | Gain |
13.7 Mpps | 14.0 Mpps | %2.2 |
------------------------------
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 1 +
drivers/net/ethernet/mellanox/mlx4/en_tx.c | 38 ++++++++++++++++----------
drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 ++
3 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index e4c7a80ef5a8..d611df2f274d 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -1752,6 +1752,7 @@ int mlx4_en_start_port(struct net_device *dev)
mlx4_en_arm_cq(priv, cq);
} else {
+ mlx4_en_init_tx_xdp_ring_descs(priv, tx_ring);
mlx4_en_init_recycle_ring(priv, i);
/* XDP TX CQ should never be armed */
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index ac7254e3f909..596445a4a241 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
@@ -1085,14 +1085,35 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
#define MLX4_EN_XDP_TX_REAL_SZ (((CTRL_SIZE + MLX4_EN_XDP_TX_NRTXBB * DS_SIZE) \
/ 16) & 0x3f)
+void mlx4_en_init_tx_xdp_ring_descs(struct mlx4_en_priv *priv,
+ struct mlx4_en_tx_ring *ring)
+{
+ int i;
+
+ for (i = 0; i < ring->size; i++) {
+ struct mlx4_en_tx_info *tx_info = &ring->tx_info[i];
+ struct mlx4_en_tx_desc *tx_desc = ring->buf +
+ (i << LOG_TXBB_SIZE);
+
+ tx_info->map0_byte_count = PAGE_SIZE;
+ tx_info->nr_txbb = MLX4_EN_XDP_TX_NRTXBB;
+ tx_info->data_offset = offsetof(struct mlx4_en_tx_desc, data);
+ tx_info->ts_requested = 0;
+ tx_info->nr_maps = 1;
+ tx_info->linear = 1;
+ tx_info->inl = 0;
+
+ tx_desc->data.lkey = ring->mr_key;
+ tx_desc->ctrl.qpn_vlan.fence_size = MLX4_EN_XDP_TX_REAL_SZ;
+ tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
+ }
+}
+
netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
struct mlx4_en_rx_alloc *frame,
struct mlx4_en_priv *priv, unsigned int length,
int tx_ind, bool *doorbell_pending)
{
- union mlx4_wqe_qpn_vlan qpn_vlan = {
- .fence_size = MLX4_EN_XDP_TX_REAL_SZ,
- };
struct mlx4_en_tx_desc *tx_desc;
struct mlx4_en_tx_info *tx_info;
struct mlx4_wqe_data_seg *data;
@@ -1124,20 +1145,12 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
tx_info->page = frame->page;
frame->page = NULL;
tx_info->map0_dma = dma;
- tx_info->map0_byte_count = PAGE_SIZE;
- tx_info->nr_txbb = MLX4_EN_XDP_TX_NRTXBB;
tx_info->nr_bytes = max_t(unsigned int, length, ETH_ZLEN);
- tx_info->data_offset = offsetof(struct mlx4_en_tx_desc, data);
- tx_info->ts_requested = 0;
- tx_info->nr_maps = 1;
- tx_info->linear = 1;
- tx_info->inl = 0;
dma_sync_single_range_for_device(priv->ddev, dma, frame->page_offset,
length, PCI_DMA_TODEVICE);
data->addr = cpu_to_be64(dma + frame->page_offset);
- data->lkey = ring->mr_key;
dma_wmb();
data->byte_count = cpu_to_be32(length);
@@ -1152,9 +1165,6 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
ring->prod += MLX4_EN_XDP_TX_NRTXBB;
- tx_desc->ctrl.qpn_vlan = qpn_vlan;
- tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
-
/* Ensure new descriptor hits memory
* before setting ownership of this descriptor to HW
*/
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 8cad9b4f1936..1856e279a7e0 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -705,6 +705,8 @@ int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv,
int node, int queue_index);
void mlx4_en_destroy_tx_ring(struct mlx4_en_priv *priv,
struct mlx4_en_tx_ring **pring);
+void mlx4_en_init_tx_xdp_ring_descs(struct mlx4_en_priv *priv,
+ struct mlx4_en_tx_ring *ring);
int mlx4_en_activate_tx_ring(struct mlx4_en_priv *priv,
struct mlx4_en_tx_ring *ring,
int cq, int user_prio);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH net-next 0/3] mlx4_en XDP TX improvements
2017-10-11 10:17 [PATCH net-next 0/3] mlx4_en XDP TX improvements Tariq Toukan
` (2 preceding siblings ...)
2017-10-11 10:17 ` [PATCH net-next 3/3] net/mlx4_en: XDP_TX, assign constant values of TX descs on ring creaion Tariq Toukan
@ 2017-10-12 3:21 ` David Miller
3 siblings, 0 replies; 5+ messages in thread
From: David Miller @ 2017-10-12 3:21 UTC (permalink / raw)
To: tariqt; +Cc: netdev, eranbe, kernel-team
From: Tariq Toukan <tariqt@mellanox.com>
Date: Wed, 11 Oct 2017 13:17:24 +0300
> This patchset contains performance improvements
> to the XDP_TX use case in the mlx4 Eth driver.
>
> Patch 1 is a simple change in a function parameter type.
> Patch 2 replaces a call to a generic function with the
> relevant parts inlined.
> Patch 3 moves the write of descriptors' constant values
> from data path to control path.
>
> Series generated against net-next commit:
> 833e0e2f24fd net: dst: move cpu inside ifdef to avoid compilation warning
Series applied, thanks.
^ permalink raw reply [flat|nested] 5+ messages in thread