* [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support
@ 2016-04-28 22:36 Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 01/12] net/mlx5e: Call vxlan_get_rx_port() with rtnl lock Saeed Mahameed
` (12 more replies)
0 siblings, 13 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Saeed Mahameed
Hi Dave,
This series adds accelerated RFS support for the mlx5e driver.
I have added one patch non-related to aRFS that fixes the rtnl_lock
warning mlx5 driver been getting since b7aade15485a ('vxlan: break dependency with netdev drivers')
aRFS support in details:
A direct TIR per RQ is now required in order to have the essential building blocks
for aRFS. Today the driver has one direct TIR that forwards traffic to RQ[0] (core 0),
and one indirect TIR for RSS indirection table. For that we've added one direct TIR
per RQ, e.g.: TIR[i] -> RQ[i] (core i).
Publicize Modify flow rule destination and reveal it in flow steering API, to have the
ability to dynamically modify the destination TIR(core) for aRFS rules from the
ethernet driver.
Initializing CPU reverse mapping to notify upper layer on internal receive queue cpu
mappings.
Some design refactoring for mlx5e ethernet driver flow tables and flow steering API.
Now the caller of create_flow_table can choose the level of the flow table, this way
we will create the mlx5e flow tables in a reversed order and connect them as we go,
we create flow table[i+1] before flow table[i] to be able to set flow table[i + 1] as
a destination of flow table[i] once flow table[i] is created.
also we have split the main flow table in the following manner:
- From before: RX packet had to visit two flow tables until it is delivered to its receive queue:
RX packet -> vlan filter flow table -> main flow table.
> vlan filter will check the packet vlan field is allowed.
> main flow will check if the dest mac is allowed and will check the l3/l4 headers to
retrieve the RSS hash for steering the packet into its final receive queue.
- Now main flow table is split into l2 dst mac steering table and ttc (traffic type classifier) table:
RX packet -> vlan filter -> l2 table -> ttc table
> vlan filter - same as before
> L2 filter - filter packets according their destination mac address
> ttc table - classify packet headers for RSS steering
- L3/L4 classification rules to steer the packet according to thier headers hash
- in case of none of the rules applies the packet is steered to RQ[0]
After the above refactoring all left to-do is to create aRFS flow table which will manage
aRFS steering rules to forward traffic to the desired RQ (core) and just connect the ttc
table rules destinations to aRFS flow table.
aRFS flow table in case of a miss will deliver the traffic to the core where the original
ttc hash would have chosen.
TTC table is not initialized and enabled until the user explicitly asks to, i.e. setting the NETIF_F_NTUPLE
to ON. This way there is no need for ttc table to forward traffic to aRFS table unless required.
When setting back to OFF aRFS flow table is disabled and disconnected.
Thanks,
Saeed
Maor Gottlieb (10):
net/mlx5: Introduce modify flow rule destination
net/mlx5: Set number of allowed levels in priority
net/mlx5: Add user chosen levels when allocating flow tables
net/mlx5: Support different attributes for priorities in namespace
net/mlx5e: Refactor mlx5e flow steering structs
net/mlx5e: Split the main flow steering table
net/mlx5: Initializing CPU reverse mapping
net/mlx5e: Create aRFS flow tables
net/mlx5e: Add accelerated RFS support
net/mlx5e: Enabling aRFS mechanism
Matthew Finlay (1):
net/mlx5e: Call vxlan_get_rx_port() with rtnl lock
Tariq Toukan (1):
net/mlx5e: Direct TIR per RQ
drivers/infiniband/hw/mlx5/main.c | 3 +-
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 1 +
drivers/net/ethernet/mellanox/mlx5/core/en.h | 171 +++-
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 749 ++++++++++++++
.../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 24 +-
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 1060 +++++++++-----------
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 346 ++++---
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 46 +-
drivers/net/ethernet/mellanox/mlx5/core/en_tc.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 149 ++--
drivers/net/ethernet/mellanox/mlx5/core/fs_core.h | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/main.c | 18 +
include/linux/mlx5/driver.h | 3 +
include/linux/mlx5/fs.h | 9 +-
15 files changed, 1736 insertions(+), 849 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH net-next 01/12] net/mlx5e: Call vxlan_get_rx_port() with rtnl lock
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 02/12] net/mlx5e: Direct TIR per RQ Saeed Mahameed
` (11 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Matthew Finlay,
Saeed Mahameed
From: Matthew Finlay <matt@mellanox.com>
Hold the rtnl lock when calling vxlan_get_rx_port().
Fixes: b7aade15485a ("vxlan: break dependency with netdev drivers")
Signed-off-by: Matthew Finlay <matt@mellanox.com>
Reported-by: Alexander Duyck <alexander.duyck@gmail.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 8484ac4..8ffaf5c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2936,8 +2936,11 @@ static void *mlx5e_create_netdev(struct mlx5_core_dev *mdev)
goto err_tc_cleanup;
}
- if (mlx5e_vxlan_allowed(mdev))
+ if (mlx5e_vxlan_allowed(mdev)) {
+ rtnl_lock();
vxlan_get_rx_port(netdev);
+ rtnl_unlock();
+ }
mlx5e_enable_async_events(priv);
schedule_work(&priv->set_rx_mode_work);
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 02/12] net/mlx5e: Direct TIR per RQ
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 01/12] net/mlx5e: Call vxlan_get_rx_port() with rtnl lock Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 03/12] net/mlx5: Introduce modify flow rule destination Saeed Mahameed
` (10 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Tariq Toukan,
Saeed Mahameed
From: Tariq Toukan <tariqt@mellanox.com>
Introduce new TIRs for direct access per RQ.
Now we have 2 available kinds of TIRs:
- indirect TIR per traffic type, each points to one RQT (RSS RQT)
same as before.
- New direct TIR per RQ, each points to RQT with a size of one
that forwards packets to that RQ only.
Driver will open max channels (num cores) direct TIRs by default,
they will be filled with the actual RQs once channels are allocated.
Needed for downstream aRFS and ethtool direct steering functionalities.
Signed-off-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 21 +-
.../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 9 +-
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 4 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 295 ++++++++++++--------
4 files changed, 191 insertions(+), 138 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index bbc01a4..5c8e98c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -385,14 +385,7 @@ enum mlx5e_traffic_types {
MLX5E_TT_IPV6,
MLX5E_TT_ANY,
MLX5E_NUM_TT,
-};
-
-#define IS_HASHING_TT(tt) (tt != MLX5E_TT_ANY)
-
-enum mlx5e_rqt_ix {
- MLX5E_INDIRECTION_RQT,
- MLX5E_SINGLE_RQ_RQT,
- MLX5E_NUM_RQT,
+ MLX5E_NUM_INDIR_TIRS = MLX5E_TT_ANY,
};
struct mlx5e_eth_addr_info {
@@ -453,6 +446,11 @@ struct mlx5e_flow_tables {
struct mlx5e_flow_table main;
};
+struct mlx5e_direct_tir {
+ u32 tirn;
+ u32 rqtn;
+};
+
struct mlx5e_priv {
/* priv data path fields - start */
struct mlx5e_sq **txq_to_sq_map;
@@ -470,8 +468,9 @@ struct mlx5e_priv {
struct mlx5e_channel **channel;
u32 tisn[MLX5E_MAX_NUM_TC];
- u32 rqtn[MLX5E_NUM_RQT];
- u32 tirn[MLX5E_NUM_TT];
+ u32 indir_rqtn;
+ u32 indir_tirn[MLX5E_NUM_INDIR_TIRS];
+ struct mlx5e_direct_tir direct_tir[MLX5E_MAX_NUM_CHANNELS];
struct mlx5e_flow_tables fts;
struct mlx5e_eth_addr_db eth_addr;
@@ -578,7 +577,7 @@ void mlx5e_disable_vlan_filter(struct mlx5e_priv *priv);
int mlx5e_modify_rqs_vsd(struct mlx5e_priv *priv, bool vsd);
-int mlx5e_redirect_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix);
+int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz, int ix);
void mlx5e_build_tir_ctx_hash(void *tirc, struct mlx5e_priv *priv);
int mlx5e_open_locked(struct net_device *netdev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index a06958a..498d407 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -826,9 +826,8 @@ static void mlx5e_modify_tirs_hash(struct mlx5e_priv *priv, void *in, int inlen)
MLX5_SET(modify_tir_in, in, bitmask.hash, 1);
mlx5e_build_tir_ctx_hash(tirc, priv);
- for (i = 0; i < MLX5E_NUM_TT; i++)
- if (IS_HASHING_TT(i))
- mlx5_core_modify_tir(mdev, priv->tirn[i], in, inlen);
+ for (i = 0; i < MLX5E_NUM_INDIR_TIRS; i++)
+ mlx5_core_modify_tir(mdev, priv->indir_tirn[i], in, inlen);
}
static int mlx5e_set_rxfh(struct net_device *dev, const u32 *indir,
@@ -850,9 +849,11 @@ static int mlx5e_set_rxfh(struct net_device *dev, const u32 *indir,
mutex_lock(&priv->state_lock);
if (indir) {
+ u32 rqtn = priv->indir_rqtn;
+
memcpy(priv->params.indirection_rqt, indir,
sizeof(priv->params.indirection_rqt));
- mlx5e_redirect_rqt(priv, MLX5E_INDIRECTION_RQT);
+ mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, 0);
}
if (key)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index d00a242..4df49e6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -247,7 +247,7 @@ static int __mlx5e_add_eth_addr_rule(struct mlx5e_priv *priv,
outer_headers.dmac_47_16);
u8 *mv_dmac = MLX5_ADDR_OF(fte_match_param, mv,
outer_headers.dmac_47_16);
- u32 *tirn = priv->tirn;
+ u32 *tirn = priv->indir_tirn;
u32 tt_vec;
int err = 0;
@@ -274,7 +274,7 @@ static int __mlx5e_add_eth_addr_rule(struct mlx5e_priv *priv,
if (tt_vec & BIT(MLX5E_TT_ANY)) {
rule_p = &ai->ft_rule[MLX5E_TT_ANY];
- dest.tir_num = tirn[MLX5E_TT_ANY];
+ dest.tir_num = priv->direct_tir[0].tirn;
*rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
MLX5_FS_DEFAULT_FLOW_TAG, &dest);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 8ffaf5c..99a1bff 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1338,48 +1338,36 @@ static void mlx5e_fill_indir_rqt_rqns(struct mlx5e_priv *priv, void *rqtc)
for (i = 0; i < MLX5E_INDIR_RQT_SIZE; i++) {
int ix = i;
+ u32 rqn;
if (priv->params.rss_hfunc == ETH_RSS_HASH_XOR)
ix = mlx5e_bits_invert(i, MLX5E_LOG_INDIR_RQT_SIZE);
ix = priv->params.indirection_rqt[ix];
- MLX5_SET(rqtc, rqtc, rq_num[i],
- test_bit(MLX5E_STATE_OPENED, &priv->state) ?
- priv->channel[ix]->rq.rqn :
- priv->drop_rq.rqn);
+ rqn = test_bit(MLX5E_STATE_OPENED, &priv->state) ?
+ priv->channel[ix]->rq.rqn :
+ priv->drop_rq.rqn;
+ MLX5_SET(rqtc, rqtc, rq_num[i], rqn);
}
}
-static void mlx5e_fill_rqt_rqns(struct mlx5e_priv *priv, void *rqtc,
- enum mlx5e_rqt_ix rqt_ix)
+static void mlx5e_fill_direct_rqt_rqn(struct mlx5e_priv *priv, void *rqtc,
+ int ix)
{
+ u32 rqn = test_bit(MLX5E_STATE_OPENED, &priv->state) ?
+ priv->channel[ix]->rq.rqn :
+ priv->drop_rq.rqn;
- switch (rqt_ix) {
- case MLX5E_INDIRECTION_RQT:
- mlx5e_fill_indir_rqt_rqns(priv, rqtc);
-
- break;
-
- default: /* MLX5E_SINGLE_RQ_RQT */
- MLX5_SET(rqtc, rqtc, rq_num[0],
- test_bit(MLX5E_STATE_OPENED, &priv->state) ?
- priv->channel[0]->rq.rqn :
- priv->drop_rq.rqn);
-
- break;
- }
+ MLX5_SET(rqtc, rqtc, rq_num[0], rqn);
}
-static int mlx5e_create_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix)
+static int mlx5e_create_rqt(struct mlx5e_priv *priv, int sz, int ix, u32 *rqtn)
{
struct mlx5_core_dev *mdev = priv->mdev;
- u32 *in;
void *rqtc;
int inlen;
- int sz;
int err;
-
- sz = (rqt_ix == MLX5E_SINGLE_RQ_RQT) ? 1 : MLX5E_INDIR_RQT_SIZE;
+ u32 *in;
inlen = MLX5_ST_SZ_BYTES(create_rqt_in) + sizeof(u32) * sz;
in = mlx5_vzalloc(inlen);
@@ -1391,26 +1379,73 @@ static int mlx5e_create_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix)
MLX5_SET(rqtc, rqtc, rqt_actual_size, sz);
MLX5_SET(rqtc, rqtc, rqt_max_size, sz);
- mlx5e_fill_rqt_rqns(priv, rqtc, rqt_ix);
+ if (sz > 1) /* RSS */
+ mlx5e_fill_indir_rqt_rqns(priv, rqtc);
+ else
+ mlx5e_fill_direct_rqt_rqn(priv, rqtc, ix);
- err = mlx5_core_create_rqt(mdev, in, inlen, &priv->rqtn[rqt_ix]);
+ err = mlx5_core_create_rqt(mdev, in, inlen, rqtn);
kvfree(in);
+ return err;
+}
+
+static void mlx5e_destroy_rqt(struct mlx5e_priv *priv, u32 rqtn)
+{
+ mlx5_core_destroy_rqt(priv->mdev, rqtn);
+}
+
+static int mlx5e_create_rqts(struct mlx5e_priv *priv)
+{
+ int nch = mlx5e_get_max_num_channels(priv->mdev);
+ u32 *rqtn;
+ int err;
+ int ix;
+
+ /* Indirect RQT */
+ rqtn = &priv->indir_rqtn;
+ err = mlx5e_create_rqt(priv, MLX5E_INDIR_RQT_SIZE, 0, rqtn);
+ if (err)
+ return err;
+
+ /* Direct RQTs */
+ for (ix = 0; ix < nch; ix++) {
+ rqtn = &priv->direct_tir[ix].rqtn;
+ err = mlx5e_create_rqt(priv, 1 /*size */, ix, rqtn);
+ if (err)
+ goto err_destroy_rqts;
+ }
+
+ return 0;
+
+err_destroy_rqts:
+ for (ix--; ix >= 0; ix--)
+ mlx5e_destroy_rqt(priv, priv->direct_tir[ix].rqtn);
+
+ mlx5e_destroy_rqt(priv, priv->indir_rqtn);
return err;
}
-int mlx5e_redirect_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix)
+static void mlx5e_destroy_rqts(struct mlx5e_priv *priv)
+{
+ int nch = mlx5e_get_max_num_channels(priv->mdev);
+ int i;
+
+ for (i = 0; i < nch; i++)
+ mlx5e_destroy_rqt(priv, priv->direct_tir[i].rqtn);
+
+ mlx5e_destroy_rqt(priv, priv->indir_rqtn);
+}
+
+int mlx5e_redirect_rqt(struct mlx5e_priv *priv, u32 rqtn, int sz, int ix)
{
struct mlx5_core_dev *mdev = priv->mdev;
- u32 *in;
void *rqtc;
int inlen;
- int sz;
+ u32 *in;
int err;
- sz = (rqt_ix == MLX5E_SINGLE_RQ_RQT) ? 1 : MLX5E_INDIR_RQT_SIZE;
-
inlen = MLX5_ST_SZ_BYTES(modify_rqt_in) + sizeof(u32) * sz;
in = mlx5_vzalloc(inlen);
if (!in)
@@ -1419,27 +1454,31 @@ int mlx5e_redirect_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix)
rqtc = MLX5_ADDR_OF(modify_rqt_in, in, ctx);
MLX5_SET(rqtc, rqtc, rqt_actual_size, sz);
-
- mlx5e_fill_rqt_rqns(priv, rqtc, rqt_ix);
+ if (sz > 1) /* RSS */
+ mlx5e_fill_indir_rqt_rqns(priv, rqtc);
+ else
+ mlx5e_fill_direct_rqt_rqn(priv, rqtc, ix);
MLX5_SET(modify_rqt_in, in, bitmask.rqn_list, 1);
- err = mlx5_core_modify_rqt(mdev, priv->rqtn[rqt_ix], in, inlen);
+ err = mlx5_core_modify_rqt(mdev, rqtn, in, inlen);
kvfree(in);
return err;
}
-static void mlx5e_destroy_rqt(struct mlx5e_priv *priv, enum mlx5e_rqt_ix rqt_ix)
-{
- mlx5_core_destroy_rqt(priv->mdev, priv->rqtn[rqt_ix]);
-}
-
static void mlx5e_redirect_rqts(struct mlx5e_priv *priv)
{
- mlx5e_redirect_rqt(priv, MLX5E_INDIRECTION_RQT);
- mlx5e_redirect_rqt(priv, MLX5E_SINGLE_RQ_RQT);
+ u32 rqtn;
+ int ix;
+
+ rqtn = priv->indir_rqtn;
+ mlx5e_redirect_rqt(priv, rqtn, MLX5E_INDIR_RQT_SIZE, 0);
+ for (ix = 0; ix < priv->params.num_channels; ix++) {
+ rqtn = priv->direct_tir[ix].rqtn;
+ mlx5e_redirect_rqt(priv, rqtn, 1, ix);
+ }
}
static void mlx5e_build_tir_ctx_lro(void *tirc, struct mlx5e_priv *priv)
@@ -1484,6 +1523,7 @@ static int mlx5e_modify_tirs_lro(struct mlx5e_priv *priv)
int inlen;
int err;
int tt;
+ int ix;
inlen = MLX5_ST_SZ_BYTES(modify_tir_in);
in = mlx5_vzalloc(inlen);
@@ -1495,23 +1535,32 @@ static int mlx5e_modify_tirs_lro(struct mlx5e_priv *priv)
mlx5e_build_tir_ctx_lro(tirc, priv);
- for (tt = 0; tt < MLX5E_NUM_TT; tt++) {
- err = mlx5_core_modify_tir(mdev, priv->tirn[tt], in, inlen);
+ for (tt = 0; tt < MLX5E_NUM_INDIR_TIRS; tt++) {
+ err = mlx5_core_modify_tir(mdev, priv->indir_tirn[tt], in,
+ inlen);
if (err)
- break;
+ goto free_in;
}
+ for (ix = 0; ix < mlx5e_get_max_num_channels(mdev); ix++) {
+ err = mlx5_core_modify_tir(mdev, priv->direct_tir[ix].tirn,
+ in, inlen);
+ if (err)
+ goto free_in;
+ }
+
+free_in:
kvfree(in);
return err;
}
-static int mlx5e_refresh_tir_self_loopback_enable(struct mlx5_core_dev *mdev,
- u32 tirn)
+static int mlx5e_refresh_tirs_self_loopback_enable(struct mlx5e_priv *priv)
{
void *in;
int inlen;
int err;
+ int i;
inlen = MLX5_ST_SZ_BYTES(modify_tir_in);
in = mlx5_vzalloc(inlen);
@@ -1520,25 +1569,23 @@ static int mlx5e_refresh_tir_self_loopback_enable(struct mlx5_core_dev *mdev,
MLX5_SET(modify_tir_in, in, bitmask.self_lb_en, 1);
- err = mlx5_core_modify_tir(mdev, tirn, in, inlen);
-
- kvfree(in);
-
- return err;
-}
-
-static int mlx5e_refresh_tirs_self_loopback_enable(struct mlx5e_priv *priv)
-{
- int err;
- int i;
+ for (i = 0; i < MLX5E_NUM_INDIR_TIRS; i++) {
+ err = mlx5_core_modify_tir(priv->mdev, priv->indir_tirn[i], in,
+ inlen);
+ if (err)
+ return err;
+ }
- for (i = 0; i < MLX5E_NUM_TT; i++) {
- err = mlx5e_refresh_tir_self_loopback_enable(priv->mdev,
- priv->tirn[i]);
+ for (i = 0; i < priv->params.num_channels; i++) {
+ err = mlx5_core_modify_tir(priv->mdev,
+ priv->direct_tir[i].tirn, in,
+ inlen);
if (err)
return err;
}
+ kvfree(in);
+
return 0;
}
@@ -1849,7 +1896,8 @@ static void mlx5e_destroy_tises(struct mlx5e_priv *priv)
mlx5e_destroy_tis(priv, tc);
}
-static void mlx5e_build_tir_ctx(struct mlx5e_priv *priv, u32 *tirc, int tt)
+static void mlx5e_build_indir_tir_ctx(struct mlx5e_priv *priv, u32 *tirc,
+ enum mlx5e_traffic_types tt)
{
void *hfso = MLX5_ADDR_OF(tirc, tirc, rx_hash_field_selector_outer);
@@ -1870,19 +1918,8 @@ static void mlx5e_build_tir_ctx(struct mlx5e_priv *priv, u32 *tirc, int tt)
mlx5e_build_tir_ctx_lro(tirc, priv);
MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT);
-
- switch (tt) {
- case MLX5E_TT_ANY:
- MLX5_SET(tirc, tirc, indirect_table,
- priv->rqtn[MLX5E_SINGLE_RQ_RQT]);
- MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
- break;
- default:
- MLX5_SET(tirc, tirc, indirect_table,
- priv->rqtn[MLX5E_INDIRECTION_RQT]);
- mlx5e_build_tir_ctx_hash(tirc, priv);
- break;
- }
+ MLX5_SET(tirc, tirc, indirect_table, priv->indir_rqtn);
+ mlx5e_build_tir_ctx_hash(tirc, priv);
switch (tt) {
case MLX5E_TT_IPV4_TCP:
@@ -1962,64 +1999,90 @@ static void mlx5e_build_tir_ctx(struct mlx5e_priv *priv, u32 *tirc, int tt)
MLX5_SET(rx_hash_field_select, hfso, selected_fields,
MLX5_HASH_IP);
break;
+ default:
+ WARN_ONCE(true,
+ "mlx5e_build_indir_tir_ctx: bad traffic type!\n");
}
}
-static int mlx5e_create_tir(struct mlx5e_priv *priv, int tt)
+static void mlx5e_build_direct_tir_ctx(struct mlx5e_priv *priv, u32 *tirc,
+ u32 rqtn)
{
- struct mlx5_core_dev *mdev = priv->mdev;
- u32 *in;
+ MLX5_SET(tirc, tirc, transport_domain, priv->tdn);
+
+ mlx5e_build_tir_ctx_lro(tirc, priv);
+
+ MLX5_SET(tirc, tirc, disp_type, MLX5_TIRC_DISP_TYPE_INDIRECT);
+ MLX5_SET(tirc, tirc, indirect_table, rqtn);
+ MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FN_INVERTED_XOR8);
+}
+
+static int mlx5e_create_tirs(struct mlx5e_priv *priv)
+{
+ int nch = mlx5e_get_max_num_channels(priv->mdev);
void *tirc;
int inlen;
+ u32 *tirn;
int err;
+ u32 *in;
+ int ix;
+ int tt;
inlen = MLX5_ST_SZ_BYTES(create_tir_in);
in = mlx5_vzalloc(inlen);
if (!in)
return -ENOMEM;
- tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
-
- mlx5e_build_tir_ctx(priv, tirc, tt);
-
- err = mlx5_core_create_tir(mdev, in, inlen, &priv->tirn[tt]);
-
- kvfree(in);
-
- return err;
-}
-
-static void mlx5e_destroy_tir(struct mlx5e_priv *priv, int tt)
-{
- mlx5_core_destroy_tir(priv->mdev, priv->tirn[tt]);
-}
-
-static int mlx5e_create_tirs(struct mlx5e_priv *priv)
-{
- int err;
- int i;
-
- for (i = 0; i < MLX5E_NUM_TT; i++) {
- err = mlx5e_create_tir(priv, i);
+ /* indirect tirs */
+ for (tt = 0; tt < MLX5E_NUM_INDIR_TIRS; tt++) {
+ memset(in, 0, inlen);
+ tirn = &priv->indir_tirn[tt];
+ tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
+ mlx5e_build_indir_tir_ctx(priv, tirc, tt);
+ err = mlx5_core_create_tir(priv->mdev, in, inlen, tirn);
if (err)
goto err_destroy_tirs;
}
+ /* direct tirs */
+ for (ix = 0; ix < nch; ix++) {
+ memset(in, 0, inlen);
+ tirn = &priv->direct_tir[ix].tirn;
+ tirc = MLX5_ADDR_OF(create_tir_in, in, ctx);
+ mlx5e_build_direct_tir_ctx(priv, tirc,
+ priv->direct_tir[ix].rqtn);
+ err = mlx5_core_create_tir(priv->mdev, in, inlen, tirn);
+ if (err)
+ goto err_destroy_ch_tirs;
+ }
+
+ kvfree(in);
+
return 0;
+err_destroy_ch_tirs:
+ for (ix--; ix >= 0; ix--)
+ mlx5_core_destroy_tir(priv->mdev, priv->direct_tir[ix].tirn);
+
err_destroy_tirs:
- for (i--; i >= 0; i--)
- mlx5e_destroy_tir(priv, i);
+ for (tt--; tt >= 0; tt--)
+ mlx5_core_destroy_tir(priv->mdev, priv->indir_tirn[tt]);
+
+ kvfree(in);
return err;
}
static void mlx5e_destroy_tirs(struct mlx5e_priv *priv)
{
+ int nch = mlx5e_get_max_num_channels(priv->mdev);
int i;
- for (i = 0; i < MLX5E_NUM_TT; i++)
- mlx5e_destroy_tir(priv, i);
+ for (i = 0; i < nch; i++)
+ mlx5_core_destroy_tir(priv->mdev, priv->direct_tir[i].tirn);
+
+ for (i = 0; i < MLX5E_NUM_INDIR_TIRS; i++)
+ mlx5_core_destroy_tir(priv->mdev, priv->indir_tirn[i]);
}
int mlx5e_modify_rqs_vsd(struct mlx5e_priv *priv, bool vsd)
@@ -2892,22 +2955,16 @@ static void *mlx5e_create_netdev(struct mlx5_core_dev *mdev)
goto err_destroy_tises;
}
- err = mlx5e_create_rqt(priv, MLX5E_INDIRECTION_RQT);
+ err = mlx5e_create_rqts(priv);
if (err) {
- mlx5_core_warn(mdev, "create rqt(INDIR) failed, %d\n", err);
+ mlx5_core_warn(mdev, "create rqts failed, %d\n", err);
goto err_close_drop_rq;
}
- err = mlx5e_create_rqt(priv, MLX5E_SINGLE_RQ_RQT);
- if (err) {
- mlx5_core_warn(mdev, "create rqt(SINGLE) failed, %d\n", err);
- goto err_destroy_rqt_indir;
- }
-
err = mlx5e_create_tirs(priv);
if (err) {
mlx5_core_warn(mdev, "create tirs failed, %d\n", err);
- goto err_destroy_rqt_single;
+ goto err_destroy_rqts;
}
err = mlx5e_create_flow_tables(priv);
@@ -2957,11 +3014,8 @@ err_dealloc_q_counters:
err_destroy_tirs:
mlx5e_destroy_tirs(priv);
-err_destroy_rqt_single:
- mlx5e_destroy_rqt(priv, MLX5E_SINGLE_RQ_RQT);
-
-err_destroy_rqt_indir:
- mlx5e_destroy_rqt(priv, MLX5E_INDIRECTION_RQT);
+err_destroy_rqts:
+ mlx5e_destroy_rqts(priv);
err_close_drop_rq:
mlx5e_close_drop_rq(priv);
@@ -3015,8 +3069,7 @@ static void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, void *vpriv)
mlx5e_destroy_q_counter(priv);
mlx5e_destroy_flow_tables(priv);
mlx5e_destroy_tirs(priv);
- mlx5e_destroy_rqt(priv, MLX5E_SINGLE_RQ_RQT);
- mlx5e_destroy_rqt(priv, MLX5E_INDIRECTION_RQT);
+ mlx5e_destroy_rqts(priv);
mlx5e_close_drop_rq(priv);
mlx5e_destroy_tises(priv);
mlx5_core_destroy_mkey(priv->mdev, &priv->umr_mkey);
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 03/12] net/mlx5: Introduce modify flow rule destination
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 01/12] net/mlx5e: Call vxlan_get_rx_port() with rtnl lock Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 02/12] net/mlx5e: Direct TIR per RQ Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 04/12] net/mlx5: Set number of allowed levels in priority Saeed Mahameed
` (9 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
This API is used for modifying the flow rule destination.
This is needed for modifying the pointed flow table by the
traffic type classifier rules to point on the aRFS tables.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 4 ++--
include/linux/mlx5/fs.h | 3 +++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index 89cce97..bb2c1cd 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -615,8 +615,8 @@ static int update_root_ft_create(struct mlx5_flow_table *ft, struct fs_prio
return err;
}
-static int mlx5_modify_rule_destination(struct mlx5_flow_rule *rule,
- struct mlx5_flow_destination *dest)
+int mlx5_modify_rule_destination(struct mlx5_flow_rule *rule,
+ struct mlx5_flow_destination *dest)
{
struct mlx5_flow_table *ft;
struct mlx5_flow_group *fg;
diff --git a/include/linux/mlx5/fs.h b/include/linux/mlx5/fs.h
index 8dec550..28a5b66 100644
--- a/include/linux/mlx5/fs.h
+++ b/include/linux/mlx5/fs.h
@@ -113,4 +113,7 @@ mlx5_add_flow_rule(struct mlx5_flow_table *ft,
struct mlx5_flow_destination *dest);
void mlx5_del_flow_rule(struct mlx5_flow_rule *fr);
+int mlx5_modify_rule_destination(struct mlx5_flow_rule *rule,
+ struct mlx5_flow_destination *dest);
+
#endif
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 04/12] net/mlx5: Set number of allowed levels in priority
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (2 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 03/12] net/mlx5: Introduce modify flow rule destination Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 05/12] net/mlx5: Add user chosen levels when allocating flow tables Saeed Mahameed
` (8 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Refactors the flow steering namespace creation,
by changing the name num_fts to num_levels.
When new flow table is created, the driver assign new level
to this flow table therefore the meaning is equivalent.
Since downstream patches will introduce the ability to create more
than one flow table per level, the name num_fts is no
longer accurate.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 61 +++++++++++----------
drivers/net/ethernet/mellanox/mlx5/core/fs_core.h | 2 +-
2 files changed, 33 insertions(+), 30 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index bb2c1cd..cfb35c3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -40,18 +40,18 @@
#define INIT_TREE_NODE_ARRAY_SIZE(...) (sizeof((struct init_tree_node[]){__VA_ARGS__}) /\
sizeof(struct init_tree_node))
-#define ADD_PRIO(num_prios_val, min_level_val, max_ft_val, caps_val,\
+#define ADD_PRIO(num_prios_val, min_level_val, num_levels_val, caps_val,\
...) {.type = FS_TYPE_PRIO,\
.min_ft_level = min_level_val,\
- .max_ft = max_ft_val,\
+ .num_levels = num_levels_val,\
.num_leaf_prios = num_prios_val,\
.caps = caps_val,\
.children = (struct init_tree_node[]) {__VA_ARGS__},\
.ar_size = INIT_TREE_NODE_ARRAY_SIZE(__VA_ARGS__) \
}
-#define ADD_MULTIPLE_PRIO(num_prios_val, max_ft_val, ...)\
- ADD_PRIO(num_prios_val, 0, max_ft_val, {},\
+#define ADD_MULTIPLE_PRIO(num_prios_val, num_levels_val, ...)\
+ ADD_PRIO(num_prios_val, 0, num_levels_val, {},\
__VA_ARGS__)\
#define ADD_NS(...) {.type = FS_TYPE_NAMESPACE,\
@@ -67,17 +67,18 @@
#define FS_REQUIRED_CAPS(...) {.arr_sz = INIT_CAPS_ARRAY_SIZE(__VA_ARGS__), \
.caps = (long[]) {__VA_ARGS__} }
-#define LEFTOVERS_MAX_FT 1
+#define LEFTOVERS_NUM_LEVELS 1
#define LEFTOVERS_NUM_PRIOS 1
-#define BY_PASS_PRIO_MAX_FT 1
-#define BY_PASS_MIN_LEVEL (KENREL_MIN_LEVEL + MLX5_BY_PASS_NUM_PRIOS +\
- LEFTOVERS_MAX_FT)
-#define KERNEL_MAX_FT 3
+#define BY_PASS_PRIO_NUM_LEVELS 1
+#define BY_PASS_MIN_LEVEL (KERNEL_MIN_LEVEL + MLX5_BY_PASS_NUM_PRIOS +\
+ LEFTOVERS_NUM_PRIOS)
+
+#define KERNEL_NUM_LEVELS 3
#define KERNEL_NUM_PRIOS 2
-#define KENREL_MIN_LEVEL 2
+#define KERNEL_MIN_LEVEL 2
-#define ANCHOR_MAX_FT 1
+#define ANCHOR_NUM_LEVELS 1
#define ANCHOR_NUM_PRIOS 1
#define ANCHOR_MIN_LEVEL (BY_PASS_MIN_LEVEL + 1)
struct node_caps {
@@ -92,7 +93,7 @@ static struct init_tree_node {
int min_ft_level;
int num_leaf_prios;
int prio;
- int max_ft;
+ int num_levels;
} root_fs = {
.type = FS_TYPE_NAMESPACE,
.ar_size = 4,
@@ -102,17 +103,19 @@ static struct init_tree_node {
FS_CAP(flow_table_properties_nic_receive.modify_root),
FS_CAP(flow_table_properties_nic_receive.identified_miss_table_mode),
FS_CAP(flow_table_properties_nic_receive.flow_table_modify)),
- ADD_NS(ADD_MULTIPLE_PRIO(MLX5_BY_PASS_NUM_PRIOS, BY_PASS_PRIO_MAX_FT))),
- ADD_PRIO(0, KENREL_MIN_LEVEL, 0, {},
- ADD_NS(ADD_MULTIPLE_PRIO(KERNEL_NUM_PRIOS, KERNEL_MAX_FT))),
+ ADD_NS(ADD_MULTIPLE_PRIO(MLX5_BY_PASS_NUM_PRIOS,
+ BY_PASS_PRIO_NUM_LEVELS))),
+ ADD_PRIO(0, KERNEL_MIN_LEVEL, 0, {},
+ ADD_NS(ADD_MULTIPLE_PRIO(KERNEL_NUM_PRIOS,
+ KERNEL_NUM_LEVELS))),
ADD_PRIO(0, BY_PASS_MIN_LEVEL, 0,
FS_REQUIRED_CAPS(FS_CAP(flow_table_properties_nic_receive.flow_modify_en),
FS_CAP(flow_table_properties_nic_receive.modify_root),
FS_CAP(flow_table_properties_nic_receive.identified_miss_table_mode),
FS_CAP(flow_table_properties_nic_receive.flow_table_modify)),
- ADD_NS(ADD_MULTIPLE_PRIO(LEFTOVERS_NUM_PRIOS, LEFTOVERS_MAX_FT))),
+ ADD_NS(ADD_MULTIPLE_PRIO(LEFTOVERS_NUM_PRIOS, LEFTOVERS_NUM_LEVELS))),
ADD_PRIO(0, ANCHOR_MIN_LEVEL, 0, {},
- ADD_NS(ADD_MULTIPLE_PRIO(ANCHOR_NUM_PRIOS, ANCHOR_MAX_FT))),
+ ADD_NS(ADD_MULTIPLE_PRIO(ANCHOR_NUM_PRIOS, ANCHOR_NUM_LEVELS))),
}
};
@@ -716,7 +719,7 @@ struct mlx5_flow_table *mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
err = -EINVAL;
goto unlock_root;
}
- if (fs_prio->num_ft == fs_prio->max_ft) {
+ if (fs_prio->num_ft == fs_prio->num_levels) {
err = -ENOSPC;
goto unlock_root;
}
@@ -1311,7 +1314,7 @@ struct mlx5_flow_namespace *mlx5_get_flow_namespace(struct mlx5_core_dev *dev,
EXPORT_SYMBOL(mlx5_get_flow_namespace);
static struct fs_prio *fs_create_prio(struct mlx5_flow_namespace *ns,
- unsigned prio, int max_ft)
+ unsigned int prio, int num_levels)
{
struct fs_prio *fs_prio;
@@ -1322,7 +1325,7 @@ static struct fs_prio *fs_create_prio(struct mlx5_flow_namespace *ns,
fs_prio->node.type = FS_TYPE_PRIO;
tree_init_node(&fs_prio->node, 1, NULL);
tree_add_node(&fs_prio->node, &ns->node);
- fs_prio->max_ft = max_ft;
+ fs_prio->num_levels = num_levels;
fs_prio->prio = prio;
list_add_tail(&fs_prio->node.list, &ns->node.children);
@@ -1360,7 +1363,7 @@ static int create_leaf_prios(struct mlx5_flow_namespace *ns, struct init_tree_no
int i;
for (i = 0; i < prio_metadata->num_leaf_prios; i++) {
- fs_prio = fs_create_prio(ns, i, prio_metadata->max_ft);
+ fs_prio = fs_create_prio(ns, i, prio_metadata->num_levels);
if (IS_ERR(fs_prio))
return PTR_ERR(fs_prio);
}
@@ -1406,7 +1409,7 @@ static int init_root_tree_recursive(struct mlx5_core_dev *dev,
fs_get_obj(fs_ns, fs_parent_node);
if (init_node->num_leaf_prios)
return create_leaf_prios(fs_ns, init_node);
- fs_prio = fs_create_prio(fs_ns, index, init_node->max_ft);
+ fs_prio = fs_create_prio(fs_ns, index, init_node->num_levels);
if (IS_ERR(fs_prio))
return PTR_ERR(fs_prio);
base = &fs_prio->node;
@@ -1479,9 +1482,9 @@ static int set_prio_attrs_in_ns(struct mlx5_flow_namespace *ns, int acc_level)
struct fs_prio *prio;
fs_for_each_prio(prio, ns) {
- /* This updates prio start_level and max_ft */
+ /* This updates prio start_level and num_levels */
set_prio_attrs_in_prio(prio, acc_level);
- acc_level += prio->max_ft;
+ acc_level += prio->num_levels;
}
return acc_level;
}
@@ -1493,11 +1496,11 @@ static void set_prio_attrs_in_prio(struct fs_prio *prio, int acc_level)
prio->start_level = acc_level;
fs_for_each_ns(ns, prio)
- /* This updates start_level and max_ft of ns's priority descendants */
+ /* This updates start_level and num_levels of ns's priority descendants */
acc_level_ns = set_prio_attrs_in_ns(ns, acc_level);
- if (!prio->max_ft)
- prio->max_ft = acc_level_ns - prio->start_level;
- WARN_ON(prio->max_ft < acc_level_ns - prio->start_level);
+ if (!prio->num_levels)
+ prio->num_levels = acc_level_ns - prio->start_level;
+ WARN_ON(prio->num_levels < acc_level_ns - prio->start_level);
}
static void set_prio_attrs(struct mlx5_flow_root_namespace *root_ns)
@@ -1508,7 +1511,7 @@ static void set_prio_attrs(struct mlx5_flow_root_namespace *root_ns)
fs_for_each_prio(prio, ns) {
set_prio_attrs_in_prio(prio, start_level);
- start_level += prio->max_ft;
+ start_level += prio->num_levels;
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
index f37a624..d607e56 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.h
@@ -107,7 +107,7 @@ struct fs_fte {
/* Type of children is mlx5_flow_table/namespace */
struct fs_prio {
struct fs_node node;
- unsigned int max_ft;
+ unsigned int num_levels;
unsigned int start_level;
unsigned int prio;
unsigned int num_ft;
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 05/12] net/mlx5: Add user chosen levels when allocating flow tables
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (3 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 04/12] net/mlx5: Set number of allowed levels in priority Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 06/12] net/mlx5: Support different attributes for priorities in namespace Saeed Mahameed
` (7 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Currently, consumers of the flow steering infrastructure can't
choose their own flow table levels and are limited to one
flow table per level. This just waste levels.
Instead, we introduce here the possibility to use multiple
flow tables in a level. The user is free to connect these
flow tables, while following the rule (FTEs in FT of level x
could only point to FTs of level y where y > x).
In addition this patch switch the order of the create/destroy
flow tables of the NIC(vlan and main).
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/infiniband/hw/mlx5/main.c | 3 +-
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 30 ++++++----
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 3 +-
drivers/net/ethernet/mellanox/mlx5/core/eswitch.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 66 +++++++++++++-------
include/linux/mlx5/fs.h | 6 +-
6 files changed, 70 insertions(+), 40 deletions(-)
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 99eb1c1..3ff663c 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1438,7 +1438,8 @@ static struct mlx5_ib_flow_prio *get_flow_table(struct mlx5_ib_dev *dev,
if (!ft) {
ft = mlx5_create_auto_grouped_flow_table(ns, priority,
num_entries,
- num_groups);
+ num_groups,
+ 0);
if (!IS_ERR(ft)) {
prio->refcount = 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index 4df49e6..d61171a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -37,6 +37,11 @@
#include <linux/mlx5/fs.h>
#include "en.h"
+enum {
+ MLX5E_VLAN_FT_LEVEL = 0,
+ MLX5E_MAIN_FT_LEVEL
+};
+
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
enum {
@@ -1041,7 +1046,8 @@ static int mlx5e_create_main_flow_table(struct mlx5e_priv *priv)
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_MAIN_TABLE_SIZE);
+ ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_MAIN_TABLE_SIZE,
+ MLX5E_MAIN_FT_LEVEL);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
@@ -1150,7 +1156,8 @@ static int mlx5e_create_vlan_flow_table(struct mlx5e_priv *priv)
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_VLAN_TABLE_SIZE);
+ ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_VLAN_TABLE_SIZE,
+ MLX5E_VLAN_FT_LEVEL);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
@@ -1167,11 +1174,16 @@ static int mlx5e_create_vlan_flow_table(struct mlx5e_priv *priv)
if (err)
goto err_free_g;
+ err = mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_UNTAGGED, 0);
+ if (err)
+ goto err_destroy_vlan_flow_groups;
+
return 0;
+err_destroy_vlan_flow_groups:
+ mlx5e_destroy_groups(ft);
err_free_g:
kfree(ft->g);
-
err_destroy_vlan_flow_table:
mlx5_destroy_flow_table(ft->t);
ft->t = NULL;
@@ -1194,15 +1206,11 @@ int mlx5e_create_flow_tables(struct mlx5e_priv *priv)
if (!priv->fts.ns)
return -EINVAL;
- err = mlx5e_create_vlan_flow_table(priv);
- if (err)
- return err;
-
err = mlx5e_create_main_flow_table(priv);
if (err)
- goto err_destroy_vlan_flow_table;
+ return err;
- err = mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_UNTAGGED, 0);
+ err = mlx5e_create_vlan_flow_table(priv);
if (err)
goto err_destroy_main_flow_table;
@@ -1210,8 +1218,6 @@ int mlx5e_create_flow_tables(struct mlx5e_priv *priv)
err_destroy_main_flow_table:
mlx5e_destroy_main_flow_table(priv);
-err_destroy_vlan_flow_table:
- mlx5e_destroy_vlan_flow_table(priv);
return err;
}
@@ -1219,6 +1225,6 @@ err_destroy_vlan_flow_table:
void mlx5e_destroy_flow_tables(struct mlx5e_priv *priv)
{
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_UNTAGGED, 0);
- mlx5e_destroy_main_flow_table(priv);
mlx5e_destroy_vlan_flow_table(priv);
+ mlx5e_destroy_main_flow_table(priv);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
index b3de09f..2137387 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
@@ -64,7 +64,8 @@ static struct mlx5_flow_rule *mlx5e_tc_add_flow(struct mlx5e_priv *priv,
priv->fts.tc.t =
mlx5_create_auto_grouped_flow_table(priv->fts.ns, 0,
MLX5E_TC_FLOW_TABLE_NUM_ENTRIES,
- MLX5E_TC_FLOW_TABLE_NUM_GROUPS);
+ MLX5E_TC_FLOW_TABLE_NUM_GROUPS,
+ 0);
if (IS_ERR(priv->fts.tc.t)) {
netdev_err(priv->netdev,
"Failed to create tc offload table\n");
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index bc3d9f8..ff91bb5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -401,7 +401,7 @@ static int esw_create_fdb_table(struct mlx5_eswitch *esw, int nvports)
memset(flow_group_in, 0, inlen);
table_size = BIT(MLX5_CAP_ESW_FLOWTABLE_FDB(dev, log_max_ft_size));
- fdb = mlx5_create_flow_table(root_ns, 0, table_size);
+ fdb = mlx5_create_flow_table(root_ns, 0, table_size, 0);
if (IS_ERR_OR_NULL(fdb)) {
err = PTR_ERR(fdb);
esw_warn(dev, "Failed to create FDB Table err %d\n", err);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index cfb35c3..ca55d7e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -225,19 +225,6 @@ static struct fs_prio *find_prio(struct mlx5_flow_namespace *ns,
return NULL;
}
-static unsigned int find_next_free_level(struct fs_prio *prio)
-{
- if (!list_empty(&prio->node.children)) {
- struct mlx5_flow_table *ft;
-
- ft = list_last_entry(&prio->node.children,
- struct mlx5_flow_table,
- node.list);
- return ft->level + 1;
- }
- return prio->start_level;
-}
-
static bool masked_memcmp(void *mask, void *val1, void *val2, size_t size)
{
unsigned int i;
@@ -696,9 +683,23 @@ static int connect_flow_table(struct mlx5_core_dev *dev, struct mlx5_flow_table
return err;
}
+static void list_add_flow_table(struct mlx5_flow_table *ft,
+ struct fs_prio *prio)
+{
+ struct list_head *prev = &prio->node.children;
+ struct mlx5_flow_table *iter;
+
+ fs_for_each_ft(iter, prio) {
+ if (iter->level > ft->level)
+ break;
+ prev = &iter->node.list;
+ }
+ list_add(&ft->node.list, prev);
+}
+
struct mlx5_flow_table *mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
- int prio,
- int max_fte)
+ int prio, int max_fte,
+ u32 level)
{
struct mlx5_flow_table *next_ft = NULL;
struct mlx5_flow_table *ft;
@@ -719,12 +720,15 @@ struct mlx5_flow_table *mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
err = -EINVAL;
goto unlock_root;
}
- if (fs_prio->num_ft == fs_prio->num_levels) {
+ if (level >= fs_prio->num_levels) {
err = -ENOSPC;
goto unlock_root;
}
-
- ft = alloc_flow_table(find_next_free_level(fs_prio),
+ /* The level is related to the
+ * priority level range.
+ */
+ level += fs_prio->start_level;
+ ft = alloc_flow_table(level,
roundup_pow_of_two(max_fte),
root->table_type);
if (!ft) {
@@ -745,7 +749,7 @@ struct mlx5_flow_table *mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
goto destroy_ft;
lock_ref_node(&fs_prio->node);
tree_add_node(&ft->node, &fs_prio->node);
- list_add_tail(&ft->node.list, &fs_prio->node.children);
+ list_add_flow_table(ft, fs_prio);
fs_prio->num_ft++;
unlock_ref_node(&fs_prio->node);
mutex_unlock(&root->chain_lock);
@@ -762,14 +766,15 @@ unlock_root:
struct mlx5_flow_table *mlx5_create_auto_grouped_flow_table(struct mlx5_flow_namespace *ns,
int prio,
int num_flow_table_entries,
- int max_num_groups)
+ int max_num_groups,
+ u32 level)
{
struct mlx5_flow_table *ft;
if (max_num_groups > num_flow_table_entries)
return ERR_PTR(-EINVAL);
- ft = mlx5_create_flow_table(ns, prio, num_flow_table_entries);
+ ft = mlx5_create_flow_table(ns, prio, num_flow_table_entries, level);
if (IS_ERR(ft))
return ft;
@@ -1068,6 +1073,20 @@ unlock_fg:
return rule;
}
+static bool dest_is_valid(struct mlx5_flow_destination *dest,
+ u32 action,
+ struct mlx5_flow_table *ft)
+{
+ if (!(action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST))
+ return true;
+
+ if (!dest || ((dest->type ==
+ MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE) &&
+ (dest->ft->level <= ft->level)))
+ return false;
+ return true;
+}
+
static struct mlx5_flow_rule *
_mlx5_add_flow_rule(struct mlx5_flow_table *ft,
u8 match_criteria_enable,
@@ -1080,7 +1099,7 @@ _mlx5_add_flow_rule(struct mlx5_flow_table *ft,
struct mlx5_flow_group *g;
struct mlx5_flow_rule *rule;
- if ((action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) && !dest)
+ if (!dest_is_valid(dest, action, ft))
return ERR_PTR(-EINVAL);
nested_lock_ref_node(&ft->node, FS_MUTEX_GRANDPARENT);
@@ -1517,6 +1536,7 @@ static void set_prio_attrs(struct mlx5_flow_root_namespace *root_ns)
#define ANCHOR_PRIO 0
#define ANCHOR_SIZE 1
+#define ANCHOR_LEVEL 0
static int create_anchor_flow_table(struct mlx5_core_dev
*dev)
{
@@ -1526,7 +1546,7 @@ static int create_anchor_flow_table(struct mlx5_core_dev
ns = mlx5_get_flow_namespace(dev, MLX5_FLOW_NAMESPACE_ANCHOR);
if (!ns)
return -EINVAL;
- ft = mlx5_create_flow_table(ns, ANCHOR_PRIO, ANCHOR_SIZE);
+ ft = mlx5_create_flow_table(ns, ANCHOR_PRIO, ANCHOR_SIZE, ANCHOR_LEVEL);
if (IS_ERR(ft)) {
mlx5_core_err(dev, "Failed to create last anchor flow table");
return PTR_ERR(ft);
diff --git a/include/linux/mlx5/fs.h b/include/linux/mlx5/fs.h
index 28a5b66..165ff4f 100644
--- a/include/linux/mlx5/fs.h
+++ b/include/linux/mlx5/fs.h
@@ -82,12 +82,14 @@ struct mlx5_flow_table *
mlx5_create_auto_grouped_flow_table(struct mlx5_flow_namespace *ns,
int prio,
int num_flow_table_entries,
- int max_num_groups);
+ int max_num_groups,
+ u32 level);
struct mlx5_flow_table *
mlx5_create_flow_table(struct mlx5_flow_namespace *ns,
int prio,
- int num_flow_table_entries);
+ int num_flow_table_entries,
+ u32 level);
int mlx5_destroy_flow_table(struct mlx5_flow_table *ft);
/* inbox should be set with the following values:
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 06/12] net/mlx5: Support different attributes for priorities in namespace
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (4 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 05/12] net/mlx5: Add user chosen levels when allocating flow tables Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 07/12] net/mlx5e: Refactor mlx5e flow steering structs Saeed Mahameed
` (6 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Currently, namespace could be initialized only
with priorities with the same attributes.
Add support to initialize namespace with priorities
with different attributes(e.g. different number of levels).
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 31 +++++++++++++--------
1 files changed, 19 insertions(+), 12 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index ca55d7e..2b82293 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -74,9 +74,10 @@
#define BY_PASS_MIN_LEVEL (KERNEL_MIN_LEVEL + MLX5_BY_PASS_NUM_PRIOS +\
LEFTOVERS_NUM_PRIOS)
-#define KERNEL_NUM_LEVELS 3
-#define KERNEL_NUM_PRIOS 2
-#define KERNEL_MIN_LEVEL 2
+#define KERNEL_NIC_PRIO_NUM_LEVELS 2
+#define KERNEL_NIC_NUM_PRIOS 1
+/* One more level for tc */
+#define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
#define ANCHOR_NUM_LEVELS 1
#define ANCHOR_NUM_PRIOS 1
@@ -106,8 +107,9 @@ static struct init_tree_node {
ADD_NS(ADD_MULTIPLE_PRIO(MLX5_BY_PASS_NUM_PRIOS,
BY_PASS_PRIO_NUM_LEVELS))),
ADD_PRIO(0, KERNEL_MIN_LEVEL, 0, {},
- ADD_NS(ADD_MULTIPLE_PRIO(KERNEL_NUM_PRIOS,
- KERNEL_NUM_LEVELS))),
+ ADD_NS(ADD_MULTIPLE_PRIO(1, 1),
+ ADD_MULTIPLE_PRIO(KERNEL_NIC_NUM_PRIOS,
+ KERNEL_NIC_PRIO_NUM_LEVELS))),
ADD_PRIO(0, BY_PASS_MIN_LEVEL, 0,
FS_REQUIRED_CAPS(FS_CAP(flow_table_properties_nic_receive.flow_modify_en),
FS_CAP(flow_table_properties_nic_receive.modify_root),
@@ -1375,14 +1377,14 @@ static struct mlx5_flow_namespace *fs_create_namespace(struct fs_prio *prio)
return ns;
}
-static int create_leaf_prios(struct mlx5_flow_namespace *ns, struct init_tree_node
- *prio_metadata)
+static int create_leaf_prios(struct mlx5_flow_namespace *ns, int prio,
+ struct init_tree_node *prio_metadata)
{
struct fs_prio *fs_prio;
int i;
for (i = 0; i < prio_metadata->num_leaf_prios; i++) {
- fs_prio = fs_create_prio(ns, i, prio_metadata->num_levels);
+ fs_prio = fs_create_prio(ns, prio++, prio_metadata->num_levels);
if (IS_ERR(fs_prio))
return PTR_ERR(fs_prio);
}
@@ -1409,7 +1411,7 @@ static int init_root_tree_recursive(struct mlx5_core_dev *dev,
struct init_tree_node *init_node,
struct fs_node *fs_parent_node,
struct init_tree_node *init_parent_node,
- int index)
+ int prio)
{
int max_ft_level = MLX5_CAP_FLOWTABLE(dev,
flow_table_properties_nic_receive.
@@ -1427,8 +1429,8 @@ static int init_root_tree_recursive(struct mlx5_core_dev *dev,
fs_get_obj(fs_ns, fs_parent_node);
if (init_node->num_leaf_prios)
- return create_leaf_prios(fs_ns, init_node);
- fs_prio = fs_create_prio(fs_ns, index, init_node->num_levels);
+ return create_leaf_prios(fs_ns, prio, init_node);
+ fs_prio = fs_create_prio(fs_ns, prio, init_node->num_levels);
if (IS_ERR(fs_prio))
return PTR_ERR(fs_prio);
base = &fs_prio->node;
@@ -1441,11 +1443,16 @@ static int init_root_tree_recursive(struct mlx5_core_dev *dev,
} else {
return -EINVAL;
}
+ prio = 0;
for (i = 0; i < init_node->ar_size; i++) {
err = init_root_tree_recursive(dev, &init_node->children[i],
- base, init_node, i);
+ base, init_node, prio);
if (err)
return err;
+ if (init_node->children[i].type == FS_TYPE_PRIO &&
+ init_node->children[i].num_leaf_prios) {
+ prio += init_node->children[i].num_leaf_prios;
+ }
}
return 0;
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 07/12] net/mlx5e: Refactor mlx5e flow steering structs
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (5 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 06/12] net/mlx5: Support different attributes for priorities in namespace Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 08/12] net/mlx5e: Split the main flow steering table Saeed Mahameed
` (5 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Slightly refactor and re-order the flow steering structs,
tables and data-bases for better self-containment and
flexibility to add more future steering phases
(tables/rules/data bases) e.g: aRFS.
Changes:
1. Move the vlan DB and address DB into their table structs.
2. Rename steering table structs to unique format: mlx5e_*_table,
e.g: mlx5e_vlan_table.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 73 +++++----
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 186 ++++++++++----------
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 +-
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c | 45 +++---
drivers/net/ethernet/mellanox/mlx5/core/en_tc.h | 2 +-
5 files changed, 160 insertions(+), 154 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 5c8e98c..02b9644 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -388,6 +388,17 @@ enum mlx5e_traffic_types {
MLX5E_NUM_INDIR_TIRS = MLX5E_TT_ANY,
};
+enum {
+ MLX5E_STATE_ASYNC_EVENTS_ENABLE,
+ MLX5E_STATE_OPENED,
+ MLX5E_STATE_DESTROYING,
+};
+
+struct mlx5e_vxlan_db {
+ spinlock_t lock; /* protect vxlan table */
+ struct radix_tree_root tree;
+};
+
struct mlx5e_eth_addr_info {
u8 addr[ETH_ALEN + 2];
u32 tt_vec;
@@ -396,7 +407,14 @@ struct mlx5e_eth_addr_info {
#define MLX5E_ETH_ADDR_HASH_SIZE (1 << BITS_PER_BYTE)
-struct mlx5e_eth_addr_db {
+struct mlx5e_flow_table {
+ int num_groups;
+ struct mlx5_flow_table *t;
+ struct mlx5_flow_group **g;
+};
+
+struct mlx5e_main_table {
+ struct mlx5e_flow_table ft;
struct hlist_head netdev_uc[MLX5E_ETH_ADDR_HASH_SIZE];
struct hlist_head netdev_mc[MLX5E_ETH_ADDR_HASH_SIZE];
struct mlx5e_eth_addr_info broadcast;
@@ -407,13 +425,15 @@ struct mlx5e_eth_addr_db {
bool promisc_enabled;
};
-enum {
- MLX5E_STATE_ASYNC_EVENTS_ENABLE,
- MLX5E_STATE_OPENED,
- MLX5E_STATE_DESTROYING,
+struct mlx5e_tc_table {
+ struct mlx5_flow_table *t;
+
+ struct rhashtable_params ht_params;
+ struct rhashtable ht;
};
-struct mlx5e_vlan_db {
+struct mlx5e_vlan_table {
+ struct mlx5e_flow_table ft;
unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
struct mlx5_flow_rule *active_vlans_rule[VLAN_N_VID];
struct mlx5_flow_rule *untagged_rule;
@@ -421,29 +441,11 @@ struct mlx5e_vlan_db {
bool filter_disabled;
};
-struct mlx5e_vxlan_db {
- spinlock_t lock; /* protect vxlan table */
- struct radix_tree_root tree;
-};
-
-struct mlx5e_flow_table {
- int num_groups;
- struct mlx5_flow_table *t;
- struct mlx5_flow_group **g;
-};
-
-struct mlx5e_tc_flow_table {
- struct mlx5_flow_table *t;
-
- struct rhashtable_params ht_params;
- struct rhashtable ht;
-};
-
-struct mlx5e_flow_tables {
- struct mlx5_flow_namespace *ns;
- struct mlx5e_tc_flow_table tc;
- struct mlx5e_flow_table vlan;
- struct mlx5e_flow_table main;
+struct mlx5e_flow_steering {
+ struct mlx5_flow_namespace *ns;
+ struct mlx5e_tc_table tc;
+ struct mlx5e_vlan_table vlan;
+ struct mlx5e_main_table main;
};
struct mlx5e_direct_tir {
@@ -451,6 +453,11 @@ struct mlx5e_direct_tir {
u32 rqtn;
};
+enum {
+ MLX5E_TC_PRIO = 0,
+ MLX5E_NIC_PRIO
+};
+
struct mlx5e_priv {
/* priv data path fields - start */
struct mlx5e_sq **txq_to_sq_map;
@@ -472,9 +479,7 @@ struct mlx5e_priv {
u32 indir_tirn[MLX5E_NUM_INDIR_TIRS];
struct mlx5e_direct_tir direct_tir[MLX5E_MAX_NUM_CHANNELS];
- struct mlx5e_flow_tables fts;
- struct mlx5e_eth_addr_db eth_addr;
- struct mlx5e_vlan_db vlan;
+ struct mlx5e_flow_steering fs;
struct mlx5e_vxlan_db vxlan;
struct mlx5e_params params;
@@ -556,8 +561,8 @@ struct mlx5_cqe64 *mlx5e_get_cqe(struct mlx5e_cq *cq);
void mlx5e_update_stats(struct mlx5e_priv *priv);
-int mlx5e_create_flow_tables(struct mlx5e_priv *priv);
-void mlx5e_destroy_flow_tables(struct mlx5e_priv *priv);
+int mlx5e_create_flow_steering(struct mlx5e_priv *priv);
+void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv);
void mlx5e_init_eth_addr(struct mlx5e_priv *priv);
void mlx5e_set_rx_mode_work(struct work_struct *work);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index d61171a..3ee35b0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -247,7 +247,7 @@ static int __mlx5e_add_eth_addr_rule(struct mlx5e_priv *priv,
struct mlx5_flow_destination dest;
u8 match_criteria_enable = 0;
struct mlx5_flow_rule **rule_p;
- struct mlx5_flow_table *ft = priv->fts.main.t;
+ struct mlx5_flow_table *ft = priv->fs.main.ft.t;
u8 *mc_dmac = MLX5_ADDR_OF(fte_match_param, mc,
outer_headers.dmac_47_16);
u8 *mv_dmac = MLX5_ADDR_OF(fte_match_param, mv,
@@ -477,7 +477,7 @@ static int mlx5e_vport_context_update_vlans(struct mlx5e_priv *priv)
int i;
list_size = 0;
- for_each_set_bit(vlan, priv->vlan.active_vlans, VLAN_N_VID)
+ for_each_set_bit(vlan, priv->fs.vlan.active_vlans, VLAN_N_VID)
list_size++;
max_list_size = 1 << MLX5_CAP_GEN(priv->mdev, log_max_vlan_list);
@@ -494,7 +494,7 @@ static int mlx5e_vport_context_update_vlans(struct mlx5e_priv *priv)
return -ENOMEM;
i = 0;
- for_each_set_bit(vlan, priv->vlan.active_vlans, VLAN_N_VID) {
+ for_each_set_bit(vlan, priv->fs.vlan.active_vlans, VLAN_N_VID) {
if (i >= list_size)
break;
vlans[i++] = vlan;
@@ -519,28 +519,28 @@ static int __mlx5e_add_vlan_rule(struct mlx5e_priv *priv,
enum mlx5e_vlan_rule_type rule_type,
u16 vid, u32 *mc, u32 *mv)
{
- struct mlx5_flow_table *ft = priv->fts.vlan.t;
+ struct mlx5_flow_table *ft = priv->fs.vlan.ft.t;
struct mlx5_flow_destination dest;
u8 match_criteria_enable = 0;
struct mlx5_flow_rule **rule_p;
int err = 0;
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
- dest.ft = priv->fts.main.t;
+ dest.ft = priv->fs.main.ft.t;
match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.vlan_tag);
switch (rule_type) {
case MLX5E_VLAN_RULE_TYPE_UNTAGGED:
- rule_p = &priv->vlan.untagged_rule;
+ rule_p = &priv->fs.vlan.untagged_rule;
break;
case MLX5E_VLAN_RULE_TYPE_ANY_VID:
- rule_p = &priv->vlan.any_vlan_rule;
+ rule_p = &priv->fs.vlan.any_vlan_rule;
MLX5_SET(fte_match_param, mv, outer_headers.vlan_tag, 1);
break;
default: /* MLX5E_VLAN_RULE_TYPE_MATCH_VID */
- rule_p = &priv->vlan.active_vlans_rule[vid];
+ rule_p = &priv->fs.vlan.active_vlans_rule[vid];
MLX5_SET(fte_match_param, mv, outer_headers.vlan_tag, 1);
MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.first_vid);
MLX5_SET(fte_match_param, mv, outer_headers.first_vid, vid);
@@ -594,22 +594,22 @@ static void mlx5e_del_vlan_rule(struct mlx5e_priv *priv,
{
switch (rule_type) {
case MLX5E_VLAN_RULE_TYPE_UNTAGGED:
- if (priv->vlan.untagged_rule) {
- mlx5_del_flow_rule(priv->vlan.untagged_rule);
- priv->vlan.untagged_rule = NULL;
+ if (priv->fs.vlan.untagged_rule) {
+ mlx5_del_flow_rule(priv->fs.vlan.untagged_rule);
+ priv->fs.vlan.untagged_rule = NULL;
}
break;
case MLX5E_VLAN_RULE_TYPE_ANY_VID:
- if (priv->vlan.any_vlan_rule) {
- mlx5_del_flow_rule(priv->vlan.any_vlan_rule);
- priv->vlan.any_vlan_rule = NULL;
+ if (priv->fs.vlan.any_vlan_rule) {
+ mlx5_del_flow_rule(priv->fs.vlan.any_vlan_rule);
+ priv->fs.vlan.any_vlan_rule = NULL;
}
break;
case MLX5E_VLAN_RULE_TYPE_MATCH_VID:
mlx5e_vport_context_update_vlans(priv);
- if (priv->vlan.active_vlans_rule[vid]) {
- mlx5_del_flow_rule(priv->vlan.active_vlans_rule[vid]);
- priv->vlan.active_vlans_rule[vid] = NULL;
+ if (priv->fs.vlan.active_vlans_rule[vid]) {
+ mlx5_del_flow_rule(priv->fs.vlan.active_vlans_rule[vid]);
+ priv->fs.vlan.active_vlans_rule[vid] = NULL;
}
mlx5e_vport_context_update_vlans(priv);
break;
@@ -618,10 +618,10 @@ static void mlx5e_del_vlan_rule(struct mlx5e_priv *priv,
void mlx5e_enable_vlan_filter(struct mlx5e_priv *priv)
{
- if (!priv->vlan.filter_disabled)
+ if (!priv->fs.vlan.filter_disabled)
return;
- priv->vlan.filter_disabled = false;
+ priv->fs.vlan.filter_disabled = false;
if (priv->netdev->flags & IFF_PROMISC)
return;
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID, 0);
@@ -629,10 +629,10 @@ void mlx5e_enable_vlan_filter(struct mlx5e_priv *priv)
void mlx5e_disable_vlan_filter(struct mlx5e_priv *priv)
{
- if (priv->vlan.filter_disabled)
+ if (priv->fs.vlan.filter_disabled)
return;
- priv->vlan.filter_disabled = true;
+ priv->fs.vlan.filter_disabled = true;
if (priv->netdev->flags & IFF_PROMISC)
return;
mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID, 0);
@@ -643,7 +643,7 @@ int mlx5e_vlan_rx_add_vid(struct net_device *dev, __always_unused __be16 proto,
{
struct mlx5e_priv *priv = netdev_priv(dev);
- set_bit(vid, priv->vlan.active_vlans);
+ set_bit(vid, priv->fs.vlan.active_vlans);
return mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_MATCH_VID, vid);
}
@@ -653,7 +653,7 @@ int mlx5e_vlan_rx_kill_vid(struct net_device *dev, __always_unused __be16 proto,
{
struct mlx5e_priv *priv = netdev_priv(dev);
- clear_bit(vid, priv->vlan.active_vlans);
+ clear_bit(vid, priv->fs.vlan.active_vlans);
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_MATCH_VID, vid);
@@ -687,14 +687,14 @@ static void mlx5e_sync_netdev_addr(struct mlx5e_priv *priv)
netif_addr_lock_bh(netdev);
- mlx5e_add_eth_addr_to_hash(priv->eth_addr.netdev_uc,
+ mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_uc,
priv->netdev->dev_addr);
netdev_for_each_uc_addr(ha, netdev)
- mlx5e_add_eth_addr_to_hash(priv->eth_addr.netdev_uc, ha->addr);
+ mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_uc, ha->addr);
netdev_for_each_mc_addr(ha, netdev)
- mlx5e_add_eth_addr_to_hash(priv->eth_addr.netdev_mc, ha->addr);
+ mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_mc, ha->addr);
netif_addr_unlock_bh(netdev);
}
@@ -710,11 +710,11 @@ static void mlx5e_fill_addr_array(struct mlx5e_priv *priv, int list_type,
int i = 0;
int hi;
- addr_list = is_uc ? priv->eth_addr.netdev_uc : priv->eth_addr.netdev_mc;
+ addr_list = is_uc ? priv->fs.main.netdev_uc : priv->fs.main.netdev_mc;
if (is_uc) /* Make sure our own address is pushed first */
ether_addr_copy(addr_array[i++], ndev->dev_addr);
- else if (priv->eth_addr.broadcast_enabled)
+ else if (priv->fs.main.broadcast_enabled)
ether_addr_copy(addr_array[i++], ndev->broadcast);
mlx5e_for_each_hash_node(hn, tmp, addr_list, hi) {
@@ -739,12 +739,12 @@ static void mlx5e_vport_context_update_addr_list(struct mlx5e_priv *priv,
int err;
int hi;
- size = is_uc ? 0 : (priv->eth_addr.broadcast_enabled ? 1 : 0);
+ size = is_uc ? 0 : (priv->fs.main.broadcast_enabled ? 1 : 0);
max_size = is_uc ?
1 << MLX5_CAP_GEN(priv->mdev, log_max_current_uc_list) :
1 << MLX5_CAP_GEN(priv->mdev, log_max_current_mc_list);
- addr_list = is_uc ? priv->eth_addr.netdev_uc : priv->eth_addr.netdev_mc;
+ addr_list = is_uc ? priv->fs.main.netdev_uc : priv->fs.main.netdev_mc;
mlx5e_for_each_hash_node(hn, tmp, addr_list, hi)
size++;
@@ -775,13 +775,13 @@ out:
static void mlx5e_vport_context_update(struct mlx5e_priv *priv)
{
- struct mlx5e_eth_addr_db *ea = &priv->eth_addr;
+ struct mlx5e_main_table *main_table = &priv->fs.main;
mlx5e_vport_context_update_addr_list(priv, MLX5_NVPRT_LIST_TYPE_UC);
mlx5e_vport_context_update_addr_list(priv, MLX5_NVPRT_LIST_TYPE_MC);
mlx5_modify_nic_vport_promisc(priv->mdev, 0,
- ea->allmulti_enabled,
- ea->promisc_enabled);
+ main_table->allmulti_enabled,
+ main_table->promisc_enabled);
}
static void mlx5e_apply_netdev_addr(struct mlx5e_priv *priv)
@@ -790,10 +790,10 @@ static void mlx5e_apply_netdev_addr(struct mlx5e_priv *priv)
struct hlist_node *tmp;
int i;
- mlx5e_for_each_hash_node(hn, tmp, priv->eth_addr.netdev_uc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_uc, i)
mlx5e_execute_action(priv, hn);
- mlx5e_for_each_hash_node(hn, tmp, priv->eth_addr.netdev_mc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_mc, i)
mlx5e_execute_action(priv, hn);
}
@@ -803,9 +803,9 @@ static void mlx5e_handle_netdev_addr(struct mlx5e_priv *priv)
struct hlist_node *tmp;
int i;
- mlx5e_for_each_hash_node(hn, tmp, priv->eth_addr.netdev_uc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_uc, i)
hn->action = MLX5E_ACTION_DEL;
- mlx5e_for_each_hash_node(hn, tmp, priv->eth_addr.netdev_mc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_mc, i)
hn->action = MLX5E_ACTION_DEL;
if (!test_bit(MLX5E_STATE_DESTROYING, &priv->state))
@@ -819,7 +819,7 @@ void mlx5e_set_rx_mode_work(struct work_struct *work)
struct mlx5e_priv *priv = container_of(work, struct mlx5e_priv,
set_rx_mode_work);
- struct mlx5e_eth_addr_db *ea = &priv->eth_addr;
+ struct mlx5e_main_table *main_table = &priv->fs.main;
struct net_device *ndev = priv->netdev;
bool rx_mode_enable = !test_bit(MLX5E_STATE_DESTROYING, &priv->state);
@@ -827,40 +827,40 @@ void mlx5e_set_rx_mode_work(struct work_struct *work)
bool allmulti_enabled = rx_mode_enable && (ndev->flags & IFF_ALLMULTI);
bool broadcast_enabled = rx_mode_enable;
- bool enable_promisc = !ea->promisc_enabled && promisc_enabled;
- bool disable_promisc = ea->promisc_enabled && !promisc_enabled;
- bool enable_allmulti = !ea->allmulti_enabled && allmulti_enabled;
- bool disable_allmulti = ea->allmulti_enabled && !allmulti_enabled;
- bool enable_broadcast = !ea->broadcast_enabled && broadcast_enabled;
- bool disable_broadcast = ea->broadcast_enabled && !broadcast_enabled;
+ bool enable_promisc = !main_table->promisc_enabled && promisc_enabled;
+ bool disable_promisc = main_table->promisc_enabled && !promisc_enabled;
+ bool enable_allmulti = !main_table->allmulti_enabled && allmulti_enabled;
+ bool disable_allmulti = main_table->allmulti_enabled && !allmulti_enabled;
+ bool enable_broadcast = !main_table->broadcast_enabled && broadcast_enabled;
+ bool disable_broadcast = main_table->broadcast_enabled && !broadcast_enabled;
if (enable_promisc) {
- mlx5e_add_eth_addr_rule(priv, &ea->promisc, MLX5E_PROMISC);
- if (!priv->vlan.filter_disabled)
+ mlx5e_add_eth_addr_rule(priv, &main_table->promisc, MLX5E_PROMISC);
+ if (!priv->fs.vlan.filter_disabled)
mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID,
0);
}
if (enable_allmulti)
- mlx5e_add_eth_addr_rule(priv, &ea->allmulti, MLX5E_ALLMULTI);
+ mlx5e_add_eth_addr_rule(priv, &main_table->allmulti, MLX5E_ALLMULTI);
if (enable_broadcast)
- mlx5e_add_eth_addr_rule(priv, &ea->broadcast, MLX5E_FULLMATCH);
+ mlx5e_add_eth_addr_rule(priv, &main_table->broadcast, MLX5E_FULLMATCH);
mlx5e_handle_netdev_addr(priv);
if (disable_broadcast)
- mlx5e_del_eth_addr_from_flow_table(priv, &ea->broadcast);
+ mlx5e_del_eth_addr_from_flow_table(priv, &main_table->broadcast);
if (disable_allmulti)
- mlx5e_del_eth_addr_from_flow_table(priv, &ea->allmulti);
+ mlx5e_del_eth_addr_from_flow_table(priv, &main_table->allmulti);
if (disable_promisc) {
- if (!priv->vlan.filter_disabled)
+ if (!priv->fs.vlan.filter_disabled)
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID,
0);
- mlx5e_del_eth_addr_from_flow_table(priv, &ea->promisc);
+ mlx5e_del_eth_addr_from_flow_table(priv, &main_table->promisc);
}
- ea->promisc_enabled = promisc_enabled;
- ea->allmulti_enabled = allmulti_enabled;
- ea->broadcast_enabled = broadcast_enabled;
+ main_table->promisc_enabled = promisc_enabled;
+ main_table->allmulti_enabled = allmulti_enabled;
+ main_table->broadcast_enabled = broadcast_enabled;
mlx5e_vport_context_update(priv);
}
@@ -879,7 +879,7 @@ static void mlx5e_destroy_groups(struct mlx5e_flow_table *ft)
void mlx5e_init_eth_addr(struct mlx5e_priv *priv)
{
- ether_addr_copy(priv->eth_addr.broadcast.addr, priv->netdev->broadcast);
+ ether_addr_copy(priv->fs.main.broadcast.addr, priv->netdev->broadcast);
}
#define MLX5E_MAIN_GROUP0_SIZE BIT(3)
@@ -901,8 +901,8 @@ void mlx5e_init_eth_addr(struct mlx5e_priv *priv)
MLX5E_MAIN_GROUP7_SIZE +\
MLX5E_MAIN_GROUP8_SIZE)
-static int __mlx5e_create_main_groups(struct mlx5e_flow_table *ft, u32 *in,
- int inlen)
+static int __mlx5e_create_main_table_groups(struct mlx5e_flow_table *ft, u32 *in,
+ int inlen)
{
u8 *mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
u8 *dmac = MLX5_ADDR_OF(create_flow_group_in, in,
@@ -1024,7 +1024,7 @@ err_destroy_groups:
return err;
}
-static int mlx5e_create_main_groups(struct mlx5e_flow_table *ft)
+static int mlx5e_create_main_table_groups(struct mlx5e_flow_table *ft)
{
u32 *in;
int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
@@ -1034,20 +1034,20 @@ static int mlx5e_create_main_groups(struct mlx5e_flow_table *ft)
if (!in)
return -ENOMEM;
- err = __mlx5e_create_main_groups(ft, in, inlen);
+ err = __mlx5e_create_main_table_groups(ft, in, inlen);
kvfree(in);
return err;
}
-static int mlx5e_create_main_flow_table(struct mlx5e_priv *priv)
+static int mlx5e_create_main_table(struct mlx5e_priv *priv)
{
- struct mlx5e_flow_table *ft = &priv->fts.main;
+ struct mlx5e_flow_table *ft = &priv->fs.main.ft;
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_MAIN_TABLE_SIZE,
- MLX5E_MAIN_FT_LEVEL);
+ ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
+ MLX5E_MAIN_TABLE_SIZE, MLX5E_MAIN_FT_LEVEL);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
@@ -1057,10 +1057,10 @@ static int mlx5e_create_main_flow_table(struct mlx5e_priv *priv)
ft->g = kcalloc(MLX5E_NUM_MAIN_GROUPS, sizeof(*ft->g), GFP_KERNEL);
if (!ft->g) {
err = -ENOMEM;
- goto err_destroy_main_flow_table;
+ goto err_destroy_main_table;
}
- err = mlx5e_create_main_groups(ft);
+ err = mlx5e_create_main_table_groups(ft);
if (err)
goto err_free_g;
return 0;
@@ -1068,7 +1068,7 @@ static int mlx5e_create_main_flow_table(struct mlx5e_priv *priv)
err_free_g:
kfree(ft->g);
-err_destroy_main_flow_table:
+err_destroy_main_table:
mlx5_destroy_flow_table(ft->t);
ft->t = NULL;
@@ -1083,9 +1083,9 @@ static void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft)
ft->t = NULL;
}
-static void mlx5e_destroy_main_flow_table(struct mlx5e_priv *priv)
+static void mlx5e_destroy_main_table(struct mlx5e_priv *priv)
{
- mlx5e_destroy_flow_table(&priv->fts.main);
+ mlx5e_destroy_flow_table(&priv->fs.main.ft);
}
#define MLX5E_NUM_VLAN_GROUPS 2
@@ -1094,8 +1094,8 @@ static void mlx5e_destroy_main_flow_table(struct mlx5e_priv *priv)
#define MLX5E_VLAN_TABLE_SIZE (MLX5E_VLAN_GROUP0_SIZE +\
MLX5E_VLAN_GROUP1_SIZE)
-static int __mlx5e_create_vlan_groups(struct mlx5e_flow_table *ft, u32 *in,
- int inlen)
+static int __mlx5e_create_vlan_table_groups(struct mlx5e_flow_table *ft, u32 *in,
+ int inlen)
{
int err;
int ix = 0;
@@ -1134,7 +1134,7 @@ err_destroy_groups:
return err;
}
-static int mlx5e_create_vlan_groups(struct mlx5e_flow_table *ft)
+static int mlx5e_create_vlan_table_groups(struct mlx5e_flow_table *ft)
{
u32 *in;
int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
@@ -1144,20 +1144,20 @@ static int mlx5e_create_vlan_groups(struct mlx5e_flow_table *ft)
if (!in)
return -ENOMEM;
- err = __mlx5e_create_vlan_groups(ft, in, inlen);
+ err = __mlx5e_create_vlan_table_groups(ft, in, inlen);
kvfree(in);
return err;
}
-static int mlx5e_create_vlan_flow_table(struct mlx5e_priv *priv)
+static int mlx5e_create_vlan_table(struct mlx5e_priv *priv)
{
- struct mlx5e_flow_table *ft = &priv->fts.vlan;
+ struct mlx5e_flow_table *ft = &priv->fs.vlan.ft;
int err;
ft->num_groups = 0;
- ft->t = mlx5_create_flow_table(priv->fts.ns, 1, MLX5E_VLAN_TABLE_SIZE,
- MLX5E_VLAN_FT_LEVEL);
+ ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
+ MLX5E_VLAN_TABLE_SIZE, MLX5E_VLAN_FT_LEVEL);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
@@ -1167,10 +1167,10 @@ static int mlx5e_create_vlan_flow_table(struct mlx5e_priv *priv)
ft->g = kcalloc(MLX5E_NUM_VLAN_GROUPS, sizeof(*ft->g), GFP_KERNEL);
if (!ft->g) {
err = -ENOMEM;
- goto err_destroy_vlan_flow_table;
+ goto err_destroy_vlan_table;
}
- err = mlx5e_create_vlan_groups(ft);
+ err = mlx5e_create_vlan_table_groups(ft);
if (err)
goto err_free_g;
@@ -1184,47 +1184,47 @@ err_destroy_vlan_flow_groups:
mlx5e_destroy_groups(ft);
err_free_g:
kfree(ft->g);
-err_destroy_vlan_flow_table:
+err_destroy_vlan_table:
mlx5_destroy_flow_table(ft->t);
ft->t = NULL;
return err;
}
-static void mlx5e_destroy_vlan_flow_table(struct mlx5e_priv *priv)
+static void mlx5e_destroy_vlan_table(struct mlx5e_priv *priv)
{
- mlx5e_destroy_flow_table(&priv->fts.vlan);
+ mlx5e_destroy_flow_table(&priv->fs.vlan.ft);
}
-int mlx5e_create_flow_tables(struct mlx5e_priv *priv)
+int mlx5e_create_flow_steering(struct mlx5e_priv *priv)
{
int err;
- priv->fts.ns = mlx5_get_flow_namespace(priv->mdev,
+ priv->fs.ns = mlx5_get_flow_namespace(priv->mdev,
MLX5_FLOW_NAMESPACE_KERNEL);
- if (!priv->fts.ns)
+ if (!priv->fs.ns)
return -EINVAL;
- err = mlx5e_create_main_flow_table(priv);
+ err = mlx5e_create_main_table(priv);
if (err)
return err;
- err = mlx5e_create_vlan_flow_table(priv);
+ err = mlx5e_create_vlan_table(priv);
if (err)
- goto err_destroy_main_flow_table;
+ goto err_destroy_main_table;
return 0;
-err_destroy_main_flow_table:
- mlx5e_destroy_main_flow_table(priv);
+err_destroy_main_table:
+ mlx5e_destroy_main_table(priv);
return err;
}
-void mlx5e_destroy_flow_tables(struct mlx5e_priv *priv)
+void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv)
{
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_UNTAGGED, 0);
- mlx5e_destroy_vlan_flow_table(priv);
- mlx5e_destroy_main_flow_table(priv);
+ mlx5e_destroy_vlan_table(priv);
+ mlx5e_destroy_main_table(priv);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 99a1bff..3fad66f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2967,9 +2967,9 @@ static void *mlx5e_create_netdev(struct mlx5_core_dev *mdev)
goto err_destroy_rqts;
}
- err = mlx5e_create_flow_tables(priv);
+ err = mlx5e_create_flow_steering(priv);
if (err) {
- mlx5_core_warn(mdev, "create flow tables failed, %d\n", err);
+ mlx5_core_warn(mdev, "create flow steering failed, %d\n", err);
goto err_destroy_tirs;
}
@@ -3009,7 +3009,7 @@ err_tc_cleanup:
err_dealloc_q_counters:
mlx5e_destroy_q_counter(priv);
- mlx5e_destroy_flow_tables(priv);
+ mlx5e_destroy_flow_steering(priv);
err_destroy_tirs:
mlx5e_destroy_tirs(priv);
@@ -3067,7 +3067,7 @@ static void mlx5e_destroy_netdev(struct mlx5_core_dev *mdev, void *vpriv)
mlx5e_tc_cleanup(priv);
mlx5e_vxlan_cleanup(priv);
mlx5e_destroy_q_counter(priv);
- mlx5e_destroy_flow_tables(priv);
+ mlx5e_destroy_flow_steering(priv);
mlx5e_destroy_tirs(priv);
mlx5e_destroy_rqts(priv);
mlx5e_close_drop_rq(priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
index 2137387..ef017c0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
@@ -46,8 +46,8 @@ struct mlx5e_tc_flow {
struct mlx5_flow_rule *rule;
};
-#define MLX5E_TC_FLOW_TABLE_NUM_ENTRIES 1024
-#define MLX5E_TC_FLOW_TABLE_NUM_GROUPS 4
+#define MLX5E_TC_TABLE_NUM_ENTRIES 1024
+#define MLX5E_TC_TABLE_NUM_GROUPS 4
static struct mlx5_flow_rule *mlx5e_tc_add_flow(struct mlx5e_priv *priv,
u32 *match_c, u32 *match_v,
@@ -55,34 +55,35 @@ static struct mlx5_flow_rule *mlx5e_tc_add_flow(struct mlx5e_priv *priv,
{
struct mlx5_flow_destination dest = {
.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE,
- {.ft = priv->fts.vlan.t},
+ {.ft = priv->fs.vlan.ft.t},
};
struct mlx5_flow_rule *rule;
bool table_created = false;
- if (IS_ERR_OR_NULL(priv->fts.tc.t)) {
- priv->fts.tc.t =
- mlx5_create_auto_grouped_flow_table(priv->fts.ns, 0,
- MLX5E_TC_FLOW_TABLE_NUM_ENTRIES,
- MLX5E_TC_FLOW_TABLE_NUM_GROUPS,
+ if (IS_ERR_OR_NULL(priv->fs.tc.t)) {
+ priv->fs.tc.t =
+ mlx5_create_auto_grouped_flow_table(priv->fs.ns,
+ MLX5E_TC_PRIO,
+ MLX5E_TC_TABLE_NUM_ENTRIES,
+ MLX5E_TC_TABLE_NUM_GROUPS,
0);
- if (IS_ERR(priv->fts.tc.t)) {
+ if (IS_ERR(priv->fs.tc.t)) {
netdev_err(priv->netdev,
"Failed to create tc offload table\n");
- return ERR_CAST(priv->fts.tc.t);
+ return ERR_CAST(priv->fs.tc.t);
}
table_created = true;
}
- rule = mlx5_add_flow_rule(priv->fts.tc.t, MLX5_MATCH_OUTER_HEADERS,
+ rule = mlx5_add_flow_rule(priv->fs.tc.t, MLX5_MATCH_OUTER_HEADERS,
match_c, match_v,
action, flow_tag,
action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST ? &dest : NULL);
if (IS_ERR(rule) && table_created) {
- mlx5_destroy_flow_table(priv->fts.tc.t);
- priv->fts.tc.t = NULL;
+ mlx5_destroy_flow_table(priv->fs.tc.t);
+ priv->fs.tc.t = NULL;
}
return rule;
@@ -94,8 +95,8 @@ static void mlx5e_tc_del_flow(struct mlx5e_priv *priv,
mlx5_del_flow_rule(rule);
if (!mlx5e_tc_num_filters(priv)) {
- mlx5_destroy_flow_table(priv->fts.tc.t);
- priv->fts.tc.t = NULL;
+ mlx5_destroy_flow_table(priv->fs.tc.t);
+ priv->fs.tc.t = NULL;
}
}
@@ -311,7 +312,7 @@ static int parse_tc_actions(struct mlx5e_priv *priv, struct tcf_exts *exts,
int mlx5e_configure_flower(struct mlx5e_priv *priv, __be16 protocol,
struct tc_cls_flower_offload *f)
{
- struct mlx5e_tc_flow_table *tc = &priv->fts.tc;
+ struct mlx5e_tc_table *tc = &priv->fs.tc;
u32 *match_c;
u32 *match_v;
int err = 0;
@@ -377,7 +378,7 @@ int mlx5e_delete_flower(struct mlx5e_priv *priv,
struct tc_cls_flower_offload *f)
{
struct mlx5e_tc_flow *flow;
- struct mlx5e_tc_flow_table *tc = &priv->fts.tc;
+ struct mlx5e_tc_table *tc = &priv->fs.tc;
flow = rhashtable_lookup_fast(&tc->ht, &f->cookie,
tc->ht_params);
@@ -402,7 +403,7 @@ static const struct rhashtable_params mlx5e_tc_flow_ht_params = {
int mlx5e_tc_init(struct mlx5e_priv *priv)
{
- struct mlx5e_tc_flow_table *tc = &priv->fts.tc;
+ struct mlx5e_tc_table *tc = &priv->fs.tc;
tc->ht_params = mlx5e_tc_flow_ht_params;
return rhashtable_init(&tc->ht, &tc->ht_params);
@@ -419,12 +420,12 @@ static void _mlx5e_tc_del_flow(void *ptr, void *arg)
void mlx5e_tc_cleanup(struct mlx5e_priv *priv)
{
- struct mlx5e_tc_flow_table *tc = &priv->fts.tc;
+ struct mlx5e_tc_table *tc = &priv->fs.tc;
rhashtable_free_and_destroy(&tc->ht, _mlx5e_tc_del_flow, priv);
- if (!IS_ERR_OR_NULL(priv->fts.tc.t)) {
- mlx5_destroy_flow_table(priv->fts.tc.t);
- priv->fts.tc.t = NULL;
+ if (!IS_ERR_OR_NULL(tc->t)) {
+ mlx5_destroy_flow_table(tc->t);
+ tc->t = NULL;
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
index d677428..a4f17b9 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.h
@@ -45,7 +45,7 @@ int mlx5e_delete_flower(struct mlx5e_priv *priv,
static inline int mlx5e_tc_num_filters(struct mlx5e_priv *priv)
{
- return atomic_read(&priv->fts.tc.ht.nelems);
+ return atomic_read(&priv->fs.tc.ht.nelems);
}
#endif /* __MLX5_EN_TC_H__ */
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 08/12] net/mlx5e: Split the main flow steering table
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (6 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 07/12] net/mlx5e: Refactor mlx5e flow steering structs Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 09/12] net/mlx5: Initializing CPU reverse mapping Saeed Mahameed
` (4 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Currently, the main flow table is used for two purposes:
One is to do mac filtering and the other is to classify
the packet l3-l4 header in order to steer the packet to
the right RSS TIR.
This design is very complex, for each configured mac address we
have to add eleven rules (rule for each traffic type), the same if the
device is put to promiscuous/allmulti mode.
This scheme isn't scalable for future features like aRFS.
In order to simplify it, the main flow table is split to two flow
tables:
1. l2 table - filter the packet dmac address, if there is a match
we forward to the ttc flow table.
2. TTC (Traffic Type Classifier) table - classify the traffic
type of the packet and steer the packet to the right TIR.
In this new design, when new mac address is added, the driver adds
only one flow rule instead of eleven.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 42 +-
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 979 +++++++++------------
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 2 +-
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 2 +-
4 files changed, 462 insertions(+), 563 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 02b9644..2c9879c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -399,31 +399,18 @@ struct mlx5e_vxlan_db {
struct radix_tree_root tree;
};
-struct mlx5e_eth_addr_info {
+struct mlx5e_l2_rule {
u8 addr[ETH_ALEN + 2];
- u32 tt_vec;
- struct mlx5_flow_rule *ft_rule[MLX5E_NUM_TT];
+ struct mlx5_flow_rule *rule;
};
-#define MLX5E_ETH_ADDR_HASH_SIZE (1 << BITS_PER_BYTE)
-
struct mlx5e_flow_table {
int num_groups;
struct mlx5_flow_table *t;
struct mlx5_flow_group **g;
};
-struct mlx5e_main_table {
- struct mlx5e_flow_table ft;
- struct hlist_head netdev_uc[MLX5E_ETH_ADDR_HASH_SIZE];
- struct hlist_head netdev_mc[MLX5E_ETH_ADDR_HASH_SIZE];
- struct mlx5e_eth_addr_info broadcast;
- struct mlx5e_eth_addr_info allmulti;
- struct mlx5e_eth_addr_info promisc;
- bool broadcast_enabled;
- bool allmulti_enabled;
- bool promisc_enabled;
-};
+#define MLX5E_L2_ADDR_HASH_SIZE BIT(BITS_PER_BYTE)
struct mlx5e_tc_table {
struct mlx5_flow_table *t;
@@ -441,11 +428,30 @@ struct mlx5e_vlan_table {
bool filter_disabled;
};
+struct mlx5e_l2_table {
+ struct mlx5e_flow_table ft;
+ struct hlist_head netdev_uc[MLX5E_L2_ADDR_HASH_SIZE];
+ struct hlist_head netdev_mc[MLX5E_L2_ADDR_HASH_SIZE];
+ struct mlx5e_l2_rule broadcast;
+ struct mlx5e_l2_rule allmulti;
+ struct mlx5e_l2_rule promisc;
+ bool broadcast_enabled;
+ bool allmulti_enabled;
+ bool promisc_enabled;
+};
+
+/* L3/L4 traffic type classifier */
+struct mlx5e_ttc_table {
+ struct mlx5e_flow_table ft;
+ struct mlx5_flow_rule *rules[MLX5E_NUM_TT];
+};
+
struct mlx5e_flow_steering {
struct mlx5_flow_namespace *ns;
struct mlx5e_tc_table tc;
struct mlx5e_vlan_table vlan;
- struct mlx5e_main_table main;
+ struct mlx5e_l2_table l2;
+ struct mlx5e_ttc_table ttc;
};
struct mlx5e_direct_tir {
@@ -563,7 +569,7 @@ void mlx5e_update_stats(struct mlx5e_priv *priv);
int mlx5e_create_flow_steering(struct mlx5e_priv *priv);
void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv);
-void mlx5e_init_eth_addr(struct mlx5e_priv *priv);
+void mlx5e_init_l2_addr(struct mlx5e_priv *priv);
void mlx5e_set_rx_mode_work(struct work_struct *work);
void mlx5e_fill_hwstamp(struct mlx5e_tstamp *clock, u64 timestamp,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index 3ee35b0..6e353b3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -37,9 +37,16 @@
#include <linux/mlx5/fs.h>
#include "en.h"
+static int mlx5e_add_l2_flow_rule(struct mlx5e_priv *priv,
+ struct mlx5e_l2_rule *ai, int type);
+static void mlx5e_del_l2_flow_rule(struct mlx5e_priv *priv,
+ struct mlx5e_l2_rule *ai);
+
+/* NIC prio FTS */
enum {
MLX5E_VLAN_FT_LEVEL = 0,
- MLX5E_MAIN_FT_LEVEL
+ MLX5E_L2_FT_LEVEL,
+ MLX5E_TTC_FT_LEVEL
};
#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
@@ -63,21 +70,21 @@ enum {
MLX5E_ACTION_DEL = 2,
};
-struct mlx5e_eth_addr_hash_node {
+struct mlx5e_l2_hash_node {
struct hlist_node hlist;
u8 action;
- struct mlx5e_eth_addr_info ai;
+ struct mlx5e_l2_rule ai;
};
-static inline int mlx5e_hash_eth_addr(u8 *addr)
+static inline int mlx5e_hash_l2(u8 *addr)
{
return addr[5];
}
-static void mlx5e_add_eth_addr_to_hash(struct hlist_head *hash, u8 *addr)
+static void mlx5e_add_l2_to_hash(struct hlist_head *hash, u8 *addr)
{
- struct mlx5e_eth_addr_hash_node *hn;
- int ix = mlx5e_hash_eth_addr(addr);
+ struct mlx5e_l2_hash_node *hn;
+ int ix = mlx5e_hash_l2(addr);
int found = 0;
hlist_for_each_entry(hn, &hash[ix], hlist)
@@ -101,371 +108,12 @@ static void mlx5e_add_eth_addr_to_hash(struct hlist_head *hash, u8 *addr)
hlist_add_head(&hn->hlist, &hash[ix]);
}
-static void mlx5e_del_eth_addr_from_hash(struct mlx5e_eth_addr_hash_node *hn)
+static void mlx5e_del_l2_from_hash(struct mlx5e_l2_hash_node *hn)
{
hlist_del(&hn->hlist);
kfree(hn);
}
-static void mlx5e_del_eth_addr_from_flow_table(struct mlx5e_priv *priv,
- struct mlx5e_eth_addr_info *ai)
-{
- if (ai->tt_vec & BIT(MLX5E_TT_IPV6_IPSEC_ESP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV6_IPSEC_ESP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV4_IPSEC_ESP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV4_IPSEC_ESP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV6_IPSEC_AH))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV6_IPSEC_AH]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV4_IPSEC_AH))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV4_IPSEC_AH]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV6_TCP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV6_TCP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV4_TCP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV4_TCP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV6_UDP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV6_UDP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV4_UDP))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV4_UDP]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV6))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV6]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_IPV4))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_IPV4]);
-
- if (ai->tt_vec & BIT(MLX5E_TT_ANY))
- mlx5_del_flow_rule(ai->ft_rule[MLX5E_TT_ANY]);
-}
-
-static int mlx5e_get_eth_addr_type(u8 *addr)
-{
- if (is_unicast_ether_addr(addr))
- return MLX5E_UC;
-
- if ((addr[0] == 0x01) &&
- (addr[1] == 0x00) &&
- (addr[2] == 0x5e) &&
- !(addr[3] & 0x80))
- return MLX5E_MC_IPV4;
-
- if ((addr[0] == 0x33) &&
- (addr[1] == 0x33))
- return MLX5E_MC_IPV6;
-
- return MLX5E_MC_OTHER;
-}
-
-static u32 mlx5e_get_tt_vec(struct mlx5e_eth_addr_info *ai, int type)
-{
- int eth_addr_type;
- u32 ret;
-
- switch (type) {
- case MLX5E_FULLMATCH:
- eth_addr_type = mlx5e_get_eth_addr_type(ai->addr);
- switch (eth_addr_type) {
- case MLX5E_UC:
- ret =
- BIT(MLX5E_TT_IPV4_TCP) |
- BIT(MLX5E_TT_IPV6_TCP) |
- BIT(MLX5E_TT_IPV4_UDP) |
- BIT(MLX5E_TT_IPV6_UDP) |
- BIT(MLX5E_TT_IPV4_IPSEC_AH) |
- BIT(MLX5E_TT_IPV6_IPSEC_AH) |
- BIT(MLX5E_TT_IPV4_IPSEC_ESP) |
- BIT(MLX5E_TT_IPV6_IPSEC_ESP) |
- BIT(MLX5E_TT_IPV4) |
- BIT(MLX5E_TT_IPV6) |
- BIT(MLX5E_TT_ANY) |
- 0;
- break;
-
- case MLX5E_MC_IPV4:
- ret =
- BIT(MLX5E_TT_IPV4_UDP) |
- BIT(MLX5E_TT_IPV4) |
- 0;
- break;
-
- case MLX5E_MC_IPV6:
- ret =
- BIT(MLX5E_TT_IPV6_UDP) |
- BIT(MLX5E_TT_IPV6) |
- 0;
- break;
-
- case MLX5E_MC_OTHER:
- ret =
- BIT(MLX5E_TT_ANY) |
- 0;
- break;
- }
-
- break;
-
- case MLX5E_ALLMULTI:
- ret =
- BIT(MLX5E_TT_IPV4_UDP) |
- BIT(MLX5E_TT_IPV6_UDP) |
- BIT(MLX5E_TT_IPV4) |
- BIT(MLX5E_TT_IPV6) |
- BIT(MLX5E_TT_ANY) |
- 0;
- break;
-
- default: /* MLX5E_PROMISC */
- ret =
- BIT(MLX5E_TT_IPV4_TCP) |
- BIT(MLX5E_TT_IPV6_TCP) |
- BIT(MLX5E_TT_IPV4_UDP) |
- BIT(MLX5E_TT_IPV6_UDP) |
- BIT(MLX5E_TT_IPV4_IPSEC_AH) |
- BIT(MLX5E_TT_IPV6_IPSEC_AH) |
- BIT(MLX5E_TT_IPV4_IPSEC_ESP) |
- BIT(MLX5E_TT_IPV6_IPSEC_ESP) |
- BIT(MLX5E_TT_IPV4) |
- BIT(MLX5E_TT_IPV6) |
- BIT(MLX5E_TT_ANY) |
- 0;
- break;
- }
-
- return ret;
-}
-
-static int __mlx5e_add_eth_addr_rule(struct mlx5e_priv *priv,
- struct mlx5e_eth_addr_info *ai,
- int type, u32 *mc, u32 *mv)
-{
- struct mlx5_flow_destination dest;
- u8 match_criteria_enable = 0;
- struct mlx5_flow_rule **rule_p;
- struct mlx5_flow_table *ft = priv->fs.main.ft.t;
- u8 *mc_dmac = MLX5_ADDR_OF(fte_match_param, mc,
- outer_headers.dmac_47_16);
- u8 *mv_dmac = MLX5_ADDR_OF(fte_match_param, mv,
- outer_headers.dmac_47_16);
- u32 *tirn = priv->indir_tirn;
- u32 tt_vec;
- int err = 0;
-
- dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
-
- switch (type) {
- case MLX5E_FULLMATCH:
- match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
- eth_broadcast_addr(mc_dmac);
- ether_addr_copy(mv_dmac, ai->addr);
- break;
-
- case MLX5E_ALLMULTI:
- match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
- mc_dmac[0] = 0x01;
- mv_dmac[0] = 0x01;
- break;
-
- case MLX5E_PROMISC:
- break;
- }
-
- tt_vec = mlx5e_get_tt_vec(ai, type);
-
- if (tt_vec & BIT(MLX5E_TT_ANY)) {
- rule_p = &ai->ft_rule[MLX5E_TT_ANY];
- dest.tir_num = priv->direct_tir[0].tirn;
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_ANY);
- }
-
- match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
-
- if (tt_vec & BIT(MLX5E_TT_IPV4)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV4];
- dest.tir_num = tirn[MLX5E_TT_IPV4];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IP);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV4);
- }
-
- if (tt_vec & BIT(MLX5E_TT_IPV6)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV6];
- dest.tir_num = tirn[MLX5E_TT_IPV6];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IPV6);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV6);
- }
-
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ip_protocol);
- MLX5_SET(fte_match_param, mv, outer_headers.ip_protocol, IPPROTO_UDP);
-
- if (tt_vec & BIT(MLX5E_TT_IPV4_UDP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV4_UDP];
- dest.tir_num = tirn[MLX5E_TT_IPV4_UDP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IP);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV4_UDP);
- }
-
- if (tt_vec & BIT(MLX5E_TT_IPV6_UDP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV6_UDP];
- dest.tir_num = tirn[MLX5E_TT_IPV6_UDP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IPV6);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV6_UDP);
- }
-
- MLX5_SET(fte_match_param, mv, outer_headers.ip_protocol, IPPROTO_TCP);
-
- if (tt_vec & BIT(MLX5E_TT_IPV4_TCP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV4_TCP];
- dest.tir_num = tirn[MLX5E_TT_IPV4_TCP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IP);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV4_TCP);
- }
-
- if (tt_vec & BIT(MLX5E_TT_IPV6_TCP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV6_TCP];
- dest.tir_num = tirn[MLX5E_TT_IPV6_TCP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IPV6);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
-
- ai->tt_vec |= BIT(MLX5E_TT_IPV6_TCP);
- }
-
- MLX5_SET(fte_match_param, mv, outer_headers.ip_protocol, IPPROTO_AH);
-
- if (tt_vec & BIT(MLX5E_TT_IPV4_IPSEC_AH)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV4_IPSEC_AH];
- dest.tir_num = tirn[MLX5E_TT_IPV4_IPSEC_AH];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IP);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV4_IPSEC_AH);
- }
-
- if (tt_vec & BIT(MLX5E_TT_IPV6_IPSEC_AH)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV6_IPSEC_AH];
- dest.tir_num = tirn[MLX5E_TT_IPV6_IPSEC_AH];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IPV6);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV6_IPSEC_AH);
- }
-
- MLX5_SET(fte_match_param, mv, outer_headers.ip_protocol, IPPROTO_ESP);
-
- if (tt_vec & BIT(MLX5E_TT_IPV4_IPSEC_ESP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV4_IPSEC_ESP];
- dest.tir_num = tirn[MLX5E_TT_IPV4_IPSEC_ESP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IP);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV4_IPSEC_ESP);
- }
-
- if (tt_vec & BIT(MLX5E_TT_IPV6_IPSEC_ESP)) {
- rule_p = &ai->ft_rule[MLX5E_TT_IPV6_IPSEC_ESP];
- dest.tir_num = tirn[MLX5E_TT_IPV6_IPSEC_ESP];
- MLX5_SET(fte_match_param, mv, outer_headers.ethertype,
- ETH_P_IPV6);
- *rule_p = mlx5_add_flow_rule(ft, match_criteria_enable, mc, mv,
- MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
- MLX5_FS_DEFAULT_FLOW_TAG, &dest);
- if (IS_ERR_OR_NULL(*rule_p))
- goto err_del_ai;
- ai->tt_vec |= BIT(MLX5E_TT_IPV6_IPSEC_ESP);
- }
-
- return 0;
-
-err_del_ai:
- err = PTR_ERR(*rule_p);
- *rule_p = NULL;
- mlx5e_del_eth_addr_from_flow_table(priv, ai);
-
- return err;
-}
-
-static int mlx5e_add_eth_addr_rule(struct mlx5e_priv *priv,
- struct mlx5e_eth_addr_info *ai, int type)
-{
- u32 *match_criteria;
- u32 *match_value;
- int err = 0;
-
- match_value = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
- match_criteria = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
- if (!match_value || !match_criteria) {
- netdev_err(priv->netdev, "%s: alloc failed\n", __func__);
- err = -ENOMEM;
- goto add_eth_addr_rule_out;
- }
-
- err = __mlx5e_add_eth_addr_rule(priv, ai, type, match_criteria,
- match_value);
-
-add_eth_addr_rule_out:
- kvfree(match_criteria);
- kvfree(match_value);
-
- return err;
-}
-
static int mlx5e_vport_context_update_vlans(struct mlx5e_priv *priv)
{
struct net_device *ndev = priv->netdev;
@@ -526,7 +174,7 @@ static int __mlx5e_add_vlan_rule(struct mlx5e_priv *priv,
int err = 0;
dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
- dest.ft = priv->fs.main.ft.t;
+ dest.ft = priv->fs.l2.ft.t;
match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.vlan_tag);
@@ -661,21 +309,21 @@ int mlx5e_vlan_rx_kill_vid(struct net_device *dev, __always_unused __be16 proto,
}
#define mlx5e_for_each_hash_node(hn, tmp, hash, i) \
- for (i = 0; i < MLX5E_ETH_ADDR_HASH_SIZE; i++) \
+ for (i = 0; i < MLX5E_L2_ADDR_HASH_SIZE; i++) \
hlist_for_each_entry_safe(hn, tmp, &hash[i], hlist)
-static void mlx5e_execute_action(struct mlx5e_priv *priv,
- struct mlx5e_eth_addr_hash_node *hn)
+static void mlx5e_execute_l2_action(struct mlx5e_priv *priv,
+ struct mlx5e_l2_hash_node *hn)
{
switch (hn->action) {
case MLX5E_ACTION_ADD:
- mlx5e_add_eth_addr_rule(priv, &hn->ai, MLX5E_FULLMATCH);
+ mlx5e_add_l2_flow_rule(priv, &hn->ai, MLX5E_FULLMATCH);
hn->action = MLX5E_ACTION_NONE;
break;
case MLX5E_ACTION_DEL:
- mlx5e_del_eth_addr_from_flow_table(priv, &hn->ai);
- mlx5e_del_eth_addr_from_hash(hn);
+ mlx5e_del_l2_flow_rule(priv, &hn->ai);
+ mlx5e_del_l2_from_hash(hn);
break;
}
}
@@ -687,14 +335,14 @@ static void mlx5e_sync_netdev_addr(struct mlx5e_priv *priv)
netif_addr_lock_bh(netdev);
- mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_uc,
- priv->netdev->dev_addr);
+ mlx5e_add_l2_to_hash(priv->fs.l2.netdev_uc,
+ priv->netdev->dev_addr);
netdev_for_each_uc_addr(ha, netdev)
- mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_uc, ha->addr);
+ mlx5e_add_l2_to_hash(priv->fs.l2.netdev_uc, ha->addr);
netdev_for_each_mc_addr(ha, netdev)
- mlx5e_add_eth_addr_to_hash(priv->fs.main.netdev_mc, ha->addr);
+ mlx5e_add_l2_to_hash(priv->fs.l2.netdev_mc, ha->addr);
netif_addr_unlock_bh(netdev);
}
@@ -704,17 +352,17 @@ static void mlx5e_fill_addr_array(struct mlx5e_priv *priv, int list_type,
{
bool is_uc = (list_type == MLX5_NVPRT_LIST_TYPE_UC);
struct net_device *ndev = priv->netdev;
- struct mlx5e_eth_addr_hash_node *hn;
+ struct mlx5e_l2_hash_node *hn;
struct hlist_head *addr_list;
struct hlist_node *tmp;
int i = 0;
int hi;
- addr_list = is_uc ? priv->fs.main.netdev_uc : priv->fs.main.netdev_mc;
+ addr_list = is_uc ? priv->fs.l2.netdev_uc : priv->fs.l2.netdev_mc;
if (is_uc) /* Make sure our own address is pushed first */
ether_addr_copy(addr_array[i++], ndev->dev_addr);
- else if (priv->fs.main.broadcast_enabled)
+ else if (priv->fs.l2.broadcast_enabled)
ether_addr_copy(addr_array[i++], ndev->broadcast);
mlx5e_for_each_hash_node(hn, tmp, addr_list, hi) {
@@ -730,7 +378,7 @@ static void mlx5e_vport_context_update_addr_list(struct mlx5e_priv *priv,
int list_type)
{
bool is_uc = (list_type == MLX5_NVPRT_LIST_TYPE_UC);
- struct mlx5e_eth_addr_hash_node *hn;
+ struct mlx5e_l2_hash_node *hn;
u8 (*addr_array)[ETH_ALEN] = NULL;
struct hlist_head *addr_list;
struct hlist_node *tmp;
@@ -739,12 +387,12 @@ static void mlx5e_vport_context_update_addr_list(struct mlx5e_priv *priv,
int err;
int hi;
- size = is_uc ? 0 : (priv->fs.main.broadcast_enabled ? 1 : 0);
+ size = is_uc ? 0 : (priv->fs.l2.broadcast_enabled ? 1 : 0);
max_size = is_uc ?
1 << MLX5_CAP_GEN(priv->mdev, log_max_current_uc_list) :
1 << MLX5_CAP_GEN(priv->mdev, log_max_current_mc_list);
- addr_list = is_uc ? priv->fs.main.netdev_uc : priv->fs.main.netdev_mc;
+ addr_list = is_uc ? priv->fs.l2.netdev_uc : priv->fs.l2.netdev_mc;
mlx5e_for_each_hash_node(hn, tmp, addr_list, hi)
size++;
@@ -775,37 +423,37 @@ out:
static void mlx5e_vport_context_update(struct mlx5e_priv *priv)
{
- struct mlx5e_main_table *main_table = &priv->fs.main;
+ struct mlx5e_l2_table *ea = &priv->fs.l2;
mlx5e_vport_context_update_addr_list(priv, MLX5_NVPRT_LIST_TYPE_UC);
mlx5e_vport_context_update_addr_list(priv, MLX5_NVPRT_LIST_TYPE_MC);
mlx5_modify_nic_vport_promisc(priv->mdev, 0,
- main_table->allmulti_enabled,
- main_table->promisc_enabled);
+ ea->allmulti_enabled,
+ ea->promisc_enabled);
}
static void mlx5e_apply_netdev_addr(struct mlx5e_priv *priv)
{
- struct mlx5e_eth_addr_hash_node *hn;
+ struct mlx5e_l2_hash_node *hn;
struct hlist_node *tmp;
int i;
- mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_uc, i)
- mlx5e_execute_action(priv, hn);
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.l2.netdev_uc, i)
+ mlx5e_execute_l2_action(priv, hn);
- mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_mc, i)
- mlx5e_execute_action(priv, hn);
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.l2.netdev_mc, i)
+ mlx5e_execute_l2_action(priv, hn);
}
static void mlx5e_handle_netdev_addr(struct mlx5e_priv *priv)
{
- struct mlx5e_eth_addr_hash_node *hn;
+ struct mlx5e_l2_hash_node *hn;
struct hlist_node *tmp;
int i;
- mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_uc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.l2.netdev_uc, i)
hn->action = MLX5E_ACTION_DEL;
- mlx5e_for_each_hash_node(hn, tmp, priv->fs.main.netdev_mc, i)
+ mlx5e_for_each_hash_node(hn, tmp, priv->fs.l2.netdev_mc, i)
hn->action = MLX5E_ACTION_DEL;
if (!test_bit(MLX5E_STATE_DESTROYING, &priv->state))
@@ -819,7 +467,7 @@ void mlx5e_set_rx_mode_work(struct work_struct *work)
struct mlx5e_priv *priv = container_of(work, struct mlx5e_priv,
set_rx_mode_work);
- struct mlx5e_main_table *main_table = &priv->fs.main;
+ struct mlx5e_l2_table *ea = &priv->fs.l2;
struct net_device *ndev = priv->netdev;
bool rx_mode_enable = !test_bit(MLX5E_STATE_DESTROYING, &priv->state);
@@ -827,40 +475,40 @@ void mlx5e_set_rx_mode_work(struct work_struct *work)
bool allmulti_enabled = rx_mode_enable && (ndev->flags & IFF_ALLMULTI);
bool broadcast_enabled = rx_mode_enable;
- bool enable_promisc = !main_table->promisc_enabled && promisc_enabled;
- bool disable_promisc = main_table->promisc_enabled && !promisc_enabled;
- bool enable_allmulti = !main_table->allmulti_enabled && allmulti_enabled;
- bool disable_allmulti = main_table->allmulti_enabled && !allmulti_enabled;
- bool enable_broadcast = !main_table->broadcast_enabled && broadcast_enabled;
- bool disable_broadcast = main_table->broadcast_enabled && !broadcast_enabled;
+ bool enable_promisc = !ea->promisc_enabled && promisc_enabled;
+ bool disable_promisc = ea->promisc_enabled && !promisc_enabled;
+ bool enable_allmulti = !ea->allmulti_enabled && allmulti_enabled;
+ bool disable_allmulti = ea->allmulti_enabled && !allmulti_enabled;
+ bool enable_broadcast = !ea->broadcast_enabled && broadcast_enabled;
+ bool disable_broadcast = ea->broadcast_enabled && !broadcast_enabled;
if (enable_promisc) {
- mlx5e_add_eth_addr_rule(priv, &main_table->promisc, MLX5E_PROMISC);
+ mlx5e_add_l2_flow_rule(priv, &ea->promisc, MLX5E_PROMISC);
if (!priv->fs.vlan.filter_disabled)
mlx5e_add_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID,
0);
}
if (enable_allmulti)
- mlx5e_add_eth_addr_rule(priv, &main_table->allmulti, MLX5E_ALLMULTI);
+ mlx5e_add_l2_flow_rule(priv, &ea->allmulti, MLX5E_ALLMULTI);
if (enable_broadcast)
- mlx5e_add_eth_addr_rule(priv, &main_table->broadcast, MLX5E_FULLMATCH);
+ mlx5e_add_l2_flow_rule(priv, &ea->broadcast, MLX5E_FULLMATCH);
mlx5e_handle_netdev_addr(priv);
if (disable_broadcast)
- mlx5e_del_eth_addr_from_flow_table(priv, &main_table->broadcast);
+ mlx5e_del_l2_flow_rule(priv, &ea->broadcast);
if (disable_allmulti)
- mlx5e_del_eth_addr_from_flow_table(priv, &main_table->allmulti);
+ mlx5e_del_l2_flow_rule(priv, &ea->allmulti);
if (disable_promisc) {
if (!priv->fs.vlan.filter_disabled)
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_ANY_VID,
0);
- mlx5e_del_eth_addr_from_flow_table(priv, &main_table->promisc);
+ mlx5e_del_l2_flow_rule(priv, &ea->promisc);
}
- main_table->promisc_enabled = promisc_enabled;
- main_table->allmulti_enabled = allmulti_enabled;
- main_table->broadcast_enabled = broadcast_enabled;
+ ea->promisc_enabled = promisc_enabled;
+ ea->allmulti_enabled = allmulti_enabled;
+ ea->broadcast_enabled = broadcast_enabled;
mlx5e_vport_context_update(priv);
}
@@ -877,217 +525,446 @@ static void mlx5e_destroy_groups(struct mlx5e_flow_table *ft)
ft->num_groups = 0;
}
-void mlx5e_init_eth_addr(struct mlx5e_priv *priv)
+void mlx5e_init_l2_addr(struct mlx5e_priv *priv)
{
- ether_addr_copy(priv->fs.main.broadcast.addr, priv->netdev->broadcast);
+ ether_addr_copy(priv->fs.l2.broadcast.addr, priv->netdev->broadcast);
}
-#define MLX5E_MAIN_GROUP0_SIZE BIT(3)
-#define MLX5E_MAIN_GROUP1_SIZE BIT(1)
-#define MLX5E_MAIN_GROUP2_SIZE BIT(0)
-#define MLX5E_MAIN_GROUP3_SIZE BIT(14)
-#define MLX5E_MAIN_GROUP4_SIZE BIT(13)
-#define MLX5E_MAIN_GROUP5_SIZE BIT(11)
-#define MLX5E_MAIN_GROUP6_SIZE BIT(2)
-#define MLX5E_MAIN_GROUP7_SIZE BIT(1)
-#define MLX5E_MAIN_GROUP8_SIZE BIT(0)
-#define MLX5E_MAIN_TABLE_SIZE (MLX5E_MAIN_GROUP0_SIZE +\
- MLX5E_MAIN_GROUP1_SIZE +\
- MLX5E_MAIN_GROUP2_SIZE +\
- MLX5E_MAIN_GROUP3_SIZE +\
- MLX5E_MAIN_GROUP4_SIZE +\
- MLX5E_MAIN_GROUP5_SIZE +\
- MLX5E_MAIN_GROUP6_SIZE +\
- MLX5E_MAIN_GROUP7_SIZE +\
- MLX5E_MAIN_GROUP8_SIZE)
-
-static int __mlx5e_create_main_table_groups(struct mlx5e_flow_table *ft, u32 *in,
- int inlen)
+static void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft)
{
- u8 *mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
- u8 *dmac = MLX5_ADDR_OF(create_flow_group_in, in,
- match_criteria.outer_headers.dmac_47_16);
+ mlx5e_destroy_groups(ft);
+ kfree(ft->g);
+ mlx5_destroy_flow_table(ft->t);
+ ft->t = NULL;
+}
+
+static void mlx5e_cleanup_ttc_rules(struct mlx5e_ttc_table *ttc)
+{
+ int i;
+
+ for (i = 0; i < MLX5E_NUM_TT; i++) {
+ if (!IS_ERR_OR_NULL(ttc->rules[i])) {
+ mlx5_del_flow_rule(ttc->rules[i]);
+ ttc->rules[i] = NULL;
+ }
+ }
+}
+
+static struct {
+ u16 etype;
+ u8 proto;
+} ttc_rules[] = {
+ [MLX5E_TT_IPV4_TCP] = {
+ .etype = ETH_P_IP,
+ .proto = IPPROTO_TCP,
+ },
+ [MLX5E_TT_IPV6_TCP] = {
+ .etype = ETH_P_IPV6,
+ .proto = IPPROTO_TCP,
+ },
+ [MLX5E_TT_IPV4_UDP] = {
+ .etype = ETH_P_IP,
+ .proto = IPPROTO_UDP,
+ },
+ [MLX5E_TT_IPV6_UDP] = {
+ .etype = ETH_P_IPV6,
+ .proto = IPPROTO_UDP,
+ },
+ [MLX5E_TT_IPV4_IPSEC_AH] = {
+ .etype = ETH_P_IP,
+ .proto = IPPROTO_AH,
+ },
+ [MLX5E_TT_IPV6_IPSEC_AH] = {
+ .etype = ETH_P_IPV6,
+ .proto = IPPROTO_AH,
+ },
+ [MLX5E_TT_IPV4_IPSEC_ESP] = {
+ .etype = ETH_P_IP,
+ .proto = IPPROTO_ESP,
+ },
+ [MLX5E_TT_IPV6_IPSEC_ESP] = {
+ .etype = ETH_P_IPV6,
+ .proto = IPPROTO_ESP,
+ },
+ [MLX5E_TT_IPV4] = {
+ .etype = ETH_P_IP,
+ .proto = 0,
+ },
+ [MLX5E_TT_IPV6] = {
+ .etype = ETH_P_IPV6,
+ .proto = 0,
+ },
+ [MLX5E_TT_ANY] = {
+ .etype = 0,
+ .proto = 0,
+ },
+};
+
+static struct mlx5_flow_rule *mlx5e_generate_ttc_rule(struct mlx5e_priv *priv,
+ struct mlx5_flow_table *ft,
+ struct mlx5_flow_destination *dest,
+ u16 etype,
+ u8 proto)
+{
+ struct mlx5_flow_rule *rule;
+ u8 match_criteria_enable = 0;
+ u32 *match_criteria;
+ u32 *match_value;
+ int err = 0;
+
+ match_value = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ match_criteria = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ if (!match_value || !match_criteria) {
+ netdev_err(priv->netdev, "%s: alloc failed\n", __func__);
+ err = -ENOMEM;
+ goto out;
+ }
+
+ if (proto) {
+ match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria, outer_headers.ip_protocol);
+ MLX5_SET(fte_match_param, match_value, outer_headers.ip_protocol, proto);
+ }
+ if (etype) {
+ match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria, outer_headers.ethertype);
+ MLX5_SET(fte_match_param, match_value, outer_headers.ethertype, etype);
+ }
+
+ rule = mlx5_add_flow_rule(ft, match_criteria_enable,
+ match_criteria, match_value,
+ MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ MLX5_FS_DEFAULT_FLOW_TAG,
+ dest);
+ if (IS_ERR(rule)) {
+ err = PTR_ERR(rule);
+ netdev_err(priv->netdev, "%s: add rule failed\n", __func__);
+ }
+out:
+ kvfree(match_criteria);
+ kvfree(match_value);
+ return err ? ERR_PTR(err) : rule;
+}
+
+static int mlx5e_generate_ttc_table_rules(struct mlx5e_priv *priv)
+{
+ struct mlx5_flow_destination dest;
+ struct mlx5e_ttc_table *ttc;
+ struct mlx5_flow_rule **rules;
+ struct mlx5_flow_table *ft;
+ int tt;
int err;
+
+ ttc = &priv->fs.ttc;
+ ft = ttc->ft.t;
+ rules = ttc->rules;
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ for (tt = 0; tt < MLX5E_NUM_TT; tt++) {
+ if (tt == MLX5E_TT_ANY)
+ dest.tir_num = priv->direct_tir[0].tirn;
+ else
+ dest.tir_num = priv->indir_tirn[tt];
+ rules[tt] = mlx5e_generate_ttc_rule(priv, ft, &dest,
+ ttc_rules[tt].etype,
+ ttc_rules[tt].proto);
+ if (IS_ERR(rules[tt]))
+ goto del_rules;
+ }
+
+ return 0;
+
+del_rules:
+ err = PTR_ERR(rules[tt]);
+ rules[tt] = NULL;
+ mlx5e_cleanup_ttc_rules(ttc);
+ return err;
+}
+
+#define MLX5E_TTC_NUM_GROUPS 3
+#define MLX5E_TTC_GROUP1_SIZE BIT(3)
+#define MLX5E_TTC_GROUP2_SIZE BIT(1)
+#define MLX5E_TTC_GROUP3_SIZE BIT(0)
+#define MLX5E_TTC_TABLE_SIZE (MLX5E_TTC_GROUP1_SIZE +\
+ MLX5E_TTC_GROUP2_SIZE +\
+ MLX5E_TTC_GROUP3_SIZE)
+static int mlx5e_create_ttc_table_groups(struct mlx5e_ttc_table *ttc)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5e_flow_table *ft = &ttc->ft;
int ix = 0;
+ u32 *in;
+ int err;
+ u8 *mc;
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ip_protocol);
- MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP0_SIZE;
- MLX5_SET_CFG(in, end_flow_index, ix - 1);
- ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
- if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
- ft->num_groups++;
+ ft->g = kcalloc(MLX5E_TTC_NUM_GROUPS,
+ sizeof(*ft->g), GFP_KERNEL);
+ if (!ft->g)
+ return -ENOMEM;
+ in = mlx5_vzalloc(inlen);
+ if (!in) {
+ kfree(ft->g);
+ return -ENOMEM;
+ }
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
+ /* L4 Group */
+ mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+ MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ip_protocol);
MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
+ MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP1_SIZE;
+ ix += MLX5E_TTC_GROUP1_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
+ goto err;
ft->num_groups++;
- memset(in, 0, inlen);
+ /* L3 Group */
+ MLX5_SET(fte_match_param, mc, outer_headers.ip_protocol, 0);
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP2_SIZE;
+ ix += MLX5E_TTC_GROUP2_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
+ goto err;
ft->num_groups++;
+ /* Any Group */
memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ip_protocol);
- eth_broadcast_addr(dmac);
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP3_SIZE;
+ ix += MLX5E_TTC_GROUP3_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
+ goto err;
ft->num_groups++;
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
- eth_broadcast_addr(dmac);
- MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP4_SIZE;
- MLX5_SET_CFG(in, end_flow_index, ix - 1);
- ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
- if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
- ft->num_groups++;
+ kvfree(in);
+ return 0;
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- eth_broadcast_addr(dmac);
- MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP5_SIZE;
- MLX5_SET_CFG(in, end_flow_index, ix - 1);
- ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
- if (IS_ERR(ft->g[ft->num_groups]))
- goto err_destroy_groups;
- ft->num_groups++;
+err:
+ err = PTR_ERR(ft->g[ft->num_groups]);
+ ft->g[ft->num_groups] = NULL;
+ kvfree(in);
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ip_protocol);
- dmac[0] = 0x01;
+ return err;
+}
+
+static void mlx5e_destroy_ttc_table(struct mlx5e_priv *priv)
+{
+ struct mlx5e_ttc_table *ttc = &priv->fs.ttc;
+
+ mlx5e_cleanup_ttc_rules(ttc);
+ mlx5e_destroy_flow_table(&ttc->ft);
+}
+
+static int mlx5e_create_ttc_table(struct mlx5e_priv *priv)
+{
+ struct mlx5e_ttc_table *ttc = &priv->fs.ttc;
+ struct mlx5e_flow_table *ft = &ttc->ft;
+ int err;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
+ MLX5E_TTC_TABLE_SIZE, MLX5E_TTC_FT_LEVEL);
+ if (IS_ERR(ft->t)) {
+ err = PTR_ERR(ft->t);
+ ft->t = NULL;
+ return err;
+ }
+
+ err = mlx5e_create_ttc_table_groups(ttc);
+ if (err)
+ goto err;
+
+ err = mlx5e_generate_ttc_table_rules(priv);
+ if (err)
+ goto err;
+
+ return 0;
+err:
+ mlx5e_destroy_flow_table(ft);
+ return err;
+}
+
+static void mlx5e_del_l2_flow_rule(struct mlx5e_priv *priv,
+ struct mlx5e_l2_rule *ai)
+{
+ if (!IS_ERR_OR_NULL(ai->rule)) {
+ mlx5_del_flow_rule(ai->rule);
+ ai->rule = NULL;
+ }
+}
+
+static int mlx5e_add_l2_flow_rule(struct mlx5e_priv *priv,
+ struct mlx5e_l2_rule *ai, int type)
+{
+ struct mlx5_flow_table *ft = priv->fs.l2.ft.t;
+ struct mlx5_flow_destination dest;
+ u8 match_criteria_enable = 0;
+ u32 *match_criteria;
+ u32 *match_value;
+ int err = 0;
+ u8 *mc_dmac;
+ u8 *mv_dmac;
+
+ match_value = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ match_criteria = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ if (!match_value || !match_criteria) {
+ netdev_err(priv->netdev, "%s: alloc failed\n", __func__);
+ err = -ENOMEM;
+ goto add_l2_rule_out;
+ }
+
+ mc_dmac = MLX5_ADDR_OF(fte_match_param, match_criteria,
+ outer_headers.dmac_47_16);
+ mv_dmac = MLX5_ADDR_OF(fte_match_param, match_value,
+ outer_headers.dmac_47_16);
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
+ dest.ft = priv->fs.ttc.ft.t;
+
+ switch (type) {
+ case MLX5E_FULLMATCH:
+ match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ eth_broadcast_addr(mc_dmac);
+ ether_addr_copy(mv_dmac, ai->addr);
+ break;
+
+ case MLX5E_ALLMULTI:
+ match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ mc_dmac[0] = 0x01;
+ mv_dmac[0] = 0x01;
+ break;
+
+ case MLX5E_PROMISC:
+ break;
+ }
+
+ ai->rule = mlx5_add_flow_rule(ft, match_criteria_enable, match_criteria,
+ match_value,
+ MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ MLX5_FS_DEFAULT_FLOW_TAG, &dest);
+ if (IS_ERR(ai->rule)) {
+ netdev_err(priv->netdev, "%s: add l2 rule(mac:%pM) failed\n",
+ __func__, mv_dmac);
+ err = PTR_ERR(ai->rule);
+ ai->rule = NULL;
+ }
+
+add_l2_rule_out:
+ kvfree(match_criteria);
+ kvfree(match_value);
+
+ return err;
+}
+
+#define MLX5E_NUM_L2_GROUPS 3
+#define MLX5E_L2_GROUP1_SIZE BIT(0)
+#define MLX5E_L2_GROUP2_SIZE BIT(15)
+#define MLX5E_L2_GROUP3_SIZE BIT(0)
+#define MLX5E_L2_TABLE_SIZE (MLX5E_L2_GROUP1_SIZE +\
+ MLX5E_L2_GROUP2_SIZE +\
+ MLX5E_L2_GROUP3_SIZE)
+static int mlx5e_create_l2_table_groups(struct mlx5e_l2_table *l2_table)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ struct mlx5e_flow_table *ft = &l2_table->ft;
+ int ix = 0;
+ u8 *mc_dmac;
+ u32 *in;
+ int err;
+ u8 *mc;
+
+ ft->g = kcalloc(MLX5E_NUM_L2_GROUPS, sizeof(*ft->g), GFP_KERNEL);
+ if (!ft->g)
+ return -ENOMEM;
+ in = mlx5_vzalloc(inlen);
+ if (!in) {
+ kfree(ft->g);
+ return -ENOMEM;
+ }
+
+ mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+ mc_dmac = MLX5_ADDR_OF(fte_match_param, mc,
+ outer_headers.dmac_47_16);
+ /* Flow Group for promiscuous */
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP6_SIZE;
+ ix += MLX5E_L2_GROUP1_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
goto err_destroy_groups;
ft->num_groups++;
- memset(in, 0, inlen);
+ /* Flow Group for full match */
+ eth_broadcast_addr(mc_dmac);
MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- MLX5_SET_TO_ONES(fte_match_param, mc, outer_headers.ethertype);
- dmac[0] = 0x01;
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP7_SIZE;
+ ix += MLX5E_L2_GROUP2_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
goto err_destroy_groups;
ft->num_groups++;
- memset(in, 0, inlen);
- MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
- dmac[0] = 0x01;
+ /* Flow Group for allmulti */
+ eth_zero_addr(mc_dmac);
+ mc_dmac[0] = 0x01;
MLX5_SET_CFG(in, start_flow_index, ix);
- ix += MLX5E_MAIN_GROUP8_SIZE;
+ ix += MLX5E_L2_GROUP3_SIZE;
MLX5_SET_CFG(in, end_flow_index, ix - 1);
ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
if (IS_ERR(ft->g[ft->num_groups]))
goto err_destroy_groups;
ft->num_groups++;
+ kvfree(in);
return 0;
err_destroy_groups:
err = PTR_ERR(ft->g[ft->num_groups]);
ft->g[ft->num_groups] = NULL;
mlx5e_destroy_groups(ft);
+ kvfree(in);
return err;
}
-static int mlx5e_create_main_table_groups(struct mlx5e_flow_table *ft)
+static void mlx5e_destroy_l2_table(struct mlx5e_priv *priv)
{
- u32 *in;
- int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
- int err;
-
- in = mlx5_vzalloc(inlen);
- if (!in)
- return -ENOMEM;
-
- err = __mlx5e_create_main_table_groups(ft, in, inlen);
-
- kvfree(in);
- return err;
+ mlx5e_destroy_flow_table(&priv->fs.l2.ft);
}
-static int mlx5e_create_main_table(struct mlx5e_priv *priv)
+static int mlx5e_create_l2_table(struct mlx5e_priv *priv)
{
- struct mlx5e_flow_table *ft = &priv->fs.main.ft;
+ struct mlx5e_l2_table *l2_table = &priv->fs.l2;
+ struct mlx5e_flow_table *ft = &l2_table->ft;
int err;
ft->num_groups = 0;
ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
- MLX5E_MAIN_TABLE_SIZE, MLX5E_MAIN_FT_LEVEL);
+ MLX5E_L2_TABLE_SIZE, MLX5E_L2_FT_LEVEL);
if (IS_ERR(ft->t)) {
err = PTR_ERR(ft->t);
ft->t = NULL;
return err;
}
- ft->g = kcalloc(MLX5E_NUM_MAIN_GROUPS, sizeof(*ft->g), GFP_KERNEL);
- if (!ft->g) {
- err = -ENOMEM;
- goto err_destroy_main_table;
- }
- err = mlx5e_create_main_table_groups(ft);
+ err = mlx5e_create_l2_table_groups(l2_table);
if (err)
- goto err_free_g;
- return 0;
+ goto err_destroy_flow_table;
-err_free_g:
- kfree(ft->g);
+ return 0;
-err_destroy_main_table:
+err_destroy_flow_table:
mlx5_destroy_flow_table(ft->t);
ft->t = NULL;
return err;
}
-static void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft)
-{
- mlx5e_destroy_groups(ft);
- kfree(ft->g);
- mlx5_destroy_flow_table(ft->t);
- ft->t = NULL;
-}
-
-static void mlx5e_destroy_main_table(struct mlx5e_priv *priv)
-{
- mlx5e_destroy_flow_table(&priv->fs.main.ft);
-}
-
#define MLX5E_NUM_VLAN_GROUPS 2
#define MLX5E_VLAN_GROUP0_SIZE BIT(12)
#define MLX5E_VLAN_GROUP1_SIZE BIT(1)
@@ -1206,18 +1083,33 @@ int mlx5e_create_flow_steering(struct mlx5e_priv *priv)
if (!priv->fs.ns)
return -EINVAL;
- err = mlx5e_create_main_table(priv);
- if (err)
+ err = mlx5e_create_ttc_table(priv);
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
+ err);
return err;
+ }
+
+ err = mlx5e_create_l2_table(priv);
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create l2 table, err=%d\n",
+ err);
+ goto err_destroy_ttc_table;
+ }
err = mlx5e_create_vlan_table(priv);
- if (err)
- goto err_destroy_main_table;
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create vlan table, err=%d\n",
+ err);
+ goto err_destroy_l2_table;
+ }
return 0;
-err_destroy_main_table:
- mlx5e_destroy_main_table(priv);
+err_destroy_l2_table:
+ mlx5e_destroy_l2_table(priv);
+err_destroy_ttc_table:
+ mlx5e_destroy_ttc_table(priv);
return err;
}
@@ -1226,5 +1118,6 @@ void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv)
{
mlx5e_del_vlan_rule(priv, MLX5E_VLAN_RULE_TYPE_UNTAGGED, 0);
mlx5e_destroy_vlan_table(priv);
- mlx5e_destroy_main_table(priv);
+ mlx5e_destroy_l2_table(priv);
+ mlx5e_destroy_ttc_table(priv);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 3fad66f..07c596d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2975,7 +2975,7 @@ static void *mlx5e_create_netdev(struct mlx5_core_dev *mdev)
mlx5e_create_q_counter(priv);
- mlx5e_init_eth_addr(priv);
+ mlx5e_init_l2_addr(priv);
mlx5e_vxlan_init(priv);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index 2b82293..83fe864 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -74,7 +74,7 @@
#define BY_PASS_MIN_LEVEL (KERNEL_MIN_LEVEL + MLX5_BY_PASS_NUM_PRIOS +\
LEFTOVERS_NUM_PRIOS)
-#define KERNEL_NIC_PRIO_NUM_LEVELS 2
+#define KERNEL_NIC_PRIO_NUM_LEVELS 3
#define KERNEL_NIC_NUM_PRIOS 1
/* One more level for tc */
#define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 09/12] net/mlx5: Initializing CPU reverse mapping
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (7 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 08/12] net/mlx5e: Split the main flow steering table Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables Saeed Mahameed
` (3 subsequent siblings)
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Allocating CPU rmap and add entry for each IRQ.
CPU rmap is used in aRFS to get the RX queue number
of the RX completion interrupts.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 +++
drivers/net/ethernet/mellanox/mlx5/core/main.c | 18 ++++++++++++++++++
include/linux/mlx5/driver.h | 3 +++
3 files changed, 24 insertions(+), 0 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 07c596d..a2b3297 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -1689,6 +1689,9 @@ int mlx5e_open_locked(struct net_device *netdev)
mlx5e_redirect_rqts(priv);
mlx5e_update_carrier(priv);
mlx5e_timestamp_init(priv);
+#ifdef CONFIG_RFS_ACCEL
+ priv->netdev->rx_cpu_rmap = priv->mdev->rmap;
+#endif
schedule_delayed_work(&priv->update_stats_work, 0);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 6892746..6feef7f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -48,6 +48,9 @@
#include <linux/kmod.h>
#include <linux/delay.h>
#include <linux/mlx5/mlx5_ifc.h>
+#ifdef CONFIG_RFS_ACCEL
+#include <linux/cpu_rmap.h>
+#endif
#include "mlx5_core.h"
#include "fs_core.h"
#ifdef CONFIG_MLX5_CORE_EN
@@ -665,6 +668,12 @@ static void free_comp_eqs(struct mlx5_core_dev *dev)
struct mlx5_eq_table *table = &dev->priv.eq_table;
struct mlx5_eq *eq, *n;
+#ifdef CONFIG_RFS_ACCEL
+ if (dev->rmap) {
+ free_irq_cpu_rmap(dev->rmap);
+ dev->rmap = NULL;
+ }
+#endif
spin_lock(&table->lock);
list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
list_del(&eq->list);
@@ -691,6 +700,11 @@ static int alloc_comp_eqs(struct mlx5_core_dev *dev)
INIT_LIST_HEAD(&table->comp_eqs_list);
ncomp_vec = table->num_comp_vectors;
nent = MLX5_COMP_EQ_SIZE;
+#ifdef CONFIG_RFS_ACCEL
+ dev->rmap = alloc_irq_cpu_rmap(ncomp_vec);
+ if (!dev->rmap)
+ return -ENOMEM;
+#endif
for (i = 0; i < ncomp_vec; i++) {
eq = kzalloc(sizeof(*eq), GFP_KERNEL);
if (!eq) {
@@ -698,6 +712,10 @@ static int alloc_comp_eqs(struct mlx5_core_dev *dev)
goto clean;
}
+#ifdef CONFIG_RFS_ACCEL
+ irq_cpu_rmap_add(dev->rmap,
+ dev->priv.msix_arr[i + MLX5_EQ_VEC_COMP_BASE].vector);
+#endif
snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_comp%d", i);
err = mlx5_create_map_eq(dev, eq,
i + MLX5_EQ_VEC_COMP_BASE, nent, 0,
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 96a428d..d552944 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -560,6 +560,9 @@ struct mlx5_core_dev {
struct mlx5_profile *profile;
atomic_t num_qps;
u32 issi;
+#ifdef CONFIG_RFS_ACCEL
+ struct cpu_rmap *rmap;
+#endif
};
struct mlx5_db {
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (8 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 09/12] net/mlx5: Initializing CPU reverse mapping Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-05-01 2:18 ` Alexei Starovoitov
2016-04-28 22:36 ` [PATCH net-next 11/12] net/mlx5e: Add accelerated RFS support Saeed Mahameed
` (2 subsequent siblings)
12 siblings, 1 reply; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Create the following four flow tables for aRFS usage:
1. IPv4 TCP - filtering 4-tuple of IPv4 TCP packets.
2. IPv6 TCP - filtering 4-tuple of IPv6 TCP packets.
3. IPv4 UDP - filtering 4-tuple of IPv4 UDP packets.
4. IPv6 UDP - filtering 4-tuple of IPv6 UDP packets.
Each flow table has two flow groups: one for the 4-tuple
filtering (full match) and the other contains * rule for miss rule.
Full match rule means a hit for aRFS and packet will be forwarded
to the dedicated RQ/Core, miss rule packets will be forwarded to
default RSS hashing.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/Makefile | 1 +
drivers/net/ethernet/mellanox/mlx5/core/en.h | 41 ++++
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 251 +++++++++++++++++++++
drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 23 +-
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 +-
drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 3 +-
6 files changed, 313 insertions(+), 14 deletions(-)
create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
index 4fc45ee..679e18f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
+++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
@@ -9,3 +9,4 @@ mlx5_core-$(CONFIG_MLX5_CORE_EN) += wq.o eswitch.o \
en_txrx.o en_clock.o vxlan.o en_tc.o
mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o
+mlx5_core-$(CONFIG_RFS_ACCEL) += en_arfs.o
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 2c9879c..999e058 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -48,6 +48,8 @@
#include "mlx5_core.h"
#include "en_stats.h"
+#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
+
#define MLX5E_MAX_NUM_TC 8
#define MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE 0x6
@@ -446,12 +448,38 @@ struct mlx5e_ttc_table {
struct mlx5_flow_rule *rules[MLX5E_NUM_TT];
};
+struct arfs_table {
+ struct mlx5e_flow_table ft;
+ struct mlx5_flow_rule *default_rule;
+};
+
+enum arfs_type {
+ ARFS_IPV4_TCP,
+ ARFS_IPV6_TCP,
+ ARFS_IPV4_UDP,
+ ARFS_IPV6_UDP,
+ ARFS_NUM_TYPES,
+};
+
+struct mlx5e_arfs_tables {
+ struct arfs_table arfs_tables[ARFS_NUM_TYPES];
+};
+
+/* NIC prio FTS */
+enum {
+ MLX5E_VLAN_FT_LEVEL = 0,
+ MLX5E_L2_FT_LEVEL,
+ MLX5E_TTC_FT_LEVEL,
+ MLX5E_ARFS_FT_LEVEL
+};
+
struct mlx5e_flow_steering {
struct mlx5_flow_namespace *ns;
struct mlx5e_tc_table tc;
struct mlx5e_vlan_table vlan;
struct mlx5e_l2_table l2;
struct mlx5e_ttc_table ttc;
+ struct mlx5e_arfs_tables arfs;
};
struct mlx5e_direct_tir {
@@ -570,6 +598,7 @@ void mlx5e_update_stats(struct mlx5e_priv *priv);
int mlx5e_create_flow_steering(struct mlx5e_priv *priv);
void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv);
void mlx5e_init_l2_addr(struct mlx5e_priv *priv);
+void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft);
void mlx5e_set_rx_mode_work(struct work_struct *work);
void mlx5e_fill_hwstamp(struct mlx5e_tstamp *clock, u64 timestamp,
@@ -646,6 +675,18 @@ extern const struct dcbnl_rtnl_ops mlx5e_dcbnl_ops;
int mlx5e_dcbnl_ieee_setets_core(struct mlx5e_priv *priv, struct ieee_ets *ets);
#endif
+#ifndef CONFIG_RFS_ACCEL
+static inline int mlx5e_arfs_create_tables(struct mlx5e_priv *priv)
+{
+ return 0;
+}
+
+static inline void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv) {}
+#else
+int mlx5e_arfs_create_tables(struct mlx5e_priv *priv);
+void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv);
+#endif
+
u16 mlx5e_get_max_inline_cap(struct mlx5_core_dev *mdev);
#endif /* __MLX5_EN_H__ */
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
new file mode 100644
index 0000000..cd50419
--- /dev/null
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -0,0 +1,251 @@
+/*
+ * Copyright (c) 2016, Mellanox Technologies. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include "en.h"
+#include <linux/mlx5/fs.h>
+
+static void arfs_destroy_table(struct arfs_table *arfs_t)
+{
+ mlx5_del_flow_rule(arfs_t->default_rule);
+ mlx5e_destroy_flow_table(&arfs_t->ft);
+}
+
+void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv)
+{
+ int i;
+
+ if (!(priv->netdev->hw_features & NETIF_F_NTUPLE))
+ return;
+ for (i = 0; i < ARFS_NUM_TYPES; i++) {
+ if (!IS_ERR_OR_NULL(priv->fs.arfs.arfs_tables[i].ft.t))
+ arfs_destroy_table(&priv->fs.arfs.arfs_tables[i]);
+ }
+}
+
+static int arfs_add_default_rule(struct mlx5e_priv *priv,
+ enum arfs_type type)
+{
+ struct arfs_table *arfs_t = &priv->fs.arfs.arfs_tables[type];
+ struct mlx5_flow_destination dest;
+ u8 match_criteria_enable = 0;
+ u32 *tirn = priv->indir_tirn;
+ u32 *match_criteria;
+ u32 *match_value;
+ int err = 0;
+
+ match_value = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ match_criteria = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ if (!match_value || !match_criteria) {
+ netdev_err(priv->netdev, "%s: alloc failed\n", __func__);
+ err = -ENOMEM;
+ goto out;
+ }
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ switch (type) {
+ case ARFS_IPV4_TCP:
+ dest.tir_num = tirn[MLX5E_TT_IPV4_TCP];
+ break;
+ case ARFS_IPV4_UDP:
+ dest.tir_num = tirn[MLX5E_TT_IPV4_UDP];
+ break;
+ case ARFS_IPV6_TCP:
+ dest.tir_num = tirn[MLX5E_TT_IPV6_TCP];
+ break;
+ case ARFS_IPV6_UDP:
+ dest.tir_num = tirn[MLX5E_TT_IPV6_UDP];
+ break;
+ default:
+ err = -EINVAL;
+ goto out;
+ }
+
+ arfs_t->default_rule = mlx5_add_flow_rule(arfs_t->ft.t, match_criteria_enable,
+ match_criteria, match_value,
+ MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ MLX5_FS_DEFAULT_FLOW_TAG,
+ &dest);
+ if (IS_ERR(arfs_t->default_rule)) {
+ err = PTR_ERR(arfs_t->default_rule);
+ arfs_t->default_rule = NULL;
+ netdev_err(priv->netdev, "%s: add rule failed, arfs type=%d\n",
+ __func__, type);
+ }
+out:
+ kvfree(match_criteria);
+ kvfree(match_value);
+ return err;
+}
+
+#define MLX5E_ARFS_NUM_GROUPS 2
+#define MLX5E_ARFS_GROUP1_SIZE BIT(12)
+#define MLX5E_ARFS_GROUP2_SIZE BIT(0)
+#define MLX5E_ARFS_TABLE_SIZE (MLX5E_ARFS_GROUP1_SIZE +\
+ MLX5E_ARFS_GROUP2_SIZE)
+static int arfs_create_groups(struct mlx5e_flow_table *ft,
+ enum arfs_type type)
+{
+ int inlen = MLX5_ST_SZ_BYTES(create_flow_group_in);
+ void *outer_headers_c;
+ int ix = 0;
+ u32 *in;
+ int err;
+ u8 *mc;
+
+ ft->g = kcalloc(MLX5E_ARFS_NUM_GROUPS,
+ sizeof(*ft->g), GFP_KERNEL);
+ in = mlx5_vzalloc(inlen);
+ if (!in || !ft->g) {
+ kvfree(ft->g);
+ kvfree(in);
+ return -ENOMEM;
+ }
+
+ mc = MLX5_ADDR_OF(create_flow_group_in, in, match_criteria);
+ outer_headers_c = MLX5_ADDR_OF(fte_match_param, mc,
+ outer_headers);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, ethertype);
+ switch (type) {
+ case ARFS_IPV4_TCP:
+ case ARFS_IPV6_TCP:
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_dport);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, tcp_sport);
+ break;
+ case ARFS_IPV4_UDP:
+ case ARFS_IPV6_UDP:
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, udp_dport);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c, udp_sport);
+ break;
+ default:
+ err = -EINVAL;
+ goto out;
+ }
+
+ switch (type) {
+ case ARFS_IPV4_TCP:
+ case ARFS_IPV4_UDP:
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c,
+ src_ipv4_src_ipv6.ipv4_layout.ipv4);
+ MLX5_SET_TO_ONES(fte_match_set_lyr_2_4, outer_headers_c,
+ dst_ipv4_dst_ipv6.ipv4_layout.ipv4);
+ break;
+ case ARFS_IPV6_TCP:
+ case ARFS_IPV6_UDP:
+ memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
+ src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+ memset(MLX5_ADDR_OF(fte_match_set_lyr_2_4, outer_headers_c,
+ dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ 0xff, 16);
+ break;
+ default:
+ err = -EINVAL;
+ goto out;
+ }
+
+ MLX5_SET_CFG(in, match_criteria_enable, MLX5_MATCH_OUTER_HEADERS);
+ MLX5_SET_CFG(in, start_flow_index, ix);
+ ix += MLX5E_ARFS_GROUP1_SIZE;
+ MLX5_SET_CFG(in, end_flow_index, ix - 1);
+ ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
+ if (IS_ERR(ft->g[ft->num_groups]))
+ goto err;
+ ft->num_groups++;
+
+ memset(in, 0, inlen);
+ MLX5_SET_CFG(in, start_flow_index, ix);
+ ix += MLX5E_ARFS_GROUP2_SIZE;
+ MLX5_SET_CFG(in, end_flow_index, ix - 1);
+ ft->g[ft->num_groups] = mlx5_create_flow_group(ft->t, in);
+ if (IS_ERR(ft->g[ft->num_groups]))
+ goto err;
+ ft->num_groups++;
+
+ kvfree(in);
+ return 0;
+
+err:
+ err = PTR_ERR(ft->g[ft->num_groups]);
+ ft->g[ft->num_groups] = NULL;
+out:
+ kvfree(in);
+
+ return err;
+}
+
+static int arfs_create_table(struct mlx5e_priv *priv,
+ enum arfs_type type)
+{
+ struct mlx5e_arfs_tables *arfs = &priv->fs.arfs;
+ struct mlx5e_flow_table *ft = &arfs->arfs_tables[type].ft;
+ int err;
+
+ ft->t = mlx5_create_flow_table(priv->fs.ns, MLX5E_NIC_PRIO,
+ MLX5E_ARFS_TABLE_SIZE, MLX5E_ARFS_FT_LEVEL);
+ if (IS_ERR(ft->t)) {
+ err = PTR_ERR(ft->t);
+ ft->t = NULL;
+ return err;
+ }
+
+ err = arfs_create_groups(ft, type);
+ if (err)
+ goto err;
+
+ err = arfs_add_default_rule(priv, type);
+ if (err)
+ goto err;
+
+ return 0;
+err:
+ mlx5e_destroy_flow_table(ft);
+ return err;
+}
+
+int mlx5e_arfs_create_tables(struct mlx5e_priv *priv)
+{
+ int err = 0;
+ int i;
+
+ if (!(priv->netdev->hw_features & NETIF_F_NTUPLE))
+ return 0;
+
+ for (i = 0; i < ARFS_NUM_TYPES; i++) {
+ err = arfs_create_table(priv, i);
+ if (err)
+ goto err;
+ }
+ return 0;
+err:
+ mlx5e_arfs_destroy_tables(priv);
+ return err;
+}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
index 6e353b3..b327400 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_fs.c
@@ -42,15 +42,6 @@ static int mlx5e_add_l2_flow_rule(struct mlx5e_priv *priv,
static void mlx5e_del_l2_flow_rule(struct mlx5e_priv *priv,
struct mlx5e_l2_rule *ai);
-/* NIC prio FTS */
-enum {
- MLX5E_VLAN_FT_LEVEL = 0,
- MLX5E_L2_FT_LEVEL,
- MLX5E_TTC_FT_LEVEL
-};
-
-#define MLX5_SET_CFG(p, f, v) MLX5_SET(create_flow_group_in, p, f, v)
-
enum {
MLX5E_FULLMATCH = 0,
MLX5E_ALLMULTI = 1,
@@ -530,7 +521,7 @@ void mlx5e_init_l2_addr(struct mlx5e_priv *priv)
ether_addr_copy(priv->fs.l2.broadcast.addr, priv->netdev->broadcast);
}
-static void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft)
+void mlx5e_destroy_flow_table(struct mlx5e_flow_table *ft)
{
mlx5e_destroy_groups(ft);
kfree(ft->g);
@@ -1083,11 +1074,18 @@ int mlx5e_create_flow_steering(struct mlx5e_priv *priv)
if (!priv->fs.ns)
return -EINVAL;
+ err = mlx5e_arfs_create_tables(priv);
+ if (err) {
+ netdev_err(priv->netdev, "Failed to create arfs tables, err=%d\n",
+ err);
+ priv->netdev->hw_features &= ~NETIF_F_NTUPLE;
+ }
+
err = mlx5e_create_ttc_table(priv);
if (err) {
netdev_err(priv->netdev, "Failed to create ttc table, err=%d\n",
err);
- return err;
+ goto err_destroy_arfs_tables;
}
err = mlx5e_create_l2_table(priv);
@@ -1110,6 +1108,8 @@ err_destroy_l2_table:
mlx5e_destroy_l2_table(priv);
err_destroy_ttc_table:
mlx5e_destroy_ttc_table(priv);
+err_destroy_arfs_tables:
+ mlx5e_arfs_destroy_tables(priv);
return err;
}
@@ -1120,4 +1120,5 @@ void mlx5e_destroy_flow_steering(struct mlx5e_priv *priv)
mlx5e_destroy_vlan_table(priv);
mlx5e_destroy_l2_table(priv);
mlx5e_destroy_ttc_table(priv);
+ mlx5e_arfs_destroy_tables(priv);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index a2b3297..5d64534 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2801,8 +2801,12 @@ static void mlx5e_build_netdev(struct net_device *netdev)
if (FT_CAP(flow_modify_en) &&
FT_CAP(modify_root) &&
FT_CAP(identified_miss_table_mode) &&
- FT_CAP(flow_table_modify))
- priv->netdev->hw_features |= NETIF_F_HW_TC;
+ FT_CAP(flow_table_modify)) {
+ netdev->hw_features |= NETIF_F_HW_TC;
+#ifdef CONFIG_RFS_ACCEL
+ netdev->hw_features |= NETIF_F_NTUPLE;
+#endif
+ }
netdev->features |= NETIF_F_HIGHDMA;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index 83fe864..4d78d5a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -74,7 +74,8 @@
#define BY_PASS_MIN_LEVEL (KERNEL_MIN_LEVEL + MLX5_BY_PASS_NUM_PRIOS +\
LEFTOVERS_NUM_PRIOS)
-#define KERNEL_NIC_PRIO_NUM_LEVELS 3
+/* Vlan, mac, ttc, aRFS */
+#define KERNEL_NIC_PRIO_NUM_LEVELS 4
#define KERNEL_NIC_NUM_PRIOS 1
/* One more level for tc */
#define KERNEL_MIN_LEVEL (KERNEL_NIC_PRIO_NUM_LEVELS + 1)
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 11/12] net/mlx5e: Add accelerated RFS support
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (9 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 12/12] net/mlx5e: Enabling aRFS mechanism Saeed Mahameed
2016-04-29 20:31 ` [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support David Miller
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Implement ndo_rx_flow_steer ndo.
A new flow steering rule will be composed from the
skb 4-tuple and added to the hardware aRFS flow table.
Each rule is stored in an internal hash table, if such
skb 4-tuple rule already exists we update the corresponding
hardware steering rule with the new destination.
For garbage collection rps_may_expire_flow will be
invoked for a limited amount of old rules upon any
ndo_rx_flow_steer invocation.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 10 +
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 427 ++++++++++++++++++++-
2 files changed, 436 insertions(+), 1 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 999e058..21c3841 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -448,9 +448,12 @@ struct mlx5e_ttc_table {
struct mlx5_flow_rule *rules[MLX5E_NUM_TT];
};
+#define ARFS_HASH_SHIFT BITS_PER_BYTE
+#define ARFS_HASH_SIZE BIT(BITS_PER_BYTE)
struct arfs_table {
struct mlx5e_flow_table ft;
struct mlx5_flow_rule *default_rule;
+ struct hlist_head rules_hash[ARFS_HASH_SIZE];
};
enum arfs_type {
@@ -463,6 +466,11 @@ enum arfs_type {
struct mlx5e_arfs_tables {
struct arfs_table arfs_tables[ARFS_NUM_TYPES];
+ /* Protect aRFS rules list */
+ spinlock_t arfs_lock;
+ struct list_head rules;
+ int last_filter_id;
+ struct workqueue_struct *wq;
};
/* NIC prio FTS */
@@ -685,6 +693,8 @@ static inline void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv) {}
#else
int mlx5e_arfs_create_tables(struct mlx5e_priv *priv);
void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv);
+int mlx5e_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
+ u16 rxq_index, u32 flow_id);
#endif
u16 mlx5e_get_max_inline_cap(struct mlx5_core_dev *mdev);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
index cd50419..e54fbc1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -30,8 +30,47 @@
* SOFTWARE.
*/
-#include "en.h"
+#include <linux/hash.h>
#include <linux/mlx5/fs.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include "en.h"
+
+struct arfs_tuple {
+ __be16 etype;
+ u8 ip_proto;
+ union {
+ __be32 src_ipv4;
+ struct in6_addr src_ipv6;
+ };
+ union {
+ __be32 dst_ipv4;
+ struct in6_addr dst_ipv6;
+ };
+ __be16 src_port;
+ __be16 dst_port;
+};
+
+struct arfs_rule {
+ struct mlx5e_priv *priv;
+ struct work_struct arfs_work;
+ struct mlx5_flow_rule *rule;
+ struct hlist_node hlist;
+ int rxq;
+ /* Flow ID passed to ndo_rx_flow_steer */
+ int flow_id;
+ /* Filter ID returned by ndo_rx_flow_steer */
+ int filter_id;
+ struct arfs_tuple tuple;
+};
+
+#define mlx5e_for_each_arfs_rule(hn, tmp, arfs_tables, i, j) \
+ for (i = 0; i < ARFS_NUM_TYPES; i++) \
+ mlx5e_for_each_hash_arfs_rule(hn, tmp, arfs_tables[i].rules_hash, j)
+
+#define mlx5e_for_each_hash_arfs_rule(hn, tmp, hash, j) \
+ for (j = 0; j < ARFS_HASH_SIZE; j++) \
+ hlist_for_each_entry_safe(hn, tmp, &hash[j], hlist)
static void arfs_destroy_table(struct arfs_table *arfs_t)
{
@@ -39,12 +78,17 @@ static void arfs_destroy_table(struct arfs_table *arfs_t)
mlx5e_destroy_flow_table(&arfs_t->ft);
}
+static void arfs_del_rules(struct mlx5e_priv *priv);
+
void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv)
{
int i;
if (!(priv->netdev->hw_features & NETIF_F_NTUPLE))
return;
+
+ arfs_del_rules(priv);
+ destroy_workqueue(priv->fs.arfs.wq);
for (i = 0; i < ARFS_NUM_TYPES; i++) {
if (!IS_ERR_OR_NULL(priv->fs.arfs.arfs_tables[i].ft.t))
arfs_destroy_table(&priv->fs.arfs.arfs_tables[i]);
@@ -239,6 +283,12 @@ int mlx5e_arfs_create_tables(struct mlx5e_priv *priv)
if (!(priv->netdev->hw_features & NETIF_F_NTUPLE))
return 0;
+ spin_lock_init(&priv->fs.arfs.arfs_lock);
+ INIT_LIST_HEAD(&priv->fs.arfs.rules);
+ priv->fs.arfs.wq = create_singlethread_workqueue("mlx5e_arfs");
+ if (!priv->fs.arfs.wq)
+ return -ENOMEM;
+
for (i = 0; i < ARFS_NUM_TYPES; i++) {
err = arfs_create_table(priv, i);
if (err)
@@ -249,3 +299,378 @@ err:
mlx5e_arfs_destroy_tables(priv);
return err;
}
+
+#define MLX5E_ARFS_EXPIRY_QUOTA 60
+
+static void arfs_may_expire_flow(struct mlx5e_priv *priv)
+{
+ struct arfs_rule *arfs_rule;
+ struct hlist_node *htmp;
+ int quota = 0;
+ int i;
+ int j;
+
+ HLIST_HEAD(del_list);
+ spin_lock_bh(&priv->fs.arfs.arfs_lock);
+ mlx5e_for_each_arfs_rule(arfs_rule, htmp, priv->fs.arfs.arfs_tables, i, j) {
+ if (quota++ > MLX5E_ARFS_EXPIRY_QUOTA)
+ break;
+ if (!work_pending(&arfs_rule->arfs_work) &&
+ rps_may_expire_flow(priv->netdev,
+ arfs_rule->rxq, arfs_rule->flow_id,
+ arfs_rule->filter_id)) {
+ hlist_del_init(&arfs_rule->hlist);
+ hlist_add_head(&arfs_rule->hlist, &del_list);
+ }
+ }
+ spin_unlock_bh(&priv->fs.arfs.arfs_lock);
+ hlist_for_each_entry_safe(arfs_rule, htmp, &del_list, hlist) {
+ if (arfs_rule->rule)
+ mlx5_del_flow_rule(arfs_rule->rule);
+ hlist_del(&arfs_rule->hlist);
+ kfree(arfs_rule);
+ }
+}
+
+static void arfs_del_rules(struct mlx5e_priv *priv)
+{
+ struct hlist_node *htmp;
+ struct arfs_rule *rule;
+ int i;
+ int j;
+
+ HLIST_HEAD(del_list);
+ spin_lock_bh(&priv->fs.arfs.arfs_lock);
+ mlx5e_for_each_arfs_rule(rule, htmp, priv->fs.arfs.arfs_tables, i, j) {
+ hlist_del_init(&rule->hlist);
+ hlist_add_head(&rule->hlist, &del_list);
+ }
+ spin_unlock_bh(&priv->fs.arfs.arfs_lock);
+
+ hlist_for_each_entry_safe(rule, htmp, &del_list, hlist) {
+ cancel_work_sync(&rule->arfs_work);
+ if (rule->rule)
+ mlx5_del_flow_rule(rule->rule);
+ hlist_del(&rule->hlist);
+ kfree(rule);
+ }
+}
+
+static struct hlist_head *
+arfs_hash_bucket(struct arfs_table *arfs_t, __be16 src_port,
+ __be16 dst_port)
+{
+ unsigned long l;
+ int bucket_idx;
+
+ l = (__force unsigned long)src_port |
+ ((__force unsigned long)dst_port << 2);
+
+ bucket_idx = hash_long(l, ARFS_HASH_SHIFT);
+
+ return &arfs_t->rules_hash[bucket_idx];
+}
+
+static u8 arfs_get_ip_proto(const struct sk_buff *skb)
+{
+ return (skb->protocol == htons(ETH_P_IP)) ?
+ ip_hdr(skb)->protocol : ipv6_hdr(skb)->nexthdr;
+}
+
+static struct arfs_table *arfs_get_table(struct mlx5e_arfs_tables *arfs,
+ u8 ip_proto, __be16 etype)
+{
+ if (etype == htons(ETH_P_IP) && ip_proto == IPPROTO_TCP)
+ return &arfs->arfs_tables[ARFS_IPV4_TCP];
+ if (etype == htons(ETH_P_IP) && ip_proto == IPPROTO_UDP)
+ return &arfs->arfs_tables[ARFS_IPV4_UDP];
+ if (etype == htons(ETH_P_IPV6) && ip_proto == IPPROTO_TCP)
+ return &arfs->arfs_tables[ARFS_IPV6_TCP];
+ if (etype == htons(ETH_P_IPV6) && ip_proto == IPPROTO_UDP)
+ return &arfs->arfs_tables[ARFS_IPV6_UDP];
+
+ return NULL;
+}
+
+static struct mlx5_flow_rule *arfs_add_rule(struct mlx5e_priv *priv,
+ struct arfs_rule *arfs_rule)
+{
+ struct mlx5e_arfs_tables *arfs = &priv->fs.arfs;
+ struct arfs_tuple *tuple = &arfs_rule->tuple;
+ struct mlx5_flow_rule *rule = NULL;
+ struct mlx5_flow_destination dest;
+ struct arfs_table *arfs_table;
+ u8 match_criteria_enable = 0;
+ struct mlx5_flow_table *ft;
+ u32 *match_criteria;
+ u32 *match_value;
+ int err = 0;
+
+ match_value = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ match_criteria = mlx5_vzalloc(MLX5_ST_SZ_BYTES(fte_match_param));
+ if (!match_value || !match_criteria) {
+ netdev_err(priv->netdev, "%s: alloc failed\n", __func__);
+ err = -ENOMEM;
+ goto out;
+ }
+ match_criteria_enable = MLX5_MATCH_OUTER_HEADERS;
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.ethertype);
+ MLX5_SET(fte_match_param, match_value, outer_headers.ethertype,
+ ntohs(tuple->etype));
+ arfs_table = arfs_get_table(arfs, tuple->ip_proto, tuple->etype);
+ if (!arfs_table) {
+ err = -EINVAL;
+ goto out;
+ }
+
+ ft = arfs_table->ft.t;
+ if (tuple->ip_proto == IPPROTO_TCP) {
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.tcp_dport);
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.tcp_sport);
+ MLX5_SET(fte_match_param, match_value, outer_headers.tcp_dport,
+ ntohs(tuple->dst_port));
+ MLX5_SET(fte_match_param, match_value, outer_headers.tcp_sport,
+ ntohs(tuple->src_port));
+ } else {
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.udp_dport);
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.udp_sport);
+ MLX5_SET(fte_match_param, match_value, outer_headers.udp_dport,
+ ntohs(tuple->dst_port));
+ MLX5_SET(fte_match_param, match_value, outer_headers.udp_sport,
+ ntohs(tuple->src_port));
+ }
+ if (tuple->etype == htons(ETH_P_IP)) {
+ memcpy(MLX5_ADDR_OF(fte_match_param, match_value,
+ outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4),
+ &tuple->src_ipv4,
+ 4);
+ memcpy(MLX5_ADDR_OF(fte_match_param, match_value,
+ outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4),
+ &tuple->dst_ipv4,
+ 4);
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.src_ipv4_src_ipv6.ipv4_layout.ipv4);
+ MLX5_SET_TO_ONES(fte_match_param, match_criteria,
+ outer_headers.dst_ipv4_dst_ipv6.ipv4_layout.ipv4);
+ } else {
+ memcpy(MLX5_ADDR_OF(fte_match_param, match_value,
+ outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ &tuple->src_ipv6,
+ 16);
+ memcpy(MLX5_ADDR_OF(fte_match_param, match_value,
+ outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ &tuple->dst_ipv6,
+ 16);
+ memset(MLX5_ADDR_OF(fte_match_param, match_criteria,
+ outer_headers.src_ipv4_src_ipv6.ipv6_layout.ipv6),
+ 0xff,
+ 16);
+ memset(MLX5_ADDR_OF(fte_match_param, match_criteria,
+ outer_headers.dst_ipv4_dst_ipv6.ipv6_layout.ipv6),
+ 0xff,
+ 16);
+ }
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ dest.tir_num = priv->direct_tir[arfs_rule->rxq].tirn;
+ rule = mlx5_add_flow_rule(ft, match_criteria_enable, match_criteria,
+ match_value, MLX5_FLOW_CONTEXT_ACTION_FWD_DEST,
+ MLX5_FS_DEFAULT_FLOW_TAG,
+ &dest);
+ if (IS_ERR(rule)) {
+ err = PTR_ERR(rule);
+ netdev_err(priv->netdev, "%s: add rule(filter id=%d, rq idx=%d) failed, err=%d\n",
+ __func__, arfs_rule->filter_id, arfs_rule->rxq, err);
+ }
+
+out:
+ kvfree(match_criteria);
+ kvfree(match_value);
+ return err ? ERR_PTR(err) : rule;
+}
+
+static void arfs_modify_rule_rq(struct mlx5e_priv *priv,
+ struct mlx5_flow_rule *rule, u16 rxq)
+{
+ struct mlx5_flow_destination dst;
+ int err = 0;
+
+ dst.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ dst.tir_num = priv->direct_tir[rxq].tirn;
+ err = mlx5_modify_rule_destination(rule, &dst);
+ if (err)
+ netdev_warn(priv->netdev,
+ "Failed to modfiy aRFS rule destination to rq=%d\n", rxq);
+}
+
+static void arfs_handle_work(struct work_struct *work)
+{
+ struct arfs_rule *arfs_rule = container_of(work,
+ struct arfs_rule,
+ arfs_work);
+ struct mlx5e_priv *priv = arfs_rule->priv;
+ struct mlx5_flow_rule *rule;
+
+ mutex_lock(&priv->state_lock);
+ if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) {
+ spin_lock_bh(&priv->fs.arfs.arfs_lock);
+ hlist_del(&arfs_rule->hlist);
+ spin_unlock_bh(&priv->fs.arfs.arfs_lock);
+
+ mutex_unlock(&priv->state_lock);
+ kfree(arfs_rule);
+ goto out;
+ }
+ mutex_unlock(&priv->state_lock);
+
+ if (!arfs_rule->rule) {
+ rule = arfs_add_rule(priv, arfs_rule);
+ if (IS_ERR(rule))
+ goto out;
+ arfs_rule->rule = rule;
+ } else {
+ arfs_modify_rule_rq(priv, arfs_rule->rule,
+ arfs_rule->rxq);
+ }
+out:
+ arfs_may_expire_flow(priv);
+}
+
+/* return L4 destination port from ip4/6 packets */
+static __be16 arfs_get_dst_port(const struct sk_buff *skb)
+{
+ char *transport_header;
+
+ transport_header = skb_transport_header(skb);
+ if (arfs_get_ip_proto(skb) == IPPROTO_TCP)
+ return ((struct tcphdr *)transport_header)->dest;
+ return ((struct udphdr *)transport_header)->dest;
+}
+
+/* return L4 source port from ip4/6 packets */
+static __be16 arfs_get_src_port(const struct sk_buff *skb)
+{
+ char *transport_header;
+
+ transport_header = skb_transport_header(skb);
+ if (arfs_get_ip_proto(skb) == IPPROTO_TCP)
+ return ((struct tcphdr *)transport_header)->source;
+ return ((struct udphdr *)transport_header)->source;
+}
+
+static struct arfs_rule *arfs_alloc_rule(struct mlx5e_priv *priv,
+ struct arfs_table *arfs_t,
+ const struct sk_buff *skb,
+ u16 rxq, u32 flow_id)
+{
+ struct arfs_rule *rule;
+ struct arfs_tuple *tuple;
+
+ rule = kzalloc(sizeof(*rule), GFP_ATOMIC);
+ if (!rule)
+ return NULL;
+
+ rule->priv = priv;
+ rule->rxq = rxq;
+ INIT_WORK(&rule->arfs_work, arfs_handle_work);
+
+ tuple = &rule->tuple;
+ tuple->etype = skb->protocol;
+ if (tuple->etype == htons(ETH_P_IP)) {
+ tuple->src_ipv4 = ip_hdr(skb)->saddr;
+ tuple->dst_ipv4 = ip_hdr(skb)->daddr;
+ } else {
+ memcpy(&tuple->src_ipv6, &ipv6_hdr(skb)->saddr,
+ sizeof(struct in6_addr));
+ memcpy(&tuple->dst_ipv6, &ipv6_hdr(skb)->daddr,
+ sizeof(struct in6_addr));
+ }
+ tuple->ip_proto = arfs_get_ip_proto(skb);
+ tuple->src_port = arfs_get_src_port(skb);
+ tuple->dst_port = arfs_get_dst_port(skb);
+
+ rule->flow_id = flow_id;
+ rule->filter_id = priv->fs.arfs.last_filter_id++ % RPS_NO_FILTER;
+
+ hlist_add_head(&rule->hlist,
+ arfs_hash_bucket(arfs_t, tuple->src_port,
+ tuple->dst_port));
+ return rule;
+}
+
+static bool arfs_cmp_ips(struct arfs_tuple *tuple,
+ const struct sk_buff *skb)
+{
+ if (tuple->etype == htons(ETH_P_IP) &&
+ tuple->src_ipv4 == ip_hdr(skb)->saddr &&
+ tuple->dst_ipv4 == ip_hdr(skb)->daddr)
+ return true;
+ if (tuple->etype == htons(ETH_P_IPV6) &&
+ (!memcmp(&tuple->src_ipv6, &ipv6_hdr(skb)->saddr,
+ sizeof(struct in6_addr))) &&
+ (!memcmp(&tuple->dst_ipv6, &ipv6_hdr(skb)->daddr,
+ sizeof(struct in6_addr))))
+ return true;
+ return false;
+}
+
+static struct arfs_rule *arfs_find_rule(struct arfs_table *arfs_t,
+ const struct sk_buff *skb)
+{
+ struct arfs_rule *arfs_rule;
+ struct hlist_head *head;
+ __be16 src_port = arfs_get_src_port(skb);
+ __be16 dst_port = arfs_get_dst_port(skb);
+
+ head = arfs_hash_bucket(arfs_t, src_port, dst_port);
+ hlist_for_each_entry(arfs_rule, head, hlist) {
+ if (arfs_rule->tuple.src_port == src_port &&
+ arfs_rule->tuple.dst_port == dst_port &&
+ arfs_cmp_ips(&arfs_rule->tuple, skb)) {
+ return arfs_rule;
+ }
+ }
+
+ return NULL;
+}
+
+int mlx5e_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
+ u16 rxq_index, u32 flow_id)
+{
+ struct mlx5e_priv *priv = netdev_priv(dev);
+ struct mlx5e_arfs_tables *arfs = &priv->fs.arfs;
+ struct arfs_table *arfs_t;
+ struct arfs_rule *arfs_rule;
+
+ if (skb->protocol != htons(ETH_P_IP) &&
+ skb->protocol != htons(ETH_P_IPV6))
+ return -EPROTONOSUPPORT;
+
+ arfs_t = arfs_get_table(arfs, arfs_get_ip_proto(skb), skb->protocol);
+ if (!arfs_t)
+ return -EPROTONOSUPPORT;
+
+ spin_lock_bh(&arfs->arfs_lock);
+ arfs_rule = arfs_find_rule(arfs_t, skb);
+ if (arfs_rule) {
+ if (arfs_rule->rxq == rxq_index) {
+ spin_unlock_bh(&arfs->arfs_lock);
+ return arfs_rule->filter_id;
+ }
+ arfs_rule->rxq = rxq_index;
+ } else {
+ arfs_rule = arfs_alloc_rule(priv, arfs_t, skb,
+ rxq_index, flow_id);
+ if (!arfs_rule) {
+ spin_unlock_bh(&arfs->arfs_lock);
+ return -ENOMEM;
+ }
+ }
+ queue_work(priv->fs.arfs.wq, &arfs_rule->arfs_work);
+ spin_unlock_bh(&arfs->arfs_lock);
+ return arfs_rule->filter_id;
+}
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH net-next 12/12] net/mlx5e: Enabling aRFS mechanism
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (10 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 11/12] net/mlx5e: Add accelerated RFS support Saeed Mahameed
@ 2016-04-28 22:36 ` Saeed Mahameed
2016-04-29 20:31 ` [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support David Miller
12 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-04-28 22:36 UTC (permalink / raw)
To: David S. Miller
Cc: netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha, Maor Gottlieb,
Saeed Mahameed
From: Maor Gottlieb <maorg@mellanox.com>
Accelerated RFS requires that ntuple filtering is enabled via
ethtool and driver supports ndo_rx_flow_steer.
When the ntuple filtering is enabled, we modify the l3_l4 ttc
rules to point on the aRFS flow tables and when the filtering
is disabled, we modify the l3_l4 ttc rules to point on the RSS
TIRs.
Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en.h | 12 +++
drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 77 +++++++++++++++++++-
.../net/ethernet/mellanox/mlx5/core/en_ethtool.c | 15 ++++
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 25 +++++++
4 files changed, 127 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 21c3841..34523c4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -690,9 +690,21 @@ static inline int mlx5e_arfs_create_tables(struct mlx5e_priv *priv)
}
static inline void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv) {}
+
+static inline int mlx5e_arfs_enable(struct mlx5e_priv *priv)
+{
+ return -ENOTSUPP;
+}
+
+static inline int mlx5e_arfs_disable(struct mlx5e_priv *priv)
+{
+ return -ENOTSUPP;
+}
#else
int mlx5e_arfs_create_tables(struct mlx5e_priv *priv);
void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv);
+int mlx5e_arfs_enable(struct mlx5e_priv *priv);
+int mlx5e_arfs_disable(struct mlx5e_priv *priv);
int mlx5e_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
u16 rxq_index, u32 flow_id);
#endif
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
index e54fbc1..b4ae0fe 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -72,14 +72,87 @@ struct arfs_rule {
for (j = 0; j < ARFS_HASH_SIZE; j++) \
hlist_for_each_entry_safe(hn, tmp, &hash[j], hlist)
+static enum mlx5e_traffic_types arfs_get_tt(enum arfs_type type)
+{
+ switch (type) {
+ case ARFS_IPV4_TCP:
+ return MLX5E_TT_IPV4_TCP;
+ case ARFS_IPV4_UDP:
+ return MLX5E_TT_IPV4_UDP;
+ case ARFS_IPV6_TCP:
+ return MLX5E_TT_IPV6_TCP;
+ case ARFS_IPV6_UDP:
+ return MLX5E_TT_IPV6_UDP;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int arfs_disable(struct mlx5e_priv *priv)
+{
+ struct mlx5_flow_destination dest;
+ u32 *tirn = priv->indir_tirn;
+ int err = 0;
+ int tt;
+ int i;
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_TIR;
+ for (i = 0; i < ARFS_NUM_TYPES; i++) {
+ dest.tir_num = tirn[i];
+ tt = arfs_get_tt(i);
+ /* Modify ttc rules destination to bypass the aRFS tables*/
+ err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
+ &dest);
+ if (err) {
+ netdev_err(priv->netdev,
+ "%s: modify ttc destination failed\n",
+ __func__);
+ return err;
+ }
+ }
+ return 0;
+}
+
+static void arfs_del_rules(struct mlx5e_priv *priv);
+
+int mlx5e_arfs_disable(struct mlx5e_priv *priv)
+{
+ arfs_del_rules(priv);
+
+ return arfs_disable(priv);
+}
+
+int mlx5e_arfs_enable(struct mlx5e_priv *priv)
+{
+ struct mlx5_flow_destination dest;
+ int err = 0;
+ int tt;
+ int i;
+
+ dest.type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
+ for (i = 0; i < ARFS_NUM_TYPES; i++) {
+ dest.ft = priv->fs.arfs.arfs_tables[i].ft.t;
+ tt = arfs_get_tt(i);
+ /* Modify ttc rules destination to point on the aRFS FTs */
+ err = mlx5_modify_rule_destination(priv->fs.ttc.rules[tt],
+ &dest);
+ if (err) {
+ netdev_err(priv->netdev,
+ "%s: modify ttc destination failed err=%d\n",
+ __func__, err);
+ arfs_disable(priv);
+ return err;
+ }
+ }
+ return 0;
+}
+
static void arfs_destroy_table(struct arfs_table *arfs_t)
{
mlx5_del_flow_rule(arfs_t->default_rule);
mlx5e_destroy_flow_table(&arfs_t->ft);
}
-static void arfs_del_rules(struct mlx5e_priv *priv);
-
void mlx5e_arfs_destroy_tables(struct mlx5e_priv *priv)
{
int i;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index 498d407..534d99e 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -456,6 +456,7 @@ static int mlx5e_set_channels(struct net_device *dev,
struct mlx5e_priv *priv = netdev_priv(dev);
int ncv = mlx5e_get_max_num_channels(priv->mdev);
unsigned int count = ch->combined_count;
+ bool arfs_enabled;
bool was_opened;
int err = 0;
@@ -484,13 +485,27 @@ static int mlx5e_set_channels(struct net_device *dev,
if (was_opened)
mlx5e_close_locked(dev);
+ arfs_enabled = dev->features & NETIF_F_NTUPLE;
+ if (arfs_enabled)
+ mlx5e_arfs_disable(priv);
+
priv->params.num_channels = count;
mlx5e_build_default_indir_rqt(priv->mdev, priv->params.indirection_rqt,
MLX5E_INDIR_RQT_SIZE, count);
if (was_opened)
err = mlx5e_open_locked(dev);
+ if (err)
+ goto out;
+ if (arfs_enabled) {
+ err = mlx5e_arfs_enable(priv);
+ if (err)
+ netdev_err(dev, "%s: mlx5e_arfs_enable failed: %d\n",
+ __func__, err);
+ }
+
+out:
mutex_unlock(&priv->state_lock);
return err;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 5d64534..b40949a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2306,6 +2306,21 @@ static int set_feature_rx_vlan(struct net_device *netdev, bool enable)
return err;
}
+#ifdef CONFIG_RFS_ACCEL
+static int set_feature_arfs(struct net_device *netdev, bool enable)
+{
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ int err;
+
+ if (enable)
+ err = mlx5e_arfs_enable(priv);
+ else
+ err = mlx5e_arfs_disable(priv);
+
+ return err;
+}
+#endif
+
static int mlx5e_handle_feature(struct net_device *netdev,
netdev_features_t wanted_features,
netdev_features_t feature,
@@ -2345,6 +2360,10 @@ static int mlx5e_set_features(struct net_device *netdev,
set_feature_rx_all);
err |= mlx5e_handle_feature(netdev, features, NETIF_F_HW_VLAN_CTAG_RX,
set_feature_rx_vlan);
+#ifdef CONFIG_RFS_ACCEL
+ err |= mlx5e_handle_feature(netdev, features, NETIF_F_NTUPLE,
+ set_feature_arfs);
+#endif
return err ? -EINVAL : 0;
}
@@ -2560,6 +2579,9 @@ static const struct net_device_ops mlx5e_netdev_ops_basic = {
.ndo_set_features = mlx5e_set_features,
.ndo_change_mtu = mlx5e_change_mtu,
.ndo_do_ioctl = mlx5e_ioctl,
+#ifdef CONFIG_RFS_ACCEL
+ .ndo_rx_flow_steer = mlx5e_rx_flow_steer,
+#endif
};
static const struct net_device_ops mlx5e_netdev_ops_sriov = {
@@ -2579,6 +2601,9 @@ static const struct net_device_ops mlx5e_netdev_ops_sriov = {
.ndo_add_vxlan_port = mlx5e_add_vxlan_port,
.ndo_del_vxlan_port = mlx5e_del_vxlan_port,
.ndo_features_check = mlx5e_features_check,
+#ifdef CONFIG_RFS_ACCEL
+ .ndo_rx_flow_steer = mlx5e_rx_flow_steer,
+#endif
.ndo_set_vf_mac = mlx5e_set_vf_mac,
.ndo_set_vf_vlan = mlx5e_set_vf_vlan,
.ndo_get_vf_config = mlx5e_get_vf_config,
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
` (11 preceding siblings ...)
2016-04-28 22:36 ` [PATCH net-next 12/12] net/mlx5e: Enabling aRFS mechanism Saeed Mahameed
@ 2016-04-29 20:31 ` David Miller
12 siblings, 0 replies; 16+ messages in thread
From: David Miller @ 2016-04-29 20:31 UTC (permalink / raw)
To: saeedm; +Cc: netdev, ogerlitz, talal, eranbe
From: Saeed Mahameed <saeedm@mellanox.com>
Date: Fri, 29 Apr 2016 01:36:30 +0300
> This series adds accelerated RFS support for the mlx5e driver.
Series applied, thanks Saeed.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables
2016-04-28 22:36 ` [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables Saeed Mahameed
@ 2016-05-01 2:18 ` Alexei Starovoitov
2016-05-01 8:12 ` Saeed Mahameed
0 siblings, 1 reply; 16+ messages in thread
From: Alexei Starovoitov @ 2016-05-01 2:18 UTC (permalink / raw)
To: Saeed Mahameed
Cc: David S. Miller, netdev, Or Gerlitz, Tal Alon, Eran Ben Elisha,
Maor Gottlieb
On Fri, Apr 29, 2016 at 01:36:40AM +0300, Saeed Mahameed wrote:
> From: Maor Gottlieb <maorg@mellanox.com>
>
> Create the following four flow tables for aRFS usage:
> 1. IPv4 TCP - filtering 4-tuple of IPv4 TCP packets.
> 2. IPv6 TCP - filtering 4-tuple of IPv6 TCP packets.
> 3. IPv4 UDP - filtering 4-tuple of IPv4 UDP packets.
> 4. IPv6 UDP - filtering 4-tuple of IPv6 UDP packets.
>
> Each flow table has two flow groups: one for the 4-tuple
> filtering (full match) and the other contains * rule for miss rule.
>
> Full match rule means a hit for aRFS and packet will be forwarded
> to the dedicated RQ/Core, miss rule packets will be forwarded to
> default RSS hashing.
>
> Signed-off-by: Maor Gottlieb <maorg@mellanox.com>
> Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>
> ---
> drivers/net/ethernet/mellanox/mlx5/core/Makefile | 1 +
> drivers/net/ethernet/mellanox/mlx5/core/en.h | 41 ++++
> drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c | 251 +++++++++++++++++++++
> drivers/net/ethernet/mellanox/mlx5/core/en_fs.c | 23 +-
> drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 +-
> drivers/net/ethernet/mellanox/mlx5/core/fs_core.c | 3 +-
> 6 files changed, 313 insertions(+), 14 deletions(-)
> create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
> index 4fc45ee..679e18f 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile
> @@ -9,3 +9,4 @@ mlx5_core-$(CONFIG_MLX5_CORE_EN) += wq.o eswitch.o \
> en_txrx.o en_clock.o vxlan.o en_tc.o
>
> mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o
> +mlx5_core-$(CONFIG_RFS_ACCEL) += en_arfs.o
this line breaks the build when RFS is on
and CONFIG_MLX5_CORE_EN is off.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables
2016-05-01 2:18 ` Alexei Starovoitov
@ 2016-05-01 8:12 ` Saeed Mahameed
0 siblings, 0 replies; 16+ messages in thread
From: Saeed Mahameed @ 2016-05-01 8:12 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Saeed Mahameed, David S. Miller, Linux Netdev List, Or Gerlitz,
Tal Alon, Eran Ben Elisha, Maor Gottlieb
On Sun, May 1, 2016 at 5:18 AM, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
> On Fri, Apr 29, 2016 at 01:36:40AM +0300, Saeed Mahameed wrote:
>> From: Maor Gottlieb <maorg@mellanox.com>
>> mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o
>> +mlx5_core-$(CONFIG_RFS_ACCEL) += en_arfs.o
>
> this line breaks the build when RFS is on
> and CONFIG_MLX5_CORE_EN is off.
>
Will Fix.
Thank you Alexei.
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2016-05-01 8:12 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-28 22:36 [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 01/12] net/mlx5e: Call vxlan_get_rx_port() with rtnl lock Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 02/12] net/mlx5e: Direct TIR per RQ Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 03/12] net/mlx5: Introduce modify flow rule destination Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 04/12] net/mlx5: Set number of allowed levels in priority Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 05/12] net/mlx5: Add user chosen levels when allocating flow tables Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 06/12] net/mlx5: Support different attributes for priorities in namespace Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 07/12] net/mlx5e: Refactor mlx5e flow steering structs Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 08/12] net/mlx5e: Split the main flow steering table Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 09/12] net/mlx5: Initializing CPU reverse mapping Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 10/12] net/mlx5e: Create aRFS flow tables Saeed Mahameed
2016-05-01 2:18 ` Alexei Starovoitov
2016-05-01 8:12 ` Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 11/12] net/mlx5e: Add accelerated RFS support Saeed Mahameed
2016-04-28 22:36 ` [PATCH net-next 12/12] net/mlx5e: Enabling aRFS mechanism Saeed Mahameed
2016-04-29 20:31 ` [PATCH net-next 00/12] Mellanox 100G mlx5 ethernet aRFS support David Miller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).