[PATCH net-next 0/2] mlx4/en_netdev: allow offloading VXLAN over VLAN

[PATCH net-next 0/2] mlx4/en_netdev: allow offloading VXLAN over VLAN

[Davide Caratti]

When VXLAN offload is enabled on ConnectX-3 Pro devices, the NIC can
segment correctly also VXLAN packet with a VLAN tag: this series ensures
that a VLAN created on top of a mlx4_en NIC inherits the tunnel offload
capabilities.

Davide Caratti (2):
  mlx4/en_netdev: update vlan features with tunnel offloads
  mlx4/en_netdev: call notifiers when hw_enc_features change

 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 60 ++++++++++++-------
 1 file changed, 37 insertions(+), 23 deletions(-)

-- 
2.20.1

[PATCH net-next 1/2] mlx4/en_netdev: update vlan features with tunnel offloads

[Davide Caratti]

ConnectX-3 Pro can offload transmission of VLAN packets with VXLAN inside:
enable tunnel offloads in dev->vlan_features, like it's done with other
NIC drivers (e.g. be2net and ixgbe).

Signed-off-by: Davide Caratti <dcaratti@redhat.com>
---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 21 ++++++++++---------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index c1438ae52a11..52500f744a0e 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -3415,6 +3415,17 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 	if (mdev->LSO_support)
 		dev->hw_features |= NETIF_F_TSO | NETIF_F_TSO6;
 
+	if (mdev->dev->caps.tunnel_offload_mode ==
+	    MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
+		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
+	}
+
 	dev->vlan_features = dev->hw_features;
 
 	dev->hw_features |= NETIF_F_RXCSUM | NETIF_F_RXHASH;
@@ -3483,16 +3494,6 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 		priv->rss_hash_fn = ETH_RSS_HASH_TOP;
 	}
 
-	if (mdev->dev->caps.tunnel_offload_mode == MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
-		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				    NETIF_F_GSO_PARTIAL;
-		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				    NETIF_F_GSO_PARTIAL;
-		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
-	}
-
 	/* MTU range: 68 - hw-specific max */
 	dev->min_mtu = ETH_MIN_MTU;
 	dev->max_mtu = priv->max_mtu;
-- 
2.20.1

[PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Davide Caratti]

ensure to call netdev_features_change() when the driver flips its
hw_enc_features bits.

Signed-off-by: Davide Caratti <dcaratti@redhat.com>
---
 .../net/ethernet/mellanox/mlx4/en_netdev.c    | 39 ++++++++++++-------
 1 file changed, 26 insertions(+), 13 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index 52500f744a0e..1b484dc6e1c2 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -2628,6 +2628,30 @@ static int mlx4_en_get_phys_port_id(struct net_device *dev,
 	return 0;
 }
 
+#define MLX4_GSO_PARTIAL_FEATURES (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
+				   NETIF_F_RXCSUM | \
+				   NETIF_F_TSO | NETIF_F_TSO6 | \
+				   NETIF_F_GSO_UDP_TUNNEL | \
+				   NETIF_F_GSO_UDP_TUNNEL_CSUM | \
+				   NETIF_F_GSO_PARTIAL)
+
+static void mlx4_set_vxlan_offloads(struct net_device *dev, bool enable)
+{
+	netdev_features_t hw_enc_features;
+
+	rtnl_lock();
+	hw_enc_features = dev->hw_enc_features;
+	if (enable)
+		dev->hw_enc_features |= MLX4_GSO_PARTIAL_FEATURES;
+	else
+		dev->hw_enc_features &= ~MLX4_GSO_PARTIAL_FEATURES;
+
+	if (hw_enc_features ^ dev->hw_enc_features)
+		netdev_features_change(dev);
+
+	rtnl_unlock();
+}
+
 static void mlx4_en_add_vxlan_offloads(struct work_struct *work)
 {
 	int ret;
@@ -2647,12 +2671,7 @@ static void mlx4_en_add_vxlan_offloads(struct work_struct *work)
 	}
 
 	/* set offloads */
-	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
-				      NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_TSO6 |
-				      NETIF_F_GSO_UDP_TUNNEL |
-				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
-				      NETIF_F_GSO_PARTIAL;
+	mlx4_set_vxlan_offloads(priv->dev, true);
 }
 
 static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
@@ -2661,13 +2680,7 @@ static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
 	struct mlx4_en_priv *priv = container_of(work, struct mlx4_en_priv,
 						 vxlan_del_task);
 	/* unset offloads */
-	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
-					NETIF_F_RXCSUM |
-					NETIF_F_TSO | NETIF_F_TSO6 |
-					NETIF_F_GSO_UDP_TUNNEL |
-					NETIF_F_GSO_UDP_TUNNEL_CSUM |
-					NETIF_F_GSO_PARTIAL);
-
+	mlx4_set_vxlan_offloads(priv->dev, false);
 	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
 				  VXLAN_STEER_BY_OUTER_MAC, 0);
 	if (ret)
-- 
2.20.1

Re: [PATCH net-next 1/2] mlx4/en_netdev: update vlan features with tunnel offloads

[Saeed Mahameed]

On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> ConnectX-3 Pro can offload transmission of VLAN packets with VXLAN
> inside:
> enable tunnel offloads in dev->vlan_features, like it's done with
> other
> NIC drivers (e.g. be2net and ixgbe).
> 
> Signed-off-by: Davide Caratti <dcaratti@redhat.com>


LGTM
Reviewed-by: Saeed Mahameed <saeedm@mellanox.com>

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Saeed Mahameed]

On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> ensure to call netdev_features_change() when the driver flips its
> hw_enc_features bits.
> 
> Signed-off-by: Davide Caratti <dcaratti@redhat.com>

The patch is correct, but can you explain how did you come to this ? 
did you encounter any issue with the current code ?

I am asking just because i think the whole dynamic changing of dev-
>hw_enc_features is redundant since mlx4 has the featutres_check
callback.

> ---
>  .../net/ethernet/mellanox/mlx4/en_netdev.c    | 39 ++++++++++++-----
> --
>  1 file changed, 26 insertions(+), 13 deletions(-)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> index 52500f744a0e..1b484dc6e1c2 100644
> --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
> @@ -2628,6 +2628,30 @@ static int mlx4_en_get_phys_port_id(struct
> net_device *dev,
>  	return 0;
>  }
>  
> +#define MLX4_GSO_PARTIAL_FEATURES (NETIF_F_IP_CSUM |
> NETIF_F_IPV6_CSUM | \
> +				   NETIF_F_RXCSUM | \
> +				   NETIF_F_TSO | NETIF_F_TSO6 | \
> +				   NETIF_F_GSO_UDP_TUNNEL | \
> +				   NETIF_F_GSO_UDP_TUNNEL_CSUM | \
> +				   NETIF_F_GSO_PARTIAL)
> +
> +static void mlx4_set_vxlan_offloads(struct net_device *dev, bool
> enable)
> +{
> +	netdev_features_t hw_enc_features;
> +
> +	rtnl_lock();
> +	hw_enc_features = dev->hw_enc_features;
> +	if (enable)
> +		dev->hw_enc_features |= MLX4_GSO_PARTIAL_FEATURES;
> +	else
> +		dev->hw_enc_features &= ~MLX4_GSO_PARTIAL_FEATURES;
> +
> +	if (hw_enc_features ^ dev->hw_enc_features)
> +		netdev_features_change(dev);
> +
> +	rtnl_unlock();
> +}
> +
>  static void mlx4_en_add_vxlan_offloads(struct work_struct *work)
>  {
>  	int ret;
> @@ -2647,12 +2671,7 @@ static void mlx4_en_add_vxlan_offloads(struct
> work_struct *work)
>  	}
>  
>  	/* set offloads */
> -	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM |
> NETIF_F_IPV6_CSUM |
> -				      NETIF_F_RXCSUM |
> -				      NETIF_F_TSO | NETIF_F_TSO6 |
> -				      NETIF_F_GSO_UDP_TUNNEL |
> -				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
> -				      NETIF_F_GSO_PARTIAL;
> +	mlx4_set_vxlan_offloads(priv->dev, true);
>  }
>  
>  static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
> @@ -2661,13 +2680,7 @@ static void mlx4_en_del_vxlan_offloads(struct
> work_struct *work)
>  	struct mlx4_en_priv *priv = container_of(work, struct
> mlx4_en_priv,
>  						 vxlan_del_task);
>  	/* unset offloads */
> -	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM |
> NETIF_F_IPV6_CSUM |
> -					NETIF_F_RXCSUM |
> -					NETIF_F_TSO | NETIF_F_TSO6 |
> -					NETIF_F_GSO_UDP_TUNNEL |
> -					NETIF_F_GSO_UDP_TUNNEL_CSUM |
> -					NETIF_F_GSO_PARTIAL);
> -
> +	mlx4_set_vxlan_offloads(priv->dev, false);
>  	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
>  				  VXLAN_STEER_BY_OUTER_MAC, 0);
>  	if (ret)

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Davide Caratti]

On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> > ensure to call netdev_features_change() when the driver flips its
> > hw_enc_features bits.
> > 
> > Signed-off-by: Davide Caratti <dcaratti@redhat.com>
> 
> The patch is correct, 

hello Saeed, and thanks for looking at this!

> but can you explain how did you come to this ? 
> did you encounter any issue with the current code ?
> 
> I am asking just because i think the whole dynamic changing of dev-
> > hw_enc_features is redundant since mlx4 has the featutres_check
> callback.

we need it to ensure that vlan_transfer_features() updates
the (new) value of hw_enc_features in the overlying vlan: otherwise,
segmentation will happen anyway when skb passes from vxlan to vlan, if the
vxlan is added after the vlan device has been created (see: 7dad9937e064
("net: vlan: add support for tunnel offload") ).

thanks!
-- 
davide

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Saeed Mahameed]

On Thu, 2019-07-25 at 14:25 +0200, Davide Caratti wrote:
> On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> > On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> > > ensure to call netdev_features_change() when the driver flips its
> > > hw_enc_features bits.
> > > 
> > > Signed-off-by: Davide Caratti <dcaratti@redhat.com>
> > 
> > The patch is correct, 
> 
> hello Saeed, and thanks for looking at this!
> 
> > but can you explain how did you come to this ? 
> > did you encounter any issue with the current code ?
> > 
> > I am asking just because i think the whole dynamic changing of dev-
> > > hw_enc_features is redundant since mlx4 has the featutres_check
> > callback.
> 
> we need it to ensure that vlan_transfer_features() updates
> the (new) value of hw_enc_features in the overlying vlan: otherwise,
> segmentation will happen anyway when skb passes from vxlan to vlan,
> if the
> vxlan is added after the vlan device has been created (see:
> 7dad9937e064
> ("net: vlan: add support for tunnel offload") ).
> 

but in previous patch you made sure that the vlan always sees the
correct hw_enc_features on driver load, we don't need to have this
dynamic update mechanism, features_check ndo should take care of
protocols we don't support.

> thanks!

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Davide Caratti]

On Thu, 2019-07-25 at 21:27 +0000, Saeed Mahameed wrote:
> On Thu, 2019-07-25 at 14:25 +0200, Davide Caratti wrote:
> > On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> > > On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> > > > ensure to call netdev_features_change() when the driver flips its
> > > > hw_enc_features bits.
> > > > 
> > > > Signed-off-by: Davide Caratti <dcaratti@redhat.com>
> > > 
> > > The patch is correct, 
> > 
> > hello Saeed, and thanks for looking at this!
> > 
> > > but can you explain how did you come to this ? 
> > > did you encounter any issue with the current code ?
> > > 
> > > I am asking just because i think the whole dynamic changing of dev-
> > > > hw_enc_features is redundant since mlx4 has the featutres_check
> > > callback.
> > 
> > we need it to ensure that vlan_transfer_features() updates
> > the (new) value of hw_enc_features in the overlying vlan: otherwise,
> > segmentation will happen anyway when skb passes from vxlan to vlan,
> > if the
> > vxlan is added after the vlan device has been created (see:
> > 7dad9937e064
> > ("net: vlan: add support for tunnel offload") ).
> > 
> 
> but in previous patch you made sure that the vlan always sees the
> correct hw_enc_features on driver load, we don't need to have this
> dynamic update mechanism,

ok, but the mlx4 driver flips the value of hw_enc_features when VXLAN
tunnels are added or removed. So, assume eth0 is a Cx3-pro, and I do:
 
 # ip link add name vlan5 link eth0 type vlan id 5
 # ip link add dev vxlan6 type vxlan id 6  [...]  dev vlan5
 
the value of dev->hw_enc_features is 0 for vlan5, and as a consequence
VXLAN over VLAN traffic becomes segmented by the VLAN, even if eth0, at
the end of this sequence, has the "right" features bits.

> features_check ndo should take care of
> protocols we don't support.

I just had a look at mlx4_en_features_check(), I see it checks if the
packet is tunneled in VXLAN and the destination port matches the
configured value of priv->vxlan_port (when that value is not zero). Now:

On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> I am asking just because i think the whole dynamic changing of 
> dev-> hw_enc_features is redundant since mlx4 has the featutres_check
> callback.

I read your initial proposal again. Would it be correct if I just use
patch 1/2, where I add an assignment of

dev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
                       NETIF_F_RXCSUM | \
                       NETIF_F_TSO | NETIF_F_TSO6 | \
                       NETIF_F_GSO_UDP_TUNNEL | \
                       NETIF_F_GSO_UDP_TUNNEL_CSUM | \
                       NETIF_F_GSO_PARTIAL;

in mlx4_en_init_netdev(), and then remove the code that flips
dev->hw_enc_features in mlx4_en_add_vxlan_offloads() and
mlx4_en_del_vxlan_offloads() ?

thanks,
--
davide

Re: [PATCH net-next 2/2] mlx4/en_netdev: call notifiers when hw_enc_features change

[Saeed Mahameed]

On Fri, 2019-07-26 at 12:39 +0200, Davide Caratti wrote:
> On Thu, 2019-07-25 at 21:27 +0000, Saeed Mahameed wrote:
> > On Thu, 2019-07-25 at 14:25 +0200, Davide Caratti wrote:
> > > On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> > > > On Wed, 2019-07-24 at 16:02 +0200, Davide Caratti wrote:
> > > > > ensure to call netdev_features_change() when the driver flips
> > > > > its
> > > > > hw_enc_features bits.
> > > > > 
> > > > > Signed-off-by: Davide Caratti <dcaratti@redhat.com>
> > > > 
> > > > The patch is correct, 
> > > 
> > > hello Saeed, and thanks for looking at this!
> > > 
> > > > but can you explain how did you come to this ? 
> > > > did you encounter any issue with the current code ?
> > > > 
> > > > I am asking just because i think the whole dynamic changing of
> > > > dev-
> > > > > hw_enc_features is redundant since mlx4 has the
> > > > > featutres_check
> > > > callback.
> > > 
> > > we need it to ensure that vlan_transfer_features() updates
> > > the (new) value of hw_enc_features in the overlying vlan:
> > > otherwise,
> > > segmentation will happen anyway when skb passes from vxlan to
> > > vlan,
> > > if the
> > > vxlan is added after the vlan device has been created (see:
> > > 7dad9937e064
> > > ("net: vlan: add support for tunnel offload") ).
> > > 
> > 
> > but in previous patch you made sure that the vlan always sees the
> > correct hw_enc_features on driver load, we don't need to have this
> > dynamic update mechanism,
> 
> ok, but the mlx4 driver flips the value of hw_enc_features when VXLAN
> tunnels are added or removed. So, assume eth0 is a Cx3-pro, and I do:
>  
>  # ip link add name vlan5 link eth0 type vlan id 5
>  # ip link add dev vxlan6 type vxlan id 6  [...]  dev vlan5
>  
> the value of dev->hw_enc_features is 0 for vlan5, and as a
> consequence
> VXLAN over VLAN traffic becomes segmented by the VLAN, even if eth0,
> at
> the end of this sequence, has the "right" features bits.
> 

your patch handled this issue already, no need for flipping and
updating features if features check ndo will cover the cases we don't
support.

> > features_check ndo should take care of
> > protocols we don't support.
> 
> I just had a look at mlx4_en_features_check(), I see it checks if the
> packet is tunneled in VXLAN and the destination port matches the
> configured value of priv->vxlan_port (when that value is not zero).
> Now:
> 
> On Wed, 2019-07-24 at 20:47 +0000, Saeed Mahameed wrote:
> > I am asking just because i think the whole dynamic changing of 
> > dev-> hw_enc_features is redundant since mlx4 has the
> > featutres_check
> > callback.
> 
> I read your initial proposal again. Would it be correct if I just use
> patch 1/2, where I add an assignment of
> 
> dev->hw_enc_features = NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | \
>                        NETIF_F_RXCSUM | \
>                        NETIF_F_TSO | NETIF_F_TSO6 | \
>                        NETIF_F_GSO_UDP_TUNNEL | \
>                        NETIF_F_GSO_UDP_TUNNEL_CSUM | \
>                        NETIF_F_GSO_PARTIAL;
> 
> in mlx4_en_init_netdev(), and then remove the code that flips
> dev->hw_enc_features in mlx4_en_add_vxlan_offloads() and
> mlx4_en_del_vxlan_offloads() ?
> 

yes, this is exactly what I meant.

Thanks,
Saeed.