[PATCH v2 net-next 0/4] gro: Fixes for tunnels and GRO

[PATCH v2 net-next 0/4] gro: Fixes for tunnels and GRO

[Tom Herbert]

This patch set addresses some issue related to tunneling and GRO:

- Ensure headers are pull into skb->head when putting 1st packet onto
  GRO list
- Fix remote checksum offload to properly deal with frag0 in GRO.
- Add support for GRO at VXLAN tunnel (call gro_cells)

Testing: Ran one netperf TCP_STREAM to highlight impact of different
configurations:

GUE
  Zero UDP checksum
    4628.42 MBps
  UDP checksums enabled
    6800.51 MBps
  UDP checksums and remote checksum offload
    7663.82 MBps
  UDP checksums and remote checksum offload using no-partial
    7287.25 MBps

VXLAN
  Zero UDP checksum
    4112.02
  UDP checksums enabled
    6785.80 MBps
  UDP checksums and remote checksum offload
    7075.56 MBps
    
v2:
  - Drop "gro: Pull headers into skb head for 1st skb in gro list"
    from patch set
  - In vxlan_remcsum and gue_remcsum return immediately if remcsum
    processing was already done
  - Add gro callbacks for sit offload
  - Use WARN_ON_ONCE if we get a GUE protocol that does not have
    GRO offload support


Tom Herbert (4):
  gro: Fix remcsum offload to deal with frags in GRO
  vxlan: GRO support at tunnel layer
  ipv6: Add gro functions to sit_offloads
  fou: Do WARN_ON_ONCE in gue_gro_receive for bad proto callbacks

 drivers/net/vxlan.c       | 32 ++++++++++++++++----------------
 include/linux/netdevice.h | 44 ++++++++++++++++++++++++++++++++------------
 include/net/vxlan.h       |  1 +
 net/ipv4/fou.c            | 30 +++++++++++++-----------------
 net/ipv6/ip6_offload.c    |  2 ++
 5 files changed, 64 insertions(+), 45 deletions(-)

-- 
1.8.5.6

[PATCH v2 net-next 1/4] gro: Fix remcsum offload to deal with frags in GRO

[Tom Herbert]

The remote checksum offload GRO did not consider the case that frag0
might be in use. This patch fixes that by accessing headers using the
skb_gro functions and not saving offsets relative to skb->head.

Signed-off-by: Tom Herbert <tom@herbertland.com>
---
 drivers/net/vxlan.c       | 23 +++++++++--------------
 include/linux/netdevice.h | 44 ++++++++++++++++++++++++++++++++------------
 net/ipv4/fou.c            | 28 ++++++++++++----------------
 3 files changed, 53 insertions(+), 42 deletions(-)

diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index e90f7a4..60b5b42 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c
@@ -519,10 +519,10 @@ static struct vxlanhdr *vxlan_gro_remcsum(struct sk_buff *skb,
 					  u32 data, struct gro_remcsum *grc,
 					  bool nopartial)
 {
-	size_t start, offset, plen;
+	size_t start, offset;
 
 	if (skb->remcsum_offload)
-		return NULL;
+		return vh;
 
 	if (!NAPI_GRO_CB(skb)->csum_valid)
 		return NULL;
@@ -532,17 +532,8 @@ static struct vxlanhdr *vxlan_gro_remcsum(struct sk_buff *skb,
 			  offsetof(struct udphdr, check) :
 			  offsetof(struct tcphdr, check));
 
-	plen = hdrlen + offset + sizeof(u16);
-
-	/* Pull checksum that will be written */
-	if (skb_gro_header_hard(skb, off + plen)) {
-		vh = skb_gro_header_slow(skb, off + plen, off);
-		if (!vh)
-			return NULL;
-	}
-
-	skb_gro_remcsum_process(skb, (void *)vh + hdrlen,
-				start, offset, grc, nopartial);
+	vh = skb_gro_remcsum_process(skb, (void *)vh, off, hdrlen,
+				     start, offset, grc, nopartial);
 
 	skb->remcsum_offload = 1;
 
@@ -573,7 +564,6 @@ static struct sk_buff **vxlan_gro_receive(struct sk_buff **head,
 			goto out;
 	}
 
-	skb_gro_pull(skb, sizeof(struct vxlanhdr)); /* pull vxlan header */
 	skb_gro_postpull_rcsum(skb, vh, sizeof(struct vxlanhdr));
 
 	flags = ntohl(vh->vx_flags);
@@ -588,6 +578,8 @@ static struct sk_buff **vxlan_gro_receive(struct sk_buff **head,
 			goto out;
 	}
 
+	skb_gro_pull(skb, sizeof(struct vxlanhdr)); /* pull vxlan header */
+
 	flush = 0;
 
 	for (p = *head; p; p = p->next) {
@@ -1110,6 +1102,9 @@ static struct vxlanhdr *vxlan_remcsum(struct sk_buff *skb, struct vxlanhdr *vh,
 {
 	size_t start, offset, plen;
 
+	if (skb->remcsum_offload)
+		return vh;
+
 	start = (data & VXLAN_RCO_MASK) << VXLAN_RCO_SHIFT;
 	offset = start + ((data & VXLAN_RCO_UDP) ?
 			  offsetof(struct udphdr, check) :
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 607b5f4..568d7ae 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2303,8 +2303,7 @@ __sum16 __skb_gro_checksum_complete(struct sk_buff *skb);
 
 static inline bool skb_at_gro_remcsum_start(struct sk_buff *skb)
 {
-	return (NAPI_GRO_CB(skb)->gro_remcsum_start - skb_headroom(skb) ==
-		skb_gro_offset(skb));
+	return (NAPI_GRO_CB(skb)->gro_remcsum_start == skb_gro_offset(skb));
 }
 
 static inline bool __skb_gro_checksum_validate_needed(struct sk_buff *skb,
@@ -2400,37 +2399,58 @@ static inline void skb_gro_remcsum_init(struct gro_remcsum *grc)
 	grc->delta = 0;
 }
 
-static inline void skb_gro_remcsum_process(struct sk_buff *skb, void *ptr,
-					   int start, int offset,
-					   struct gro_remcsum *grc,
-					   bool nopartial)
+static inline void *skb_gro_remcsum_process(struct sk_buff *skb, void *ptr,
+					    unsigned int off, size_t hdrlen,
+					    int start, int offset,
+					    struct gro_remcsum *grc,
+					    bool nopartial)
 {
 	__wsum delta;
+	size_t plen = hdrlen + max_t(size_t, offset + sizeof(u16), start);
 
 	BUG_ON(!NAPI_GRO_CB(skb)->csum_valid);
 
 	if (!nopartial) {
-		NAPI_GRO_CB(skb)->gro_remcsum_start =
-		    ((unsigned char *)ptr + start) - skb->head;
-		return;
+		NAPI_GRO_CB(skb)->gro_remcsum_start = off + hdrlen + start;
+		return ptr;
+	}
+
+	ptr = skb_gro_header_fast(skb, off);
+	if (skb_gro_header_hard(skb, off + plen)) {
+		ptr = skb_gro_header_slow(skb, off + plen, off);
+		if (!ptr)
+			return NULL;
 	}
 
-	delta = remcsum_adjust(ptr, NAPI_GRO_CB(skb)->csum, start, offset);
+	delta = remcsum_adjust(ptr + hdrlen, NAPI_GRO_CB(skb)->csum,
+			       start, offset);
 
 	/* Adjust skb->csum since we changed the packet */
 	NAPI_GRO_CB(skb)->csum = csum_add(NAPI_GRO_CB(skb)->csum, delta);
 
-	grc->offset = (ptr + offset) - (void *)skb->head;
+	grc->offset = off + hdrlen + offset;
 	grc->delta = delta;
+
+	return ptr;
 }
 
 static inline void skb_gro_remcsum_cleanup(struct sk_buff *skb,
 					   struct gro_remcsum *grc)
 {
+	void *ptr;
+	size_t plen = grc->offset + sizeof(u16);
+
 	if (!grc->delta)
 		return;
 
-	remcsum_unadjust((__sum16 *)(skb->head + grc->offset), grc->delta);
+	ptr = skb_gro_header_fast(skb, grc->offset);
+	if (skb_gro_header_hard(skb, grc->offset + sizeof(u16))) {
+		ptr = skb_gro_header_slow(skb, plen, grc->offset);
+		if (!ptr)
+			return;
+	}
+
+	remcsum_unadjust((__sum16 *)ptr, grc->delta);
 }
 
 static inline int dev_hard_header(struct sk_buff *skb, struct net_device *dev,
diff --git a/net/ipv4/fou.c b/net/ipv4/fou.c
index 34968cd..eb11f95 100644
--- a/net/ipv4/fou.c
+++ b/net/ipv4/fou.c
@@ -79,7 +79,11 @@ static struct guehdr *gue_remcsum(struct sk_buff *skb, struct guehdr *guehdr,
 	__be16 *pd = data;
 	size_t start = ntohs(pd[0]);
 	size_t offset = ntohs(pd[1]);
-	size_t plen = hdrlen + max_t(size_t, offset + sizeof(u16), start);
+	size_t plen = sizeof(struct udphdr) + hdrlen +
+	    max_t(size_t, offset + sizeof(u16), start);
+
+	if (skb->remcsum_offload)
+		return guehdr;
 
 	if (!pskb_may_pull(skb, plen))
 		return NULL;
@@ -221,29 +225,21 @@ out_unlock:
 
 static struct guehdr *gue_gro_remcsum(struct sk_buff *skb, unsigned int off,
 				      struct guehdr *guehdr, void *data,
-				      size_t hdrlen, u8 ipproto,
-				      struct gro_remcsum *grc, bool nopartial)
+				      size_t hdrlen, struct gro_remcsum *grc,
+				      bool nopartial)
 {
 	__be16 *pd = data;
 	size_t start = ntohs(pd[0]);
 	size_t offset = ntohs(pd[1]);
-	size_t plen = hdrlen + max_t(size_t, offset + sizeof(u16), start);
 
 	if (skb->remcsum_offload)
-		return NULL;
+		return guehdr;
 
 	if (!NAPI_GRO_CB(skb)->csum_valid)
 		return NULL;
 
-	/* Pull checksum that will be written */
-	if (skb_gro_header_hard(skb, off + plen)) {
-		guehdr = skb_gro_header_slow(skb, off + plen, off);
-		if (!guehdr)
-			return NULL;
-	}
-
-	skb_gro_remcsum_process(skb, (void *)guehdr + hdrlen,
-				start, offset, grc, nopartial);
+	guehdr = skb_gro_remcsum_process(skb, (void *)guehdr, off, hdrlen,
+					 start, offset, grc, nopartial);
 
 	skb->remcsum_offload = 1;
 
@@ -307,10 +303,10 @@ static struct sk_buff **gue_gro_receive(struct sk_buff **head,
 
 		if (flags & GUE_PFLAG_REMCSUM) {
 			guehdr = gue_gro_remcsum(skb, off, guehdr,
-						 data + doffset, hdrlen,
-						 guehdr->proto_ctype, &grc,
+						 data + doffset, hdrlen, &grc,
 						 !!(fou->flags &
 						    FOU_F_REMCSUM_NOPARTIAL));
+
 			if (!guehdr)
 				goto out;
 
-- 
1.8.5.6

[PATCH v2 net-next 2/4] vxlan: GRO support at tunnel layer

[Tom Herbert]

Add calls to gro_cells infrastructure to do GRO when receiving on a tunnel.

Testing:

Ran 200 netperf TCP_STREAM instance

  - With fix (GRO enabled on VXLAN interface)

    Verify GRO is happening.

    9084 MBps tput
    3.44% CPU utilization

  - Without fix (GRO disabled on VXLAN interface)

    Verified no GRO is happening.

    9084 MBps tput
    5.54% CPU utilization

Signed-off-by: Tom Herbert <tom@herbertland.com>
---
 drivers/net/vxlan.c | 9 +++++++--
 include/net/vxlan.h | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/net/vxlan.c b/drivers/net/vxlan.c
index 60b5b42..ef30037 100644
--- a/drivers/net/vxlan.c
+++ b/drivers/net/vxlan.c
@@ -1208,7 +1208,7 @@ static void vxlan_rcv(struct vxlan_sock *vs, struct sk_buff *skb,
 	stats->rx_bytes += skb->len;
 	u64_stats_update_end(&stats->syncp);
 
-	netif_rx(skb);
+	gro_cells_receive(&vxlan->gro_cells, skb);
 
 	return;
 drop:
@@ -2436,6 +2436,8 @@ static void vxlan_setup(struct net_device *dev)
 
 	vxlan->dev = dev;
 
+	gro_cells_init(&vxlan->gro_cells, dev);
+
 	for (h = 0; h < FDB_HASH_SIZE; ++h)
 		INIT_HLIST_HEAD(&vxlan->fdb_head[h]);
 }
@@ -2880,6 +2882,7 @@ static void vxlan_dellink(struct net_device *dev, struct list_head *head)
 		hlist_del_rcu(&vxlan->hlist);
 	spin_unlock(&vn->sock_lock);
 
+	gro_cells_destroy(&vxlan->gro_cells);
 	list_del(&vxlan->next);
 	unregister_netdevice_queue(dev, head);
 }
@@ -3088,8 +3091,10 @@ static void __net_exit vxlan_exit_net(struct net *net)
 		/* If vxlan->dev is in the same netns, it has already been added
 		 * to the list by the previous loop.
 		 */
-		if (!net_eq(dev_net(vxlan->dev), net))
+		if (!net_eq(dev_net(vxlan->dev), net)) {
+			gro_cells_destroy(&vxlan->gro_cells);
 			unregister_netdevice_queue(vxlan->dev, &list);
+		}
 	}
 
 	unregister_netdevice_many(&list);
diff --git a/include/net/vxlan.h b/include/net/vxlan.h
index eb8d721..21f1ff5 100644
--- a/include/net/vxlan.h
+++ b/include/net/vxlan.h
@@ -161,6 +161,7 @@ struct vxlan_dev {
 	struct timer_list age_timer;
 	spinlock_t	  hash_lock;
 	unsigned int	  addrcnt;
+	struct gro_cells  gro_cells;
 
 	struct vxlan_config	cfg;
 
-- 
1.8.5.6

[PATCH v2 net-next 3/4] ipv6: Add gro functions to sit_offloads

[Tom Herbert]

For GRO to work with sit we need gro_receive and gro_complete populated
in the sit_offload structure.

Signed-off-by: Tom Herbert <tom@herbertland.com>
---
 net/ipv6/ip6_offload.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
index 08b6204..e893cd1 100644
--- a/net/ipv6/ip6_offload.c
+++ b/net/ipv6/ip6_offload.c
@@ -292,6 +292,8 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
 static const struct net_offload sit_offload = {
 	.callbacks = {
 		.gso_segment	= ipv6_gso_segment,
+		.gro_receive	= ipv6_gro_receive,
+		.gro_complete	= ipv6_gro_complete,
 	},
 };
 
-- 
1.8.5.6

[PATCH v2 net-next 4/4] fou: Do WARN_ON_ONCE in gue_gro_receive for bad proto callbacks

[Tom Herbert]

Do WARN_ON_ONCE instead of WARN_ON in gue_gro_receive when the offload
callcaks are bad (either don't exist or gro_receive is not specified).

Signed-off-by: Tom Herbert <tom@herbertland.com>
---
 net/ipv4/fou.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/ipv4/fou.c b/net/ipv4/fou.c
index eb11f95..2d1646c 100644
--- a/net/ipv4/fou.c
+++ b/net/ipv4/fou.c
@@ -347,7 +347,7 @@ static struct sk_buff **gue_gro_receive(struct sk_buff **head,
 	rcu_read_lock();
 	offloads = NAPI_GRO_CB(skb)->is_ipv6 ? inet6_offloads : inet_offloads;
 	ops = rcu_dereference(offloads[guehdr->proto_ctype]);
-	if (WARN_ON(!ops || !ops->callbacks.gro_receive))
+	if (WARN_ON_ONCE(!ops || !ops->callbacks.gro_receive))
 		goto out_unlock;
 
 	pp = ops->callbacks.gro_receive(head, skb);
-- 
1.8.5.6

Re: [PATCH v2 net-next 3/4] ipv6: Add gro functions to sit_offloads

[Jesse Gross]

On Mon, Aug 3, 2015 at 10:11 AM, Tom Herbert <tom@herbertland.com> wrote:
> For GRO to work with sit we need gro_receive and gro_complete populated
> in the sit_offload structure.
>
> Signed-off-by: Tom Herbert <tom@herbertland.com>

You might want to checkout the recent history on this file unless
there's something that's changed in the last couple of weeks:

commit fdbf5b097bbd9693a86c0b8bfdd071a9a2117cfc
Author: Herbert Xu <herbert@gondor.apana.org.au>
Date:   Mon Jul 20 17:55:38 2015 +0800

    Revert "sit: Add gro callbacks to sit_offload"

    This patch reverts 19424e052fb44da2f00d1a868cbb51f3e9f4bbb5 ("sit:
    Add gro callbacks to sit_offload") because it generates packets
    that cannot be handled even by our own GSO.

    Reported-by: Wolfgang Walter <linux@stwm.de>
    Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
    Signed-off-by: David S. Miller <davem@davemloft.net>

Re: [PATCH v2 net-next 0/4] gro: Fixes for tunnels and GRO

[David Miller]

From: Tom Herbert <tom@herbertland.com>
Date: Mon, 3 Aug 2015 10:11:14 -0700

> This patch set addresses some issue related to tunneling and GRO:

As others have mentioned, the SIT GRO support was explicitly reverted
because it is broken and there were bug reports against it.

Until those are resolved you cannot put the GRO methods back into
the SIT driver.

THanks.

Re: [PATCH v2 net-next 3/4] ipv6: Add gro functions to sit_offloads

[Eric Dumazet]

On Thu, 2015-08-06 at 17:15 -0700, Jesse Gross wrote:
> On Mon, Aug 3, 2015 at 10:11 AM, Tom Herbert <tom@herbertland.com> wrote:
> > For GRO to work with sit we need gro_receive and gro_complete populated
> > in the sit_offload structure.
> >
> > Signed-off-by: Tom Herbert <tom@herbertland.com>
> 
> You might want to checkout the recent history on this file unless
> there's something that's changed in the last couple of weeks:
> 
> commit fdbf5b097bbd9693a86c0b8bfdd071a9a2117cfc
> Author: Herbert Xu <herbert@gondor.apana.org.au>
> Date:   Mon Jul 20 17:55:38 2015 +0800
> 
>     Revert "sit: Add gro callbacks to sit_offload"
> 
>     This patch reverts 19424e052fb44da2f00d1a868cbb51f3e9f4bbb5 ("sit:
>     Add gro callbacks to sit_offload") because it generates packets
>     that cannot be handled even by our own GSO.
> 
>     Reported-by: Wolfgang Walter <linux@stwm.de>
>     Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
> --


What about the following more complete patch ?

We properly set skb->encapsulation and inner network header as some NIC
drivers depend on it. Our GSO should also work properly I think.

Wolfgang, could you please test it ? (this is a patch on top of David
Miller net-next tree)

Both Google and Facebook are eager to get proper GRO/SIT support ;)

Thanks !

diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
index 08b62047c67f..eeca943f12dc 100644
--- a/net/ipv6/ip6_offload.c
+++ b/net/ipv6/ip6_offload.c
@@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 	struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
 	int err = -ENOSYS;
 
+	if (skb->encapsulation)
+		skb_set_inner_network_header(skb, nhoff);
+
 	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
 
 	rcu_read_lock();
@@ -280,6 +283,13 @@ out_unlock:
 	return err;
 }
 
+static int sit_gro_complete(struct sk_buff *skb, int nhoff)
+{
+	skb->encapsulation = 1;
+	skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
+	return ipv6_gro_complete(skb, nhoff);
+}
+
 static struct packet_offload ipv6_packet_offload __read_mostly = {
 	.type = cpu_to_be16(ETH_P_IPV6),
 	.callbacks = {
@@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
 static const struct net_offload sit_offload = {
 	.callbacks = {
 		.gso_segment	= ipv6_gso_segment,
+		.gro_receive    = ipv6_gro_receive,
+		.gro_complete   = sit_gro_complete,
 	},
 };
 

Re: [PATCH v2 net-next 3/4] ipv6: Add gro functions to sit_offloads

[Eric Dumazet]

On Fri, 2015-10-16 at 08:23 -0700, Eric Dumazet wrote:

> What about the following more complete patch ?
> 
> We properly set skb->encapsulation and inner network header as some NIC
> drivers depend on it. Our GSO should also work properly I think.
> 
> Wolfgang, could you please test it ? (this is a patch on top of David
> Miller net-next tree)
> 
> Both Google and Facebook are eager to get proper GRO/SIT support ;)
> 
> Thanks !
> 
> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
> index 08b62047c67f..eeca943f12dc 100644
> --- a/net/ipv6/ip6_offload.c
> +++ b/net/ipv6/ip6_offload.c
> @@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
>  	struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
>  	int err = -ENOSYS;
>  
> +	if (skb->encapsulation)
> +		skb_set_inner_network_header(skb, nhoff);
> +
>  	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
>  
>  	rcu_read_lock();
> @@ -280,6 +283,13 @@ out_unlock:
>  	return err;
>  }
>  
> +static int sit_gro_complete(struct sk_buff *skb, int nhoff)
> +{
> +	skb->encapsulation = 1;
> +	skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
> +	return ipv6_gro_complete(skb, nhoff);
> +}
> +
>  static struct packet_offload ipv6_packet_offload __read_mostly = {
>  	.type = cpu_to_be16(ETH_P_IPV6),
>  	.callbacks = {
> @@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
>  static const struct net_offload sit_offload = {
>  	.callbacks = {
>  		.gso_segment	= ipv6_gso_segment,
> +		.gro_receive    = ipv6_gro_receive,
> +		.gro_complete   = sit_gro_complete,
>  	},
>  };
>  
> 

FYI, I've tested this patch and confirm it works properly.

I will therefore submit an official patch.

I'll also submit a patch against dummy device to add missing features.

[PATCH net-next] ipv6: gro: support sit protocol

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Tom Herbert added SIT support to GRO with commit
19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
later reverted by Herbert Xu.

The problem came because Tom patch was building GRO
packets without proper meta data : If packets were locally
delivered, we would not care.

But if packets needed to be forwarded, GSO engine was not
able to segment individual segments.

With the following patch, we correctly set skb->encapsulation
and inner network header. We also update gso_type.

Tested:

Server :
netserver
modprobe dummy
ifconfig dummy0 8.0.0.1 netmask 255.255.255.0 up
arp -s 8.0.0.100 4e:32:51:04:47:e5
iptables -I INPUT -s 10.246.7.151 -j TEE --gateway 8.0.0.100
ifconfig sixtofour0
sixtofour0 Link encap:IPv6-in-IPv4  
          inet6 addr: 2002:af6:798::1/128 Scope:Global
          inet6 addr: 2002:af6:798::/128 Scope:Global
          UP RUNNING NOARP  MTU:1480  Metric:1
          RX packets:411169 errors:0 dropped:0 overruns:0 frame:0
          TX packets:409414 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:20319631739 (20.3 GB)  TX bytes:29529556 (29.5 MB)

Client :
netperf -H 2002:af6:798::1 -l 1000 &

Checked on server traffic copied on dummy0 and verify segments were
properly rebuilt, with proper IP headers, TCP checksums...

tcpdump on eth0 shows proper GRO aggregation takes place.

Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 net/ipv6/ip6_offload.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
index 08b62047c67f..eeca943f12dc 100644
--- a/net/ipv6/ip6_offload.c
+++ b/net/ipv6/ip6_offload.c
@@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
 	struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
 	int err = -ENOSYS;
 
+	if (skb->encapsulation)
+		skb_set_inner_network_header(skb, nhoff);
+
 	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
 
 	rcu_read_lock();
@@ -280,6 +283,13 @@ out_unlock:
 	return err;
 }
 
+static int sit_gro_complete(struct sk_buff *skb, int nhoff)
+{
+	skb->encapsulation = 1;
+	skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
+	return ipv6_gro_complete(skb, nhoff);
+}
+
 static struct packet_offload ipv6_packet_offload __read_mostly = {
 	.type = cpu_to_be16(ETH_P_IPV6),
 	.callbacks = {
@@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
 static const struct net_offload sit_offload = {
 	.callbacks = {
 		.gso_segment	= ipv6_gso_segment,
+		.gro_receive    = ipv6_gro_receive,
+		.gro_complete   = sit_gro_complete,
 	},
 };
 

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Tom Herbert]

On Mon, Oct 19, 2015 at 8:40 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> From: Eric Dumazet <edumazet@google.com>
>
> Tom Herbert added SIT support to GRO with commit
> 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
> later reverted by Herbert Xu.
>
> The problem came because Tom patch was building GRO
> packets without proper meta data : If packets were locally
> delivered, we would not care.
>
> But if packets needed to be forwarded, GSO engine was not
> able to segment individual segments.
>
> With the following patch, we correctly set skb->encapsulation
> and inner network header. We also update gso_type.
>
> Tested:
>
> Server :
> netserver
> modprobe dummy
> ifconfig dummy0 8.0.0.1 netmask 255.255.255.0 up
> arp -s 8.0.0.100 4e:32:51:04:47:e5
> iptables -I INPUT -s 10.246.7.151 -j TEE --gateway 8.0.0.100
> ifconfig sixtofour0
> sixtofour0 Link encap:IPv6-in-IPv4
>           inet6 addr: 2002:af6:798::1/128 Scope:Global
>           inet6 addr: 2002:af6:798::/128 Scope:Global
>           UP RUNNING NOARP  MTU:1480  Metric:1
>           RX packets:411169 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:409414 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:20319631739 (20.3 GB)  TX bytes:29529556 (29.5 MB)
>
> Client :
> netperf -H 2002:af6:798::1 -l 1000 &
>
> Checked on server traffic copied on dummy0 and verify segments were
> properly rebuilt, with proper IP headers, TCP checksums...
>
> tcpdump on eth0 shows proper GRO aggregation takes place.
>
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Acked-by: Tom Herbert <tom@herbertland.com>

> ---
>  net/ipv6/ip6_offload.c |   12 ++++++++++++
>  1 file changed, 12 insertions(+)
>
> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
> index 08b62047c67f..eeca943f12dc 100644
> --- a/net/ipv6/ip6_offload.c
> +++ b/net/ipv6/ip6_offload.c
> @@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int nhoff)
>         struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
>         int err = -ENOSYS;
>
> +       if (skb->encapsulation)
> +               skb_set_inner_network_header(skb, nhoff);
> +
>         iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
>
>         rcu_read_lock();
> @@ -280,6 +283,13 @@ out_unlock:
>         return err;
>  }
>
> +static int sit_gro_complete(struct sk_buff *skb, int nhoff)
> +{
> +       skb->encapsulation = 1;
> +       skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
> +       return ipv6_gro_complete(skb, nhoff);
> +}
> +
>  static struct packet_offload ipv6_packet_offload __read_mostly = {
>         .type = cpu_to_be16(ETH_P_IPV6),
>         .callbacks = {
> @@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload __read_mostly = {
>  static const struct net_offload sit_offload = {
>         .callbacks = {
>                 .gso_segment    = ipv6_gso_segment,
> +               .gro_receive    = ipv6_gro_receive,
> +               .gro_complete   = sit_gro_complete,
>         },
>  };
>
>
>

Re: [PATCH v2 net-next 3/4] ipv6: Add gro functions to sit_offloads

[Wolfgang Walter]

Hello Eric!

Am Freitag, 16. Oktober 2015, 08:23:49 schrieb Eric Dumazet:
> On Thu, 2015-08-06 at 17:15 -0700, Jesse Gross wrote:
> > On Mon, Aug 3, 2015 at 10:11 AM, Tom Herbert <tom@herbertland.com> wrote:
> > > For GRO to work with sit we need gro_receive and gro_complete populated
> > > in the sit_offload structure.
> > > 
> > > Signed-off-by: Tom Herbert <tom@herbertland.com>
> > 
> > You might want to checkout the recent history on this file unless
> > there's something that's changed in the last couple of weeks:
> > 
> > commit fdbf5b097bbd9693a86c0b8bfdd071a9a2117cfc
> > Author: Herbert Xu <herbert@gondor.apana.org.au>
> > Date:   Mon Jul 20 17:55:38 2015 +0800
> > 
> >     Revert "sit: Add gro callbacks to sit_offload"
> >     
> >     This patch reverts 19424e052fb44da2f00d1a868cbb51f3e9f4bbb5 ("sit:
> >     Add gro callbacks to sit_offload") because it generates packets
> >     that cannot be handled even by our own GSO.
> >     
> >     Reported-by: Wolfgang Walter <linux@stwm.de>
> >     Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
> >     Signed-off-by: David S. Miller <davem@davemloft.net>
> > 
> > --
> 
> What about the following more complete patch ?
> 
> We properly set skb->encapsulation and inner network header as some NIC
> drivers depend on it. Our GSO should also work properly I think.
> 
> Wolfgang, could you please test it ? (this is a patch on top of David
> Miller net-next tree)
> 
> Both Google and Facebook are eager to get proper GRO/SIT support ;)
> 
> Thanks !

In the moment I can't test it, sorry.

Will test it next week. But as I see you already did that yourself.

> 
> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
> index 08b62047c67f..eeca943f12dc 100644
> --- a/net/ipv6/ip6_offload.c
> +++ b/net/ipv6/ip6_offload.c
> @@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int
> nhoff) struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
>  	int err = -ENOSYS;
> 
> +	if (skb->encapsulation)
> +		skb_set_inner_network_header(skb, nhoff);
> +
>  	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
> 
>  	rcu_read_lock();
> @@ -280,6 +283,13 @@ out_unlock:
>  	return err;
>  }
> 
> +static int sit_gro_complete(struct sk_buff *skb, int nhoff)
> +{
> +	skb->encapsulation = 1;
> +	skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
> +	return ipv6_gro_complete(skb, nhoff);
> +}
> +
>  static struct packet_offload ipv6_packet_offload __read_mostly = {
>  	.type = cpu_to_be16(ETH_P_IPV6),
>  	.callbacks = {
> @@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload
> __read_mostly = { static const struct net_offload sit_offload = {
>  	.callbacks = {
>  		.gso_segment	= ipv6_gso_segment,
> +		.gro_receive    = ipv6_gro_receive,
> +		.gro_complete   = sit_gro_complete,
>  	},
>  };

Thank you very much,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: [PATCH net-next] ipv6: gro: support sit protocol

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Mon, 19 Oct 2015 20:40:17 -0700

> From: Eric Dumazet <edumazet@google.com>
> 
> Tom Herbert added SIT support to GRO with commit
> 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
> later reverted by Herbert Xu.
> 
> The problem came because Tom patch was building GRO
> packets without proper meta data : If packets were locally
> delivered, we would not care.
> 
> But if packets needed to be forwarded, GSO engine was not
> able to segment individual segments.
> 
> With the following patch, we correctly set skb->encapsulation
> and inner network header. We also update gso_type.
> 
> Tested:
 ...
> Checked on server traffic copied on dummy0 and verify segments were
> properly rebuilt, with proper IP headers, TCP checksums...
> 
> tcpdump on eth0 shows proper GRO aggregation takes place.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>

Applied.

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Wolfgang Walter]

Am Montag, 19. Oktober 2015, 20:40:17 schrieb Eric Dumazet:
> From: Eric Dumazet <edumazet@google.com>
> 
> Tom Herbert added SIT support to GRO with commit
> 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
> later reverted by Herbert Xu.
> 
> The problem came because Tom patch was building GRO
> packets without proper meta data : If packets were locally
> delivered, we would not care.
> 
> But if packets needed to be forwarded, GSO engine was not
> able to segment individual segments.
> 
> With the following patch, we correctly set skb->encapsulation
> and inner network header. We also update gso_type.
> 

I'm running 4.1.11 / 4.1.12 with this patch on top now since over a week. 
ISATAP works fine.
 

> Tested:
> 
> Server :
> netserver
> modprobe dummy
> ifconfig dummy0 8.0.0.1 netmask 255.255.255.0 up
> arp -s 8.0.0.100 4e:32:51:04:47:e5
> iptables -I INPUT -s 10.246.7.151 -j TEE --gateway 8.0.0.100
> ifconfig sixtofour0
> sixtofour0 Link encap:IPv6-in-IPv4
>           inet6 addr: 2002:af6:798::1/128 Scope:Global
>           inet6 addr: 2002:af6:798::/128 Scope:Global
>           UP RUNNING NOARP  MTU:1480  Metric:1
>           RX packets:411169 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:409414 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:20319631739 (20.3 GB)  TX bytes:29529556 (29.5 MB)
> 
> Client :
> netperf -H 2002:af6:798::1 -l 1000 &
> 
> Checked on server traffic copied on dummy0 and verify segments were
> properly rebuilt, with proper IP headers, TCP checksums...
> 
> tcpdump on eth0 shows proper GRO aggregation takes place.
> 
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
>  net/ipv6/ip6_offload.c |   12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
> index 08b62047c67f..eeca943f12dc 100644
> --- a/net/ipv6/ip6_offload.c
> +++ b/net/ipv6/ip6_offload.c
> @@ -264,6 +264,9 @@ static int ipv6_gro_complete(struct sk_buff *skb, int
> nhoff) struct ipv6hdr *iph = (struct ipv6hdr *)(skb->data + nhoff);
>  	int err = -ENOSYS;
> 
> +	if (skb->encapsulation)
> +		skb_set_inner_network_header(skb, nhoff);
> +
>  	iph->payload_len = htons(skb->len - nhoff - sizeof(*iph));
> 
>  	rcu_read_lock();
> @@ -280,6 +283,13 @@ out_unlock:
>  	return err;
>  }
> 
> +static int sit_gro_complete(struct sk_buff *skb, int nhoff)
> +{
> +	skb->encapsulation = 1;
> +	skb_shinfo(skb)->gso_type |= SKB_GSO_SIT;
> +	return ipv6_gro_complete(skb, nhoff);
> +}
> +
>  static struct packet_offload ipv6_packet_offload __read_mostly = {
>  	.type = cpu_to_be16(ETH_P_IPV6),
>  	.callbacks = {
> @@ -292,6 +302,8 @@ static struct packet_offload ipv6_packet_offload
> __read_mostly = { static const struct net_offload sit_offload = {
>  	.callbacks = {
>  		.gso_segment	= ipv6_gso_segment,
> +		.gro_receive    = ipv6_gro_receive,
> +		.gro_complete   = sit_gro_complete,
>  	},
>  };

Thanks,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Eric Dumazet]

On Tue, 2015-11-03 at 13:57 +0100, Wolfgang Walter wrote:
> Am Montag, 19. Oktober 2015, 20:40:17 schrieb Eric Dumazet:
> > From: Eric Dumazet <edumazet@google.com>
> > 
> > Tom Herbert added SIT support to GRO with commit
> > 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
> > later reverted by Herbert Xu.
> > 
> > The problem came because Tom patch was building GRO
> > packets without proper meta data : If packets were locally
> > delivered, we would not care.
> > 
> > But if packets needed to be forwarded, GSO engine was not
> > able to segment individual segments.
> > 
> > With the following patch, we correctly set skb->encapsulation
> > and inner network header. We also update gso_type.
> > 
> 
> I'm running 4.1.11 / 4.1.12 with this patch on top now since over a week. 
> ISATAP works fine.

Perfect ! thanks a lot for testing !

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Tom Herbert]

On Tue, Nov 3, 2015 at 10:07 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Tue, 2015-11-03 at 13:57 +0100, Wolfgang Walter wrote:
>> Am Montag, 19. Oktober 2015, 20:40:17 schrieb Eric Dumazet:
>> > From: Eric Dumazet <edumazet@google.com>
>> >
>> > Tom Herbert added SIT support to GRO with commit
>> > 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
>> > later reverted by Herbert Xu.
>> >
>> > The problem came because Tom patch was building GRO
>> > packets without proper meta data : If packets were locally
>> > delivered, we would not care.
>> >
>> > But if packets needed to be forwarded, GSO engine was not
>> > able to segment individual segments.
>> >
>> > With the following patch, we correctly set skb->encapsulation
>> > and inner network header. We also update gso_type.
>> >
>>
>> I'm running 4.1.11 / 4.1.12 with this patch on top now since over a week.
>> ISATAP works fine.
>
> Perfect ! thanks a lot for testing !
>

Thanks for all the work on this!

Tom

>

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Wolfgang Walter]

Am Dienstag, 3. November 2015, 05:07:33 schrieb Eric Dumazet:
> On Tue, 2015-11-03 at 13:57 +0100, Wolfgang Walter wrote:
> > Am Montag, 19. Oktober 2015, 20:40:17 schrieb Eric Dumazet:
> > > From: Eric Dumazet <edumazet@google.com>
> > > 
> > > Tom Herbert added SIT support to GRO with commit
> > > 19424e052fb4 ("sit: Add gro callbacks to sit_offload"),
> > > later reverted by Herbert Xu.
> > > 
> > > The problem came because Tom patch was building GRO
> > > packets without proper meta data : If packets were locally
> > > delivered, we would not care.
> > > 
> > > But if packets needed to be forwarded, GSO engine was not
> > > able to segment individual segments.
> > > 
> > > With the following patch, we correctly set skb->encapsulation
> > > and inner network header. We also update gso_type.
> > 
> > I'm running 4.1.11 / 4.1.12 with this patch on top now since over a week.
> > ISATAP works fine.
> 
> Perfect ! thanks a lot for testing !

Today I found a problem: on a router forwarding GRE-packets (ipv4) (it is not 
the endpount) the interface (intel igb) stops sending packets after some time. 
I think this happens when an ISATAP packet is inside the GRE-packet.

	gre packets arrives on eth0
	eth1 stops sending (receiving still works)
	ethtool -r eth1
	eth1 works again for some time

Switching GRO off on eth0 "fixes" the problem.

I didn't test vanilla 4.1.12 yet, though. Until today 4.1.11 has been running 
on the router. What I tested was your patch
	"gre_gso_segment() chokes if SIT frames were aggregated by GRO engine."
but did not solve the problem.

So I would not recommend to backport it to longterm 4.1.

My plans are:

* test vanilla 4.1.12
* test 4.3

I want to test 4.3 on another router first, though.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Eric Dumazet]

On Wed, 2015-11-04 at 13:19 +0100, Wolfgang Walter wrote:

> Today I found a problem: on a router forwarding GRE-packets (ipv4) (it is not 
> the endpount) the interface (intel igb) stops sending packets after some time. 
> I think this happens when an ISATAP packet is inside the GRE-packet.
> 
> 	gre packets arrives on eth0
> 	eth1 stops sending (receiving still works)
> 	ethtool -r eth1
> 	eth1 works again for some time
> 
> Switching GRO off on eth0 "fixes" the problem.
> 
> I didn't test vanilla 4.1.12 yet, though. Until today 4.1.11 has been running 
> on the router. What I tested was your patch
> 	"gre_gso_segment() chokes if SIT frames were aggregated by GRO engine."
> but did not solve the problem.
> 
> So I would not recommend to backport it to longterm 4.1.
> 
> My plans are:
> 
> * test vanilla 4.1.12
> * test 4.3
> 
> I want to test 4.3 on another router first, though.

If the NIC stops sending packets after some time, it might be an igb
issue.

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Wolfgang Walter]

Am Mittwoch, 4. November 2015, 04:40:51 schrieb Eric Dumazet:
> On Wed, 2015-11-04 at 13:19 +0100, Wolfgang Walter wrote:
> > Today I found a problem: on a router forwarding GRE-packets (ipv4) (it is
> > not the endpount) the interface (intel igb) stops sending packets after
> > some time. I think this happens when an ISATAP packet is inside the
> > GRE-packet.> 
> > 	gre packets arrives on eth0
> > 	eth1 stops sending (receiving still works)
> > 	ethtool -r eth1
> > 	eth1 works again for some time
> > 
> > Switching GRO off on eth0 "fixes" the problem.
> > 
> > I didn't test vanilla 4.1.12 yet, though. Until today 4.1.11 has been
> > running on the router. What I tested was your patch
> > 
> > 	"gre_gso_segment() chokes if SIT frames were aggregated by GRO 
engine."
> > 
> > but did not solve the problem.
> > 
> > So I would not recommend to backport it to longterm 4.1.
> > 
> > My plans are:
> > 
> > * test vanilla 4.1.12
> > * test 4.3
> > 
> > I want to test 4.3 on another router first, though.
> 
> If the NIC stops sending packets after some time, it might be an igb
> issue.

Yes, maybe igb has a problem sending a gro-packet if it is an isatap in gre.

igb has no problem sending gro-packets which are pure isatap or which are ipv4 
(tcp/udp) in gre with 4.1.12 + these patches.

And it had no problem with 4.1.11 with isatap in gre.

Disabling gso for the interface does help.

I'll test pure 4.1.12 soon.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Eric Dumazet]

On Wed, 2015-11-04 at 15:09 +0100, Wolfgang Walter wrote:

> 
> Yes, maybe igb has a problem sending a gro-packet if it is an isatap in gre.

We might detect this condition properly from igb ndo_features_check
method.

It currently uses plain passthru_features_check()



> 
> igb has no problem sending gro-packets which are pure isatap or which are ipv4 
> (tcp/udp) in gre with 4.1.12 + these patches.
> 
> And it had no problem with 4.1.11 with isatap in gre.
> 
> Disabling gso for the interface does help.

My patch was aimed for 4.4, not sure about backports to old kernels...

Re: [PATCH net-next] ipv6: gro: support sit protocol

[Wolfgang Walter]

Am Mittwoch, 4. November 2015, 07:13:07 schrieb Eric Dumazet:
> On Wed, 2015-11-04 at 15:09 +0100, Wolfgang Walter wrote:
> > Yes, maybe igb has a problem sending a gro-packet if it is an isatap in
> > gre.
> We might detect this condition properly from igb ndo_features_check
> method.
> 
> It currently uses plain passthru_features_check()
> 
> > igb has no problem sending gro-packets which are pure isatap or which are
> > ipv4 (tcp/udp) in gre with 4.1.12 + these patches.
> > 
> > And it had no problem with 4.1.11 with isatap in gre.
> > 
> > Disabling gso for the interface does help.
> 
> My patch was aimed for 4.4, not sure about backports to old kernels...

I know. I cannot test 4.4 (or net-next) on that router, though, as I don't 
have easy physical access to it if it crashes or I loose network connectivity. 
For such tests I must send someone in situ.

As 4.4 will be the next longterm kernel I definitivly will do that for 4.4-rc2 
or 4.4-rc3.

I think your patch is correct for 4.1 in the sense that ISATAP is correctly 
handled. Only SIT in GRE triggers this and if it is indeed igb I will see it 
probably in 4.4 ;-), too.

I now tested an unmodified 4.1.12 and it shows no problems.


Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts