* GRO: forwading ISATAP packets is very slow with kernel 4.1 @ 2015-07-14 22:16 Wolfgang Walter 2015-07-15 0:08 ` Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-14 22:16 UTC (permalink / raw) To: netdev; +Cc: Herbert Xu Hello, I upgraded routers from 3.14.x to 4.1.2. Forwarding ISATAP-packets (IPv4 packets with IPv6 payload) is very slow with 4.1 if GRO is enabled (youtube for example about 64kbit). Disabling GRO on the interfaces restores performance to values comparable to 3.14.x. The kernel is build with IPv6 support but IPv6 is disabled via kernel command line. The router is not a tunnel endpoint, it only forwards the ISATAP- packets. MTU is 1500 on both interfaces. Netfilter conntrack is not used and disabling netfilter has no effect. Regards, -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: GRO: forwading ISATAP packets is very slow with kernel 4.1 2015-07-14 22:16 GRO: forwading ISATAP packets is very slow with kernel 4.1 Wolfgang Walter @ 2015-07-15 0:08 ` Herbert Xu 2015-07-15 0:34 ` Wolfgang Walter 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-15 0:08 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev On Wed, Jul 15, 2015 at 12:16:30AM +0200, Wolfgang Walter wrote: > Hello, > > I upgraded routers from 3.14.x to 4.1.2. Forwarding ISATAP-packets (IPv4 > packets with IPv6 payload) is very slow with 4.1 if GRO is enabled (youtube > for example about 64kbit). Disabling GRO on the interfaces restores > performance to values comparable to 3.14.x. > > The kernel is build with IPv6 support but IPv6 is disabled via kernel command > line. The router is not a tunnel endpoint, it only forwards the ISATAP- > packets. MTU is 1500 on both interfaces. Netfilter conntrack is not used and > disabling netfilter has no effect. Can you run some tcpdumps and post the results in the two cases? Thanks, -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: GRO: forwading ISATAP packets is very slow with kernel 4.1 2015-07-15 0:08 ` Herbert Xu @ 2015-07-15 0:34 ` Wolfgang Walter 2015-07-15 9:50 ` Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-15 0:34 UTC (permalink / raw) To: Herbert Xu; +Cc: netdev Am Mittwoch, 15. Juli 2015, 08:08:49 schrieben Sie: > On Wed, Jul 15, 2015 at 12:16:30AM +0200, Wolfgang Walter wrote: > > Hello, > > > > I upgraded routers from 3.14.x to 4.1.2. Forwarding ISATAP-packets (IPv4 > > packets with IPv6 payload) is very slow with 4.1 if GRO is enabled > > (youtube > > for example about 64kbit). Disabling GRO on the interfaces restores > > performance to values comparable to 3.14.x. > > > > The kernel is build with IPv6 support but IPv6 is disabled via kernel > > command line. The router is not a tunnel endpoint, it only forwards the > > ISATAP- packets. MTU is 1500 on both interfaces. Netfilter conntrack is > > not used and disabling netfilter has no effect. > > Can you run some tcpdumps and post the results in the two cases? > Yes, but I need the cooperation of one of our customers. I wonder if the router may treat all ipv6-tcp connections of a host as a single flow as all those ipv6-packets are embedded in ipv4-packets with the ipv4-address of the host and the ipv4-address of the isatap-gateway and next- header is 41. It may only looks at those? Regards, -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: GRO: forwading ISATAP packets is very slow with kernel 4.1 2015-07-15 0:34 ` Wolfgang Walter @ 2015-07-15 9:50 ` Herbert Xu 2015-07-15 12:25 ` Wolfgang Walter 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-15 9:50 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev On Wed, Jul 15, 2015 at 02:34:41AM +0200, Wolfgang Walter wrote: > > I wonder if the router may treat all ipv6-tcp connections of a host as a > single flow as all those ipv6-packets are embedded in ipv4-packets with the > ipv4-address of the host and the ipv4-address of the isatap-gateway and next- > header is 41. It may only looks at those? I think GRO is actually working fine. Can you show me the output of ethtool -k eth1? If hardware TSO is enabled can you try disabling it to see if it helps? Thanks, -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: GRO: forwading ISATAP packets is very slow with kernel 4.1 2015-07-15 9:50 ` Herbert Xu @ 2015-07-15 12:25 ` Wolfgang Walter 2015-07-16 0:23 ` sit: Set SKB_GSO_SIT bit when performing GRO Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-15 12:25 UTC (permalink / raw) To: Herbert Xu; +Cc: netdev Am Mittwoch, 15. Juli 2015, 17:50:20 schrieb Herbert Xu: > On Wed, Jul 15, 2015 at 02:34:41AM +0200, Wolfgang Walter wrote: > > I wonder if the router may treat all ipv6-tcp connections of a host as a > > single flow as all those ipv6-packets are embedded in ipv4-packets with > > the > > ipv4-address of the host and the ipv4-address of the isatap-gateway and > > next- header is 41. It may only looks at those? > > I think GRO is actually working fine. Can you show me the output > of ethtool -k eth1? If hardware TSO is enabled can you try disabling > it to see if it helps? > > Thanks, Yes. Switching TSO off and leaving GRO on works, too. Thanks, -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-15 12:25 ` Wolfgang Walter @ 2015-07-16 0:23 ` Herbert Xu 2015-07-16 10:58 ` Wolfgang Walter 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-16 0:23 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev, Tom Herbert On Wed, Jul 15, 2015 at 02:25:59PM +0200, Wolfgang Walter wrote: > > Yes. Switching TSO off and leaving GRO on works, too. OK, could you please try this patch? ---8<--- We need to set the SKB_GSO_SIT bit if we detect a 6-in-4 tunnel when doing GRO. Otherwise we may throw a packet at TSO hardware that doesn't know what to do with it. Fixes: 19424e052fb4 ("sit: Add gro callbacks to sit_offload") Reported-by: Wolfgang Walter <linux@stwm.de> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index e893cd1..1252eac 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -289,11 +289,21 @@ static struct packet_offload ipv6_packet_offload __read_mostly = { }, }; +static int sit_gro_complete(struct sk_buff *skb, int nhoff) +{ + int err = ipv6_gro_complete(skb, nhoff); + + skb->encapsulation = 1; + skb_shinfo(skb)->gso_type |= SKB_GSO_SIT; + + return err; +} + static const struct net_offload sit_offload = { .callbacks = { .gso_segment = ipv6_gso_segment, .gro_receive = ipv6_gro_receive, - .gro_complete = ipv6_gro_complete, + .gro_complete = sit_gro_complete, }, }; -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-16 0:23 ` sit: Set SKB_GSO_SIT bit when performing GRO Herbert Xu @ 2015-07-16 10:58 ` Wolfgang Walter 2015-07-17 1:56 ` Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-16 10:58 UTC (permalink / raw) To: Herbert Xu; +Cc: netdev, Tom Herbert Am Donnerstag, 16. Juli 2015, 08:23:50 schrieb Herbert Xu: > On Wed, Jul 15, 2015 at 02:25:59PM +0200, Wolfgang Walter wrote: > > Yes. Switching TSO off and leaving GRO on works, too. > > OK, could you please try this patch? Patch works here. Thanks, Wolfgang > > ---8<--- > We need to set the SKB_GSO_SIT bit if we detect a 6-in-4 tunnel > when doing GRO. Otherwise we may throw a packet at TSO hardware > that doesn't know what to do with it. > > Fixes: 19424e052fb4 ("sit: Add gro callbacks to sit_offload") > Reported-by: Wolfgang Walter <linux@stwm.de> > Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> > > diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c > index e893cd1..1252eac 100644 > --- a/net/ipv6/ip6_offload.c > +++ b/net/ipv6/ip6_offload.c > @@ -289,11 +289,21 @@ static struct packet_offload ipv6_packet_offload > __read_mostly = { }, > }; > > +static int sit_gro_complete(struct sk_buff *skb, int nhoff) > +{ > + int err = ipv6_gro_complete(skb, nhoff); > + > + skb->encapsulation = 1; > + skb_shinfo(skb)->gso_type |= SKB_GSO_SIT; > + > + return err; > +} > + > static const struct net_offload sit_offload = { > .callbacks = { > .gso_segment = ipv6_gso_segment, > .gro_receive = ipv6_gro_receive, > - .gro_complete = ipv6_gro_complete, > + .gro_complete = sit_gro_complete, > }, > }; -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-16 10:58 ` Wolfgang Walter @ 2015-07-17 1:56 ` Herbert Xu 2015-07-17 15:38 ` Wolfgang Walter 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-17 1:56 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev, Tom Herbert On Thu, Jul 16, 2015 at 12:58:45PM +0200, Wolfgang Walter wrote: > Am Donnerstag, 16. Juli 2015, 08:23:50 schrieb Herbert Xu: > > On Wed, Jul 15, 2015 at 02:25:59PM +0200, Wolfgang Walter wrote: > > > Yes. Switching TSO off and leaving GRO on works, too. > > > > OK, could you please try this patch? > > Patch works here. Thanks for the confirmation. Let's add a tag for patchwork: Tested-by: Wolfgang Walter <linux@stwm.de> -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-17 1:56 ` Herbert Xu @ 2015-07-17 15:38 ` Wolfgang Walter 2015-07-20 6:14 ` Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-17 15:38 UTC (permalink / raw) To: Herbert Xu; +Cc: netdev, Tom Herbert Am Freitag, 17. Juli 2015, 09:56:51 schrieb Herbert Xu: > On Thu, Jul 16, 2015 at 12:58:45PM +0200, Wolfgang Walter wrote: > > Am Donnerstag, 16. Juli 2015, 08:23:50 schrieb Herbert Xu: > > > On Wed, Jul 15, 2015 at 02:25:59PM +0200, Wolfgang Walter wrote: > > > > Yes. Switching TSO off and leaving GRO on works, too. > > > > > > OK, could you please try this patch? > > > > Patch works here. > > Thanks for the confirmation. Let's add a tag for patchwork: > > Tested-by: Wolfgang Walter <linux@stwm.de> It seems that this patch may cause a problem with another one of our routers. Without the patch it had no problem, so I didn't tested it there. With that patch one interface blocks after some time. Not even arp requests get answered. It still receives packets though. Restarting the interface fixes the problem. Switching off gro for the other interface helps. This router is different from the other ones. It does not directly route isatap packets. It may routes isatap packets encapsulated in GRE packets, though. It is itself not an GRE-endpoint. The router does NAT. Basically it routes the GRE-tunnel packets unatted and NATs most of the rest. Not doing NAT and conntrack (and unloading all modules like nf_conntrack_ipv4, nf_defrag_ipv4) does not help. eth0: extern eth1: intern One (IPv4) GRE-tunnel is routed between eth0 und eth1. IPv6 ESP-tunnels are routed between eth0 and eth1 IPv4 UDP/TCP/ICMP from intern is natted with netfilter. eth1 stops sending with the patch after some time disabling gro on eth0 helps disabling tso or gso on eth0 and/or eth1 or both does not help eth0 and eth1 are both intel I350. Regards, -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-17 15:38 ` Wolfgang Walter @ 2015-07-20 6:14 ` Herbert Xu 2015-07-20 9:39 ` Wolfgang Walter 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-20 6:14 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev, Tom Herbert On Fri, Jul 17, 2015 at 05:38:30PM +0200, Wolfgang Walter wrote: > > eth1 stops sending with the patch after some time > disabling gro on eth0 helps > disabling tso or gso on eth0 and/or eth1 or both does not help > > eth0 and eth1 are both intel I350. What does ethtool -k eth1 say? Can you confirm that disabling tso on eth1 does not help? Because the most plausible explanation is that we're feeding some bogus TSO packet to the hardware causing a tx lockup. But in any case if it is a hardware lockup then it's no longer just a pure software bug. No matter what we do in the stack the hardware should not lock up (unless of course we're feeding it something that's completely bogus). If we can't figure this out then the safest solution would be to disable tunnel GRO completely because it's broken as it stands. Cheers, -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: sit: Set SKB_GSO_SIT bit when performing GRO 2015-07-20 6:14 ` Herbert Xu @ 2015-07-20 9:39 ` Wolfgang Walter 2015-07-20 9:55 ` Revert "sit: Add gro callbacks to sit_offload" Herbert Xu 0 siblings, 1 reply; 13+ messages in thread From: Wolfgang Walter @ 2015-07-20 9:39 UTC (permalink / raw) To: Herbert Xu; +Cc: netdev, Tom Herbert Am Montag, 20. Juli 2015, 14:14:59 schrieb Herbert Xu: > On Fri, Jul 17, 2015 at 05:38:30PM +0200, Wolfgang Walter wrote: > > eth1 stops sending with the patch after some time > > disabling gro on eth0 helps > > disabling tso or gso on eth0 and/or eth1 or both does not help > > > > eth0 and eth1 are both intel I350. > > What does ethtool -k eth1 say? With TSO enabled: # ethtool -k eth0 Features for eth0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off [fixed] receive-hashing: on highdma: on [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] hw-switch-offload: off [fixed] > > Can you confirm that disabling tso on eth1 does not help? Disabling TSO on eth1 does not help. > > Because the most plausible explanation is that we're feeding > some bogus TSO packet to the hardware causing a tx lockup. I run the unpatched 4.1.2 again since saturday without look. With your patch the network card hangs within 10 minutes or so. On the other hand I run the the patched kernel on serveral other routers (same hardware, by the way) without problems. So maybe the problem is that the former one routes GRE-tunnel-packets which contains ISATAP packets. I don't know how deep GRO/GSO inspects a packet. > > But in any case if it is a hardware lockup then it's no longer > just a pure software bug. No matter what we do in the stack > the hardware should not lock up (unless of course we're feeding > it something that's completely bogus). > > If we can't figure this out then the safest solution would be > to disable tunnel GRO completely because it's broken as it stands. > > Cheers, Regards, -- Wolfgang Walter Studentenwerk München Anstalt des öffentlichen Rechts ^ permalink raw reply [flat|nested] 13+ messages in thread
* Revert "sit: Add gro callbacks to sit_offload" 2015-07-20 9:39 ` Wolfgang Walter @ 2015-07-20 9:55 ` Herbert Xu 2015-07-20 23:53 ` David Miller 0 siblings, 1 reply; 13+ messages in thread From: Herbert Xu @ 2015-07-20 9:55 UTC (permalink / raw) To: Wolfgang Walter; +Cc: netdev, Tom Herbert, Eric Dumazet On Mon, Jul 20, 2015 at 11:39:52AM +0200, Wolfgang Walter wrote: > > > Can you confirm that disabling tso on eth1 does not help? > > Disabling TSO on eth1 does not help. OK, in that case I think the safest option is to just disable SIT GRO. If folks need this then they should make sure that it works properly, i.e., the packets generated by GRO must be in a form that GSO can grok. Also it should not cause hardware TX to lock up (though this may yet turn out to be a hardware bug). ---8<--- This patch reverts 19424e052fb44da2f00d1a868cbb51f3e9f4bbb5 ("sit: Add gro callbacks to sit_offload") because it generates packets that cannot be handled even by our own GSO. Reported-by: Wolfgang Walter <linux@stwm.de> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index e893cd1..08b6204 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -292,8 +292,6 @@ static struct packet_offload ipv6_packet_offload __read_mostly = { static const struct net_offload sit_offload = { .callbacks = { .gso_segment = ipv6_gso_segment, - .gro_receive = ipv6_gro_receive, - .gro_complete = ipv6_gro_complete, }, }; -- Email: Herbert Xu <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Revert "sit: Add gro callbacks to sit_offload" 2015-07-20 9:55 ` Revert "sit: Add gro callbacks to sit_offload" Herbert Xu @ 2015-07-20 23:53 ` David Miller 0 siblings, 0 replies; 13+ messages in thread From: David Miller @ 2015-07-20 23:53 UTC (permalink / raw) To: herbert; +Cc: linux, netdev, tom, eric.dumazet From: Herbert Xu <herbert@gondor.apana.org.au> Date: Mon, 20 Jul 2015 17:55:38 +0800 > This patch reverts 19424e052fb44da2f00d1a868cbb51f3e9f4bbb5 ("sit: > Add gro callbacks to sit_offload") because it generates packets > that cannot be handled even by our own GSO. > > Reported-by: Wolfgang Walter <linux@stwm.de> > Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au> Applied and queued up for -stable, thanks Herbert. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2015-07-20 23:53 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-07-14 22:16 GRO: forwading ISATAP packets is very slow with kernel 4.1 Wolfgang Walter 2015-07-15 0:08 ` Herbert Xu 2015-07-15 0:34 ` Wolfgang Walter 2015-07-15 9:50 ` Herbert Xu 2015-07-15 12:25 ` Wolfgang Walter 2015-07-16 0:23 ` sit: Set SKB_GSO_SIT bit when performing GRO Herbert Xu 2015-07-16 10:58 ` Wolfgang Walter 2015-07-17 1:56 ` Herbert Xu 2015-07-17 15:38 ` Wolfgang Walter 2015-07-20 6:14 ` Herbert Xu 2015-07-20 9:39 ` Wolfgang Walter 2015-07-20 9:55 ` Revert "sit: Add gro callbacks to sit_offload" Herbert Xu 2015-07-20 23:53 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).