GRO issue with kernel 3.4.94 (icmp fragmentation needed)

GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[jungwon park]

When using the linux router is turned on GRO, router send the 'fragmentation
needed' packets to the sender.
and the sender's packets are dropped.
The router seems to handle the larger than MTU(1500) with TCP segments by
GRO.

topology : [Host A] - [Linux Router] - [Host B] kernel : 3.4.94 GRO : on MTU
: 1500

tcpdmp :

10:36:36.592752 IP (tos 0x0, ttl 127, id 15685, offset 0, flags [DF], proto
TCP (6), length 198) x.x.x.x.49512 > x.x.x.x.445: P 908:1066(158) ack 587
win 254
10:36:36.592911 IP (tos 0x0, ttl 128, id 9712, offset 0, flags [DF], proto
TCP (6), length 117) x.x.x.x.445 > x.x.x.x.49512: P 587:664(77) ack 1066 win
252
10:36:36.594552 IP (tos 0x0, ttl 127, id 15686, offset 0, flags [DF], proto
TCP (6), length 390) x.x.x.x.49512 > x.x.x.x.445: P 1066:1416(350) ack 664
win 254
10:36:36.596544 IP (tos 0x0, ttl 128, id 9713, offset 0, flags [DF], proto
TCP (6), length 2960) x.x.x.x.445 > x.x.x.x.49512: . 664:3584(2920) ack 1416
win 251
10:36:36.596566 IP (tos 0xc0, ttl 64, id 8339, offset 0, flags [none], proto
ICMP (1), length 576) x.x.x.x > x.x.x.x: ICMP x.x.x.x unreachable - need to
frag (mtu 1500), length 556
	IP (tos 0x0, ttl 128, id 9713, offset 0, flags [DF], proto TCP (6),
length 2960) x.x.x.x.445 > x.x.x.x.49512:  tcp 2940 [bad hdr length 0 - too
short, < 20][|icmp]

When I turned off GRO, the router operate normally, and there is no problem.
and with 3.4.91 kernel, the router has no problem.

I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
(http://patchwork.ozlabs.org/patch/345509/)
When I revert this patch, the router has no problem.

(sorry my poor english.)

Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[Florian Westphal]

jungwon park <jwpark2@whitecode.co.kr> wrote:
> When using the linux router is turned on GRO, router send the 'fragmentation
> needed' packets to the sender.

Indeed 8-(

> When I turned off GRO, the router operate normally, and there is no problem.
> and with 3.4.91 kernel, the router has no problem.
> 
> I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
> (http://patchwork.ozlabs.org/patch/345509/)
> When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <edumazet@google.com>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of
3.4.y tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine
the size of the individual packets.  To do this, we rely on the
transport header.

Unfortunately the transport header is not set for the forward path in 3.4,
so we look at the network header instead.

RE: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[jungwon park]

Good!
When I apply the patch(ipv4: set transport header earlier), there is no
problem.
Thanks, Florian.

-----Original Message-----
From: Florian Westphal [mailto:fw@strlen.de] 
Sent: Friday, June 27, 2014 6:05 PM
To: jungwon park
Cc: netdev@vger.kernel.org
Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

jungwon park <jwpark2@whitecode.co.kr> wrote:
> When using the linux router is turned on GRO, router send the 
> 'fragmentation needed' packets to the sender.

Indeed 8-(

> When I turned off GRO, the router operate normally, and there is no
problem.
> and with 3.4.91 kernel, the router has no problem.
> 
> I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
> (http://patchwork.ozlabs.org/patch/345509/)
> When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=2
1d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <edumazet@google.com>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of 3.4.y
tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine the
size of the individual packets.  To do this, we rely on the transport
header.

Unfortunately the transport header is not set for the forward path in 3.4,
so we look at the network header instead.

Fwd: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[Thomas Jarosch]

Hi Greg,

I don't have the original patch email, but please include
the mentioned commit

------------------------------------------
commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <edumazet@google.com>
ipv4: set transport header earlier
------------------------------------------

in the 3.4.x stable tree.

After updating from 3.4.91 to 3.4.96, TCP connections
from Windows 7 clients slowed down to a crawl.
(to be precise: totally unusable)

The commit recommended by Florian Westphal fixes it immediately.

Best regards,
Thomas

----------  Forwarded Message  ----------

Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
Date: Friday, 27. June 2014, 11:05:25
From: Florian Westphal <fw@strlen.de>
To: jungwon park <jwpark2@whitecode.co.kr>
CC: netdev@vger.kernel.org

jungwon park <jwpark2@whitecode.co.kr> wrote:
> When using the linux router is turned on GRO, router send the 
'fragmentation
> needed' packets to the sender.

Indeed 8-(

> When I turned off GRO, the router operate normally, and there is no 
problem.
> and with 3.4.91 kernel, the router has no problem.
> 
> I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
> (http://patchwork.ozlabs.org/patch/345509/)
> When I revert this patch, the router has no problem.

Can you please cherry-pick following patch on top of vanilla 3.4.92?
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3

commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
Author: Eric Dumazet <edumazet@google.com>
ipv4: set transport header earlier

I think that should fix this bug, it should apply cleanly on top of
3.4.y tree.

[ patch is in 3.11, also backported to 3.10.y tree ]

The problem is that, when dealing with GRO packets, we try to determine
the size of the individual packets.  To do this, we rely on the
transport header.

Unfortunately the transport header is not set for the forward path in 3.4,
so we look at the network header instead.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
-----------------------------------------

Re: Fwd: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[Greg KH]

On Fri, Jul 11, 2014 at 05:12:55PM +0200, Thomas Jarosch wrote:
> Hi Greg,
> 
> I don't have the original patch email, but please include
> the mentioned commit
> 
> ------------------------------------------
> commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
> Author: Eric Dumazet <edumazet@google.com>
> ipv4: set transport header earlier
> ------------------------------------------
> 
> in the 3.4.x stable tree.
> 
> After updating from 3.4.91 to 3.4.96, TCP connections
> from Windows 7 clients slowed down to a crawl.
> (to be precise: totally unusable)
> 
> The commit recommended by Florian Westphal fixes it immediately.
> 
> Best regards,
> Thomas
> 
> ----------  Forwarded Message  ----------
> 
> Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
> Date: Friday, 27. June 2014, 11:05:25
> From: Florian Westphal <fw@strlen.de>
> To: jungwon park <jwpark2@whitecode.co.kr>
> CC: netdev@vger.kernel.org
> 
> jungwon park <jwpark2@whitecode.co.kr> wrote:
> > When using the linux router is turned on GRO, router send the 
> 'fragmentation
> > needed' packets to the sender.
> 
> Indeed 8-(
> 
> > When I turned off GRO, the router operate normally, and there is no 
> problem.
> > and with 3.4.91 kernel, the router has no problem.
> > 
> > I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
> > (http://patchwork.ozlabs.org/patch/345509/)
> > When I revert this patch, the router has no problem.
> 
> Can you please cherry-pick following patch on top of vanilla 3.4.92?
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
> 
> commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
> Author: Eric Dumazet <edumazet@google.com>
> ipv4: set transport header earlier
> 
> I think that should fix this bug, it should apply cleanly on top of
> 3.4.y tree.
> 
> [ patch is in 3.11, also backported to 3.10.y tree ]
> 
> The problem is that, when dealing with GRO packets, we try to determine
> the size of the individual packets.  To do this, we rely on the
> transport header.
> 
> Unfortunately the transport header is not set for the forward path in 3.4,
> so we look at the network header instead.

Why would this patch matter, as commit 45f00f99d6e, which this patch
fixes, is not in the 3.4-stable tree at all.

Also, network stable patches need to go through the network maintainer,
not me directly.

thanks,

greg k-h

Re: Fwd: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[Florian Westphal]

Greg KH <greg@kroah.com> wrote:

[ CCing David.  If I understood him correctly, he recommended to revert
  the GRO forwarding changes for oldstable 3.2/3.4.y series. I second
  this.  But see below for answer to your question ]

> On Fri, Jul 11, 2014 at 05:12:55PM +0200, Thomas Jarosch wrote:
> > Subject: Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)
> > Date: Friday, 27. June 2014, 11:05:25
> > From: Florian Westphal <fw@strlen.de>
> > To: jungwon park <jwpark2@whitecode.co.kr>
> > CC: netdev@vger.kernel.org
> > 
> > jungwon park <jwpark2@whitecode.co.kr> wrote:
> > > When using the linux router is turned on GRO, router send the 
> > 'fragmentation
> > > needed' packets to the sender.
> > 
> > Indeed 8-(
> > 
> > > When I turned off GRO, the router operate normally, and there is no 
> > problem.
> > > and with 3.4.91 kernel, the router has no problem.
> > > 
> > > I doubt 'ipv4: ip_forward: fix inverted local_df test' patch.
> > > (http://patchwork.ozlabs.org/patch/345509/)
> > > When I revert this patch, the router has no problem.
> > 
> > Can you please cherry-pick following patch on top of vanilla 3.4.92?
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
> > 
> > commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
> > Author: Eric Dumazet <edumazet@google.com>
> > ipv4: set transport header earlier
> > 
> > I think that should fix this bug, it should apply cleanly on top of
> > 3.4.y tree.
> > 
> > [ patch is in 3.11, also backported to 3.10.y tree ]
> > 
> > The problem is that, when dealing with GRO packets, we try to determine
> > the size of the individual packets.  To do this, we rely on the
> > transport header.
> > 
> > Unfortunately the transport header is not set for the forward path in 3.4,
> > so we look at the network header instead.
> 
> Why would this patch matter, as commit 45f00f99d6e, which this patch
> fixes, is not in the 3.4-stable tree at all.

It helps, because I underestimated the changes the network stacki went through
since 3.2 and 3.4 had been released.

The GRO forwarding change, (fe6cc55f3a9a053482a76f5a6b2257cee51b4663
upstream) depends on having the transport header available in the
forward path, and not just in the input/local delivery path.

The Patch from Eric Dumazet that I referenced above made this change.
So, while its unrelated to the GRO forward patch, it is a dependency.

However, given the numerous problems reported with the GRO changes in
older stable kernels I think the commit
29a3cd46644ec8098dbe1c12f89643b5c11831a9 in 3.4.y should be reverted.

[ In 3.2.y its caa5344994778a2b4725b2d75c74430f76925e4a ]

> Also, network stable patches need to go through the network maintainer,
> not me directly.

Greg, is above explanation enough for you to revert?
If not, what can I do to help?

David, in case I remember wrong and you did not recommend the revert
after all, do you want me to submit 3.2/3.4.y backport of Erics
'set transport header earlier' patch?

Thanks,
Florian

Re: GRO issue with kernel 3.4.94 (icmp fragmentation needed)

[David Miller]

From: Florian Westphal <fw@strlen.de>
Date: Wed, 30 Jul 2014 08:37:03 +0200

> However, given the numerous problems reported with the GRO changes in
> older stable kernels I think the commit
> 29a3cd46644ec8098dbe1c12f89643b5c11831a9 in 3.4.y should be reverted.
> 
> [ In 3.2.y its caa5344994778a2b4725b2d75c74430f76925e4a ]
 ...
> David, in case I remember wrong and you did not recommend the revert
> after all, do you want me to submit 3.2/3.4.y backport of Erics
> 'set transport header earlier' patch?

Yes, the above two patches above should be reverted.

Thanks Florian.