[PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Alexander Duyck]

This change makes it so that we use icmpv6_send to report PMTU issues back
into tunnels in the case that the resulting packet is larger than the MTU
of the outgoing interface.  Previously xfrm_local_error was being used in
this case, however this was resulting in no changes, I suspect due to the
fact that the tunnel itself was being kept out of the loop.

This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
behavior seen if the socket was orphaned.  Instead of requiring the socket
to be orphaned this patch simply defaults to using icmpv6_send in the case
that the frame came though a tunnel.

Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>
---
 net/ipv6/xfrm6_output.c |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/net/ipv6/xfrm6_output.c b/net/ipv6/xfrm6_output.c
index 09c76a7b474d..6f9b514d0e38 100644
--- a/net/ipv6/xfrm6_output.c
+++ b/net/ipv6/xfrm6_output.c
@@ -72,6 +72,7 @@ static int xfrm6_tunnel_check_size(struct sk_buff *skb)
 {
 	int mtu, ret = 0;
 	struct dst_entry *dst = skb_dst(skb);
+	struct xfrm_state *x = dst->xfrm;
 
 	mtu = dst_mtu(dst);
 	if (mtu < IPV6_MIN_MTU)
@@ -82,7 +83,7 @@ static int xfrm6_tunnel_check_size(struct sk_buff *skb)
 
 		if (xfrm6_local_dontfrag(skb))
 			xfrm6_local_rxpmtu(skb, mtu);
-		else if (skb->sk)
+		else if (skb->sk && x->props.mode != XFRM_MODE_TUNNEL)
 			xfrm_local_error(skb, mtu);
 		else
 			icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
@@ -149,11 +150,16 @@ static int __xfrm6_output(struct sock *sk, struct sk_buff *skb)
 	else
 		mtu = dst_mtu(skb_dst(skb));
 
-	if (skb->len > mtu && xfrm6_local_dontfrag(skb)) {
-		xfrm6_local_rxpmtu(skb, mtu);
-		return -EMSGSIZE;
-	} else if (!skb->ignore_df && skb->len > mtu && skb->sk) {
-		xfrm_local_error(skb, mtu);
+	if (!skb->ignore_df && skb->len > mtu) {
+		skb->dev = dst->dev;
+
+		if (xfrm6_local_dontfrag(skb))
+			xfrm6_local_rxpmtu(skb, mtu);
+		else if (skb->sk && x->props.mode != XFRM_MODE_TUNNEL)
+			xfrm_local_error(skb, mtu);
+		else
+			icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
+
 		return -EMSGSIZE;
 	}
 

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Herbert Xu]

On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
> This change makes it so that we use icmpv6_send to report PMTU issues back
> into tunnels in the case that the resulting packet is larger than the MTU
> of the outgoing interface.  Previously xfrm_local_error was being used in
> this case, however this was resulting in no changes, I suspect due to the
> fact that the tunnel itself was being kept out of the loop.
> 
> This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
> behavior seen if the socket was orphaned.  Instead of requiring the socket
> to be orphaned this patch simply defaults to using icmpv6_send in the case
> that the frame came though a tunnel.
> 
> Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>

Does this still work with normal tunnel mode and identical inner
and outer addresses? I recall we used to have a bug where in that
situation the kernel would interpret the ICMP message as a reduction
in outer MTU and thus resulting in a loop where the MTU keeps
getting smaller.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Steffen Klassert]

On Thu, May 28, 2015 at 12:49:19PM +0800, Herbert Xu wrote:
> On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
> > This change makes it so that we use icmpv6_send to report PMTU issues back
> > into tunnels in the case that the resulting packet is larger than the MTU
> > of the outgoing interface.  Previously xfrm_local_error was being used in
> > this case, however this was resulting in no changes, I suspect due to the
> > fact that the tunnel itself was being kept out of the loop.
> > 
> > This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
> > behavior seen if the socket was orphaned.  Instead of requiring the socket
> > to be orphaned this patch simply defaults to using icmpv6_send in the case
> > that the frame came though a tunnel.
> > 
> > Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>
> 
> Does this still work with normal tunnel mode and identical inner
> and outer addresses? I recall we used to have a bug where in that
> situation the kernel would interpret the ICMP message as a reduction
> in outer MTU and thus resulting in a loop where the MTU keeps
> getting smaller.

Right, I think this reintroduces a bug that I fixed some years ago with
commit dd767856a36e ("xfrm6: Don't call icmpv6_send on local error")

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Steffen Klassert]

On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
> This change makes it so that we use icmpv6_send to report PMTU issues back
> into tunnels in the case that the resulting packet is larger than the MTU
> of the outgoing interface.  Previously xfrm_local_error was being used in
> this case, however this was resulting in no changes, I suspect due to the
> fact that the tunnel itself was being kept out of the loop.
> 
> This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
> behavior seen if the socket was orphaned.  Instead of requiring the socket
> to be orphaned this patch simply defaults to using icmpv6_send in the case
> that the frame came though a tunnel.

We can use icmpv6_send() just in the case that the packet
was already transmitted by a tunnel device, otherwise we
get the bug back that I mentioned in my other mail.

Not sure if we have something to know that the packet
traversed a tunnel device. That's what I asked in the
thread 'Looking for a lost patch'.

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Alexander Duyck]

On 05/27/2015 10:36 PM, Steffen Klassert wrote:
> On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
>> This change makes it so that we use icmpv6_send to report PMTU issues back
>> into tunnels in the case that the resulting packet is larger than the MTU
>> of the outgoing interface.  Previously xfrm_local_error was being used in
>> this case, however this was resulting in no changes, I suspect due to the
>> fact that the tunnel itself was being kept out of the loop.
>>
>> This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
>> behavior seen if the socket was orphaned.  Instead of requiring the socket
>> to be orphaned this patch simply defaults to using icmpv6_send in the case
>> that the frame came though a tunnel.
> We can use icmpv6_send() just in the case that the packet
> was already transmitted by a tunnel device, otherwise we
> get the bug back that I mentioned in my other mail.
>
> Not sure if we have something to know that the packet
> traversed a tunnel device. That's what I asked in the
> thread 'Looking for a lost patch'.

Okay I will try to do some more digging.  From what I can tell right now 
it looks like my ping attempts are getting hung up on the 
xfrm_local_error in __xfrm6_output.  I wonder if we couldn't somehow 
make use of the skb->cb to store a pointer to the tunnel that could be 
checked to determine if we are going through a VTI or not.

- Alex

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Steffen Klassert]

On Thu, May 28, 2015 at 12:18:51AM -0700, Alexander Duyck wrote:
> On 05/27/2015 10:36 PM, Steffen Klassert wrote:
> >On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
> >>This change makes it so that we use icmpv6_send to report PMTU issues back
> >>into tunnels in the case that the resulting packet is larger than the MTU
> >>of the outgoing interface.  Previously xfrm_local_error was being used in
> >>this case, however this was resulting in no changes, I suspect due to the
> >>fact that the tunnel itself was being kept out of the loop.
> >>
> >>This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
> >>behavior seen if the socket was orphaned.  Instead of requiring the socket
> >>to be orphaned this patch simply defaults to using icmpv6_send in the case
> >>that the frame came though a tunnel.
> >We can use icmpv6_send() just in the case that the packet
> >was already transmitted by a tunnel device, otherwise we
> >get the bug back that I mentioned in my other mail.
> >
> >Not sure if we have something to know that the packet
> >traversed a tunnel device. That's what I asked in the
> >thread 'Looking for a lost patch'.
> 
> Okay I will try to do some more digging.  From what I can tell right
> now it looks like my ping attempts are getting hung up on the
> xfrm_local_error in __xfrm6_output.  I wonder if we couldn't somehow
> make use of the skb->cb to store a pointer to the tunnel that could
> be checked to determine if we are going through a VTI or not.

Maybe it is as easy as the patch below, could you please test it?

Subject: [PATCH RFC] vti6: Add pmtu handling to vti6_xmit.

We currently rely on the PMTU discovery of xfrm.
However if a packet is localy sent, the PMTU mechanism
of xfrm tries to to local socket notification what
might not work for applications like ping that don't
check for this. So add pmtu handling to vti6_xmit to
report MTU changes immediately.

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
---
 net/ipv6/ip6_vti.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index ff3bd86..13cb771 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -434,6 +434,7 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
 	struct dst_entry *dst = skb_dst(skb);
 	struct net_device *tdev;
 	struct xfrm_state *x;
+	int mtu;
 	int err = -1;
 
 	if (!dst)
@@ -468,6 +469,15 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
 	skb_dst_set(skb, dst);
 	skb->dev = skb_dst(skb)->dev;
 
+	mtu = dst_mtu(dst);
+	if (!skb->ignore_df && skb->len > mtu) {
+		skb_dst(skb)->ops->update_pmtu(dst, NULL, skb, mtu);
+
+		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
+
+		return -EMSGSIZE;
+	}
+
 	err = dst_output(skb);
 	if (net_xmit_eval(err) == 0) {
 		struct pcpu_sw_netstats *tstats = this_cpu_ptr(dev->tstats);
-- 
1.9.1

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Alexander Duyck]

On 05/28/2015 01:40 AM, Steffen Klassert wrote:
> On Thu, May 28, 2015 at 12:18:51AM -0700, Alexander Duyck wrote:
>> On 05/27/2015 10:36 PM, Steffen Klassert wrote:
>>> On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
>>>> This change makes it so that we use icmpv6_send to report PMTU issues back
>>>> into tunnels in the case that the resulting packet is larger than the MTU
>>>> of the outgoing interface.  Previously xfrm_local_error was being used in
>>>> this case, however this was resulting in no changes, I suspect due to the
>>>> fact that the tunnel itself was being kept out of the loop.
>>>>
>>>> This patch fixes PMTU problems seen on ip6_vti tunnels and is based on the
>>>> behavior seen if the socket was orphaned.  Instead of requiring the socket
>>>> to be orphaned this patch simply defaults to using icmpv6_send in the case
>>>> that the frame came though a tunnel.
>>> We can use icmpv6_send() just in the case that the packet
>>> was already transmitted by a tunnel device, otherwise we
>>> get the bug back that I mentioned in my other mail.
>>>
>>> Not sure if we have something to know that the packet
>>> traversed a tunnel device. That's what I asked in the
>>> thread 'Looking for a lost patch'.
>> Okay I will try to do some more digging.  From what I can tell right
>> now it looks like my ping attempts are getting hung up on the
>> xfrm_local_error in __xfrm6_output.  I wonder if we couldn't somehow
>> make use of the skb->cb to store a pointer to the tunnel that could
>> be checked to determine if we are going through a VTI or not.
> Maybe it is as easy as the patch below, could you please test it?
>
> Subject: [PATCH RFC] vti6: Add pmtu handling to vti6_xmit.
>
> We currently rely on the PMTU discovery of xfrm.
> However if a packet is localy sent, the PMTU mechanism
> of xfrm tries to to local socket notification what
> might not work for applications like ping that don't
> check for this. So add pmtu handling to vti6_xmit to
> report MTU changes immediately.
>
> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
> ---
>   net/ipv6/ip6_vti.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
>
> diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
> index ff3bd86..13cb771 100644
> --- a/net/ipv6/ip6_vti.c
> +++ b/net/ipv6/ip6_vti.c
> @@ -434,6 +434,7 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
>   	struct dst_entry *dst = skb_dst(skb);
>   	struct net_device *tdev;
>   	struct xfrm_state *x;
> +	int mtu;
>   	int err = -1;
>   
>   	if (!dst)
> @@ -468,6 +469,15 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
>   	skb_dst_set(skb, dst);
>   	skb->dev = skb_dst(skb)->dev;
>   
> +	mtu = dst_mtu(dst);
> +	if (!skb->ignore_df && skb->len > mtu) {
> +		skb_dst(skb)->ops->update_pmtu(dst, NULL, skb, mtu);
> +
> +		icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
> +
> +		return -EMSGSIZE;
> +	}
> +
>   	err = dst_output(skb);
>   	if (net_xmit_eval(err) == 0) {
>   		struct pcpu_sw_netstats *tstats = this_cpu_ptr(dev->tstats);

That seems to be working for me.  I'm able to ping and while the first 
packet fails the second one and all that follow make it through 
correctly after the ptmu update.

- Alex

Re: [PATCH] xfrm6: Do not use xfrm_local_error for path MTU issues in tunnels

[Alexander Duyck]

On 05/28/2015 12:15 PM, Alexander Duyck wrote:
> On 05/28/2015 01:40 AM, Steffen Klassert wrote:
>> On Thu, May 28, 2015 at 12:18:51AM -0700, Alexander Duyck wrote:
>>> On 05/27/2015 10:36 PM, Steffen Klassert wrote:
>>>> On Wed, May 27, 2015 at 10:40:32AM -0700, Alexander Duyck wrote:
>>>>> This change makes it so that we use icmpv6_send to report PMTU 
>>>>> issues back
>>>>> into tunnels in the case that the resulting packet is larger than 
>>>>> the MTU
>>>>> of the outgoing interface.  Previously xfrm_local_error was being 
>>>>> used in
>>>>> this case, however this was resulting in no changes, I suspect due 
>>>>> to the
>>>>> fact that the tunnel itself was being kept out of the loop.
>>>>>
>>>>> This patch fixes PMTU problems seen on ip6_vti tunnels and is 
>>>>> based on the
>>>>> behavior seen if the socket was orphaned.  Instead of requiring 
>>>>> the socket
>>>>> to be orphaned this patch simply defaults to using icmpv6_send in 
>>>>> the case
>>>>> that the frame came though a tunnel.
>>>> We can use icmpv6_send() just in the case that the packet
>>>> was already transmitted by a tunnel device, otherwise we
>>>> get the bug back that I mentioned in my other mail.
>>>>
>>>> Not sure if we have something to know that the packet
>>>> traversed a tunnel device. That's what I asked in the
>>>> thread 'Looking for a lost patch'.
>>> Okay I will try to do some more digging.  From what I can tell right
>>> now it looks like my ping attempts are getting hung up on the
>>> xfrm_local_error in __xfrm6_output.  I wonder if we couldn't somehow
>>> make use of the skb->cb to store a pointer to the tunnel that could
>>> be checked to determine if we are going through a VTI or not.
>> Maybe it is as easy as the patch below, could you please test it?
>>
>> Subject: [PATCH RFC] vti6: Add pmtu handling to vti6_xmit.
>>
>> We currently rely on the PMTU discovery of xfrm.
>> However if a packet is localy sent, the PMTU mechanism
>> of xfrm tries to to local socket notification what
>> might not work for applications like ping that don't
>> check for this. So add pmtu handling to vti6_xmit to
>> report MTU changes immediately.
>>
>> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
>> ---
>>   net/ipv6/ip6_vti.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
>> index ff3bd86..13cb771 100644
>> --- a/net/ipv6/ip6_vti.c
>> +++ b/net/ipv6/ip6_vti.c
>> @@ -434,6 +434,7 @@ vti6_xmit(struct sk_buff *skb, struct net_device 
>> *dev, struct flowi *fl)
>>       struct dst_entry *dst = skb_dst(skb);
>>       struct net_device *tdev;
>>       struct xfrm_state *x;
>> +    int mtu;
>>       int err = -1;
>>         if (!dst)
>> @@ -468,6 +469,15 @@ vti6_xmit(struct sk_buff *skb, struct net_device 
>> *dev, struct flowi *fl)
>>       skb_dst_set(skb, dst);
>>       skb->dev = skb_dst(skb)->dev;
>>   +    mtu = dst_mtu(dst);
>> +    if (!skb->ignore_df && skb->len > mtu) {
>> +        skb_dst(skb)->ops->update_pmtu(dst, NULL, skb, mtu);
>> +
>> +        icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
>> +
>> +        return -EMSGSIZE;
>> +    }
>> +
>>       err = dst_output(skb);
>>       if (net_xmit_eval(err) == 0) {
>>           struct pcpu_sw_netstats *tstats = this_cpu_ptr(dev->tstats);
>
> That seems to be working for me.  I'm able to ping and while the first 
> packet fails the second one and all that follow make it through 
> correctly after the ptmu update.
>
> - Alex

It looks like I spoke too soon.  It resolves it for IPv6, but IPv4 over 
the tunnel has the same issue.  Probably need to have some sort of 
protocol based check to determine which version of the call to use.

- Alex

[PATCH] vti6: Add pmtu handling to vti6_xmit.

[Alexander Duyck]

From: Steffen Klassert <steffen.klassert@secunet.com>

We currently rely on the PMTU discovery of xfrm.
However if a packet is localy sent, the PMTU mechanism
of xfrm tries to to local socket notification what
might not work for applications like ping that don't
check for this. So add pmtu handling to vti6_xmit to
report MTU changes immediately.

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>
---

So this version is slightly modified to cover the IPv4 case in addition to
the IPv6 case.  With this patch I was able to run netperf over either an
IPv4 or IPv6 address routed over the ip6_vti tunnel.

 net/ipv6/ip6_vti.c |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index d25209657edc..3b5c1ea50d2f 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -435,6 +435,7 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
 	struct net_device *tdev;
 	struct xfrm_state *x;
 	int err = -1;
+	int mtu;
 
 	if (!dst)
 		goto tx_err_link_failure;
@@ -468,6 +469,19 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
 	skb_dst_set(skb, dst);
 	skb->dev = skb_dst(skb)->dev;
 
+	mtu = dst_mtu(dst);
+	if (!skb->ignore_df && skb->len > mtu) {
+		skb_dst(skb)->ops->update_pmtu(dst, NULL, skb, mtu);
+
+		if (skb->protocol == htons(ETH_P_IPV6))
+			icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
+		else
+			icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED,
+				  htonl(mtu));
+
+		return -EMSGSIZE;
+	}
+
 	err = dst_output(skb);
 	if (net_xmit_eval(err) == 0) {
 		struct pcpu_sw_netstats *tstats = this_cpu_ptr(dev->tstats);

Re: [PATCH] vti6: Add pmtu handling to vti6_xmit.

[David Miller]

From: Alexander Duyck <alexander.h.duyck@redhat.com>
Date: Fri, 29 May 2015 11:28:26 -0700

> From: Steffen Klassert <steffen.klassert@secunet.com>
> 
> We currently rely on the PMTU discovery of xfrm.
> However if a packet is localy sent, the PMTU mechanism
> of xfrm tries to to local socket notification what
> might not work for applications like ping that don't
> check for this. So add pmtu handling to vti6_xmit to
> report MTU changes immediately.
> 
> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
> Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com>

Applied, thanks Andrew.