IPsec PMTUD problem

IPsec PMTUD problem

[Patrick McHardy]

I noticed a problem with PMTUD between two IPsec tunnel endpoints.
When sending a packet larger than the PMTU with IP_DF from one
tunnel endpoint to the other, xfrm4_output sends an ICMP frag.
required with the IPsec MTU. Since the addresses match the tunnel
endpoints, this updates the MTU for the XFRM route with the value
that was calculated for the entire bundle, which in turn causes
a decrease for the bundle, resulting in further ICMP frag. required
messages until the minimum is reached.

For example with an IPsec tunnel 10.0.0.1 <-> 10.0.0.2:

uml:~# ping 10.0.0.1 -M do -s 1500 -c 20
PING 10.0.0.1 (10.0.0.1) 1500(1528) bytes of data.
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1428)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1356)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1284)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1212)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1140)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 1068)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 996)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 924)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 852)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 780)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 708)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 636)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 564)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 552)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 492)
>From 10.0.0.2 icmp_seq=1 Frag needed and DF set (mtu = 492)
...

I can't come up with a way to fix this, so I'm hoping someone else
has an idea.

Re: IPsec PMTUD problem

[Herbert Xu]

Hi Patrick:

On Mon, Apr 02, 2007 at 04:10:25PM +0200, Patrick McHardy wrote:
> I noticed a problem with PMTUD between two IPsec tunnel endpoints.
> When sending a packet larger than the PMTU with IP_DF from one
> tunnel endpoint to the other, xfrm4_output sends an ICMP frag.
> required with the IPsec MTU. Since the addresses match the tunnel
> endpoints, this updates the MTU for the XFRM route with the value
> that was calculated for the entire bundle, which in turn causes
> a decrease for the bundle, resulting in further ICMP frag. required
> messages until the minimum is reached.

I presume you're using the same pair of addresses inside and
outside the tunnel? If so the problem is that the kernel doesn't
distinguish between internal ICMP errors and external ones.
So when an MTU update occurs for the internal pair the external
pair is also affected.

We'd need some field in the routing cache to distinguish the
two pairs.

Of course the easy work-around is to use distinct addresses
within IPsec tunnels.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: IPsec PMTUD problem

[Patrick McHardy]

Herbert Xu wrote:
> On Mon, Apr 02, 2007 at 04:10:25PM +0200, Patrick McHardy wrote:
> 
>>I noticed a problem with PMTUD between two IPsec tunnel endpoints.
>>When sending a packet larger than the PMTU with IP_DF from one
>>tunnel endpoint to the other, xfrm4_output sends an ICMP frag.
>>required with the IPsec MTU. Since the addresses match the tunnel
>>endpoints, this updates the MTU for the XFRM route with the value
>>that was calculated for the entire bundle, which in turn causes
>>a decrease for the bundle, resulting in further ICMP frag. required
>>messages until the minimum is reached.
> 
> 
> I presume you're using the same pair of addresses inside and
> outside the tunnel? If so the problem is that the kernel doesn't
> distinguish between internal ICMP errors and external ones.
> So when an MTU update occurs for the internal pair the external
> pair is also affected.

Exactly.

> We'd need some field in the routing cache to distinguish the
> two pairs.

I'm not sure I understand how this would work, the ICMP message
looks the same in both cases. Or are you suggesting to
differentiate based on the source of the ICMP message?

> Of course the easy work-around is to use distinct addresses
> within IPsec tunnels.

Yes, that would work as a workaround, but it still seems like
something worth fixing.

Re: IPsec PMTUD problem

[Herbert Xu]

On Tue, Apr 03, 2007 at 06:32:07PM +0200, Patrick McHardy wrote:
> 
> I'm not sure I understand how this would work, the ICMP message
> looks the same in both cases. Or are you suggesting to
> differentiate based on the source of the ICMP message?

Actually you're right, this can't work in the general case.  Even
if we had real devices for IPsec tunnels, there is still no way to
reliably figure out which device we should attribute a given MTU
event to if the same address appears on more than one device.

> Yes, that would work as a workaround, but it still seems like
> something worth fixing.

One possible solution is to not send MTU errors to ourselves since
we it wouldn't give us any new information.  We'd need to audit the
users of icmp_send to make sure that there isn't a legitimate case
where we'd want to do that.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: IPsec PMTUD problem

[Patrick McHardy]

Herbert Xu wrote:
> On Tue, Apr 03, 2007 at 06:32:07PM +0200, Patrick McHardy wrote:
> 
>>I'm not sure I understand how this would work, the ICMP message
>>looks the same in both cases. Or are you suggesting to
>>differentiate based on the source of the ICMP message?
> 
> 
> Actually you're right, this can't work in the general case.  Even
> if we had real devices for IPsec tunnels, there is still no way to
> reliably figure out which device we should attribute a given MTU
> event to if the same address appears on more than one device.
> 
> 
>>Yes, that would work as a workaround, but it still seems like
>>something worth fixing.
> 
> 
> One possible solution is to not send MTU errors to ourselves since
> we it wouldn't give us any new information.  We'd need to audit the
> users of icmp_send to make sure that there isn't a legitimate case
> where we'd want to do that.


One such case is delivery of errors to sockets. We'd need to make
sure the errors are delivered some other way.

Re: IPsec PMTUD problem

[Herbert Xu]

On Thu, Apr 05, 2007 at 02:09:20PM +0200, Patrick McHardy wrote:
>
> > One possible solution is to not send MTU errors to ourselves since
> > we it wouldn't give us any new information.  We'd need to audit the
> > users of icmp_send to make sure that there isn't a legitimate case
> > where we'd want to do that.
> 
> One such case is delivery of errors to sockets. We'd need to make
> sure the errors are delivered some other way.

Alternatively we can still send the ICMP error but avoid a PMTU
update if we received it from ourselves.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: IPsec PMTUD problem

[Patrick McHardy]

Herbert Xu wrote:
> On Thu, Apr 05, 2007 at 02:09:20PM +0200, Patrick McHardy wrote:
> 
>>>One possible solution is to not send MTU errors to ourselves since
>>>we it wouldn't give us any new information.  We'd need to audit the
>>>users of icmp_send to make sure that there isn't a legitimate case
>>>where we'd want to do that.
>>
>>One such case is delivery of errors to sockets. We'd need to make
>>sure the errors are delivered some other way.
> 
> 
> Alternatively we can still send the ICMP error but avoid a PMTU
> update if we received it from ourselves.


That sounds easier. I'm currently working in that area anyway, I'll
give it a try.

Re: IPsec PMTUD problem

[Herbert Xu]

On Thu, Apr 05, 2007 at 02:16:53PM +0200, Patrick McHardy wrote:
> 
> That sounds easier. I'm currently working in that area anyway, I'll
> give it a try.

Thanks Patrick!
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt