IPSec freeze

IPSec freeze

[Beschorner Daniel]

Today a new site joined our Linux IPSec VPN, now all the other routers
(all 2.6.22) freeze hard reproducible.
No oops, no sysreq, only hard reset rewakes them.

The only difference of the new site compared to the others: ADSL, thus a
MTU of 1492, the others have 1500.
Disabling IPSec und doing normal operations between the routers is fine,
PMTU is honored correctly.
If I set the MTU of the other routers to 1492 I can avoid the IPSec
crash.

Some kind of strange need-to-frag-ICMP that causes such things?
Any ideas how to debug this?

Thanks!
Daniel

Here a tcpdump of a router (1.1.1.1, obfuscated) just before it died:

07:58:23.588064 IP (tos 0x0, ttl  64, id 8192, offset 0, flags [DF],
proto: ESP (50), length: 1496) 1.1.1.1 > 2.2.2.2:
ESP(spi=0xae81babb,seq=0x15), length 1476
07:58:23.590414 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 152) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0xf), length 132
07:58:23.592928 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 104) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0x10), length 84
07:58:23.593246 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto:
ESP (50), length: 1496) 1.1.1.1 > 2.2.2.2: ESP(spi=0xae81babb,seq=0x16),
length 1476
07:58:23.596486 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 152) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0x11), length 132
07:58:23.596806 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto:
ESP (50), length: 1496) 1.1.1.1 > 2.2.2.2: ESP(spi=0xae81babb,seq=0x17),
length 1476
07:58:23.596859 IP (tos 0x0, ttl  64, id 10655, offset 0, flags [DF],
proto: ESP (50), length: 200) 1.1.1.1 > 2.2.2.2:
ESP(spi=0xae81babb,seq=0x18), length 180
07:58:23.726550 IP (tos 0x0, ttl  50, id 8192, offset 0, flags [none],
proto: ICMP (1), length: 56) 67.38.70.235 > 1.1.1.1: ICMP 2.2.2.2
unreachable - need to frag (mtu 1492), length 36
        IP (tos 0x0, ttl  45, id 8192, offset 0, flags [DF], proto: ESP
(50), length: 1496) 1.1.1.1 > 2.2.2.2: [|ESP]
07:58:23.731648 IP (tos 0x0, ttl  50, id 0, offset 0, flags [none],
proto: ICMP (1), length: 56) 67.38.70.235 > 1.1.1.1: ICMP 2.2.2.2
unreachable - need to frag (mtu 1492), length 36
        IP (tos 0x0, ttl  45, id 0, offset 0, flags [DF], proto: ESP
(50), length: 1496) 1.1.1.1 > 2.2.2.2: [|ESP]
07:58:23.734776 IP (tos 0x0, ttl  50, id 0, offset 0, flags [none],
proto: ICMP (1), length: 56) 67.38.70.235 > 1.1.1.1: ICMP 2.2.2.2
unreachable - need to frag (mtu 1492), length 36
        IP (tos 0x0, ttl  45, id 0, offset 0, flags [DF], proto: ESP
(50), length: 1496) 1.1.1.1 > 2.2.2.2: [|ESP]
07:58:23.740504 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 104) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0x12), length 84
07:58:23.743108 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 104) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0x13), length 84
07:58:23.754123 IP (tos 0x0, ttl  48, id 22785, offset 0, flags [DF],
proto: ESP (50), length: 104) 2.2.2.2 > 1.1.1.1:
ESP(spi=0x593f3550,seq=0x14), length 84

Here a log of another death from inside the tunnel (last packet is again
the time of crash):
The Tunnel MTU of 1430 is correct for an outer MTU of 1500, but the
additional -8 doesn't take place?!?

05:17:18.563448 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
05:17:18.563468 IP 192.168.200.254 > 192.168.200.1: ICMP 192.168.203.1
unreachable - need to frag (mtu 1430), length 556
05:17:18.563471 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
05:17:18.563479 IP 192.168.200.254 > 192.168.200.1: ICMP 192.168.203.1
unreachable - need to frag (mtu 1430), length 556
05:17:18.563481 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
05:17:18.563490 IP 192.168.200.254 > 192.168.200.1: ICMP 192.168.203.1
unreachable - need to frag (mtu 1430), length 556
05:17:18.563492 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
05:17:18.563499 IP 192.168.200.254 > 192.168.200.1: ICMP 192.168.203.1
unreachable - need to frag (mtu 1430), length 556
05:17:18.563616 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1390
05:17:18.882785 IP 192.168.203.1.3084 > 192.168.200.1.80: tcp 0
05:17:18.882921 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1390
05:17:18.883097 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1390
05:17:19.042207 IP 192.168.203.1.3084 > 192.168.200.1.80: tcp 0

Re: IPSec freeze

[Patrick McHardy]

Beschorner Daniel wrote:
> Today a new site joined our Linux IPSec VPN, now all the other routers
> (all 2.6.22) freeze hard reproducible.


Do the other routers all do IPsec or just one of them?

> No oops, no sysreq, only hard reset rewakes them.

> 
> The only difference of the new site compared to the others: ADSL, thus a
> MTU of 1492, the others have 1500.
> Disabling IPSec und doing normal operations between the routers is fine,
> PMTU is honored correctly.
> If I set the MTU of the other routers to 1492 I can avoid the IPSec
> crash.
> 
> Some kind of strange need-to-frag-ICMP that causes such things?
> Any ideas how to debug this?


If you can't get any information from your boxes, a testcase that can
be used to reproduce this would help.

> Here a log of another death from inside the tunnel (last packet is again
> the time of crash):
> The Tunnel MTU of 1430 is correct for an outer MTU of 1500, but the
> additional -8 doesn't take place?!?
> 
> 05:17:18.563448 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
> 05:17:18.563468 IP 192.168.200.254 > 192.168.200.1: ICMP 192.168.203.1
> unreachable - need to frag (mtu 1430), length 556


Does the router use a MTU of 1492 itself or is there another DSL
router or something like that connected by ethernet?

Re: IPSec freeze

[Beschorner Daniel]

> Beschorner Daniel wrote:
> > Today a new site joined our Linux IPSec VPN, now all the 
> other routers
> > (all 2.6.22) freeze hard reproducible.
> 
> Do the other routers all do IPsec or just one of them?

They all do IPSec, that seems to be their mistake.
The unencrypted traffic between the routers work fine, only tunnel
traffic triggers the crash.

> 
> > No oops, no sysreq, only hard reset rewakes them.
> 
> > 
> > The only difference of the new site compared to the others: 
> ADSL, thus a
> > MTU of 1492, the others have 1500.
> > Disabling IPSec und doing normal operations between the 
> routers is fine,
> > PMTU is honored correctly.
> > If I set the MTU of the other routers to 1492 I can avoid the IPSec
> > crash.
> > 
> > Some kind of strange need-to-frag-ICMP that causes such things?
> > Any ideas how to debug this?
> 
> 
> If you can't get any information from your boxes, a testcase that can
> be used to reproduce this would help.

I'll try to get more information about what is the fatal in this
configuration.
I'm surely not the first one doing VPN with different MTUs.

The only "anomaly" I've seen is that the need-to-frag complaining router
DOES fragment the oversized packet and relays it ignoring the DF flag.

Is there something I can do to debug the network stack in any way? As I
said, I can perfectly reproduce it, but I'm poor in getting information
what's going on in the kernel.

I will do some more tests, disabling Ipsec compression, etc.

> 
> > Here a log of another death from inside the tunnel (last 
> packet is again
> > the time of crash):
> > The Tunnel MTU of 1430 is correct for an outer MTU of 1500, but the
> > additional -8 doesn't take place?!?
> > 
> > 05:17:18.563448 IP 192.168.200.1.80 > 192.168.203.1.3084: tcp 1460
> > 05:17:18.563468 IP 192.168.200.254 > 192.168.200.1: ICMP 
> 192.168.203.1
> > unreachable - need to frag (mtu 1430), length 556
> 
> 
> Does the router use a MTU of 1492 itself or is there another DSL
> router or something like that connected by ethernet?

The router on the ADSL line (behind an ADSL Netopia router via Ethernet)
crashed too with 1500, now it uses 1492.

Re: IPSec freeze

[Beschorner Daniel]

> > > Today a new site joined our Linux IPSec VPN, now all the 
> > other routers
> > > (all 2.6.22) freeze hard reproducible.

The problem is more general und ugly than I thought.

I took 2 arbitrary boxes, one behind an Ethernet (A, Kernel 2.6.21, MTU
1500), one behind ADSL (B, 2.4.x, 1492).
Established a tunnel, copied a file from site A to B through the tunnel
and router A died in the same moment.

Out of my feeling this worked fine some kernel releases earlier.

As written in this thread before, I see an external need-to-frag-ICMP,
no tunnel need-to-frag will be thrown, box freezes.

You should be able to reproduce it with any network path with a smaller
MTU?!?

Daniel

Re: IPSec freeze

[Patrick McHardy]

Beschorner Daniel wrote:
>>>>Today a new site joined our Linux IPSec VPN, now all the 
>>>
>>>other routers
>>>
>>>>(all 2.6.22) freeze hard reproducible.
> 
> 
> The problem is more general und ugly than I thought.
> 
> I took 2 arbitrary boxes, one behind an Ethernet (A, Kernel 2.6.21, MTU
> 1500), one behind ADSL (B, 2.4.x, 1492).
> Established a tunnel, copied a file from site A to B through the tunnel
> and router A died in the same moment.
> 
> Out of my feeling this worked fine some kernel releases earlier.
> 
> As written in this thread before, I see an external need-to-frag-ICMP,
> no tunnel need-to-frag will be thrown, box freezes.
> 
> You should be able to reproduce it with any network path with a smaller
> MTU?!?


I'm running IPsec in the same setup as you describe above without
problems. I'm probably not seeing ICMP frag requireds on the wire
though since I believe the entire path is >= 1492.

Could you try to find out whether those are responsible?

Re: IPSec freeze

[Beschorner Daniel]

> >>>>Today a new site joined our Linux IPSec VPN, now all the 
> >>>
> >>>other routers
> >>>
> >>>>(all 2.6.22) freeze hard reproducible.
> > 
> > 
> > The problem is more general und ugly than I thought.
> > 
> > I took 2 arbitrary boxes, one behind an Ethernet (A, Kernel 
> 2.6.21, MTU
> > 1500), one behind ADSL (B, 2.4.x, 1492).
> > Established a tunnel, copied a file from site A to B 
> through the tunnel
> > and router A died in the same moment.
> > 
> > Out of my feeling this worked fine some kernel releases earlier.
> > 
> > As written in this thread before, I see an external 
> need-to-frag-ICMP,
> > no tunnel need-to-frag will be thrown, box freezes.
> > 
> > You should be able to reproduce it with any network path 
> with a smaller
> > MTU?!?
> 
> 
> I'm running IPsec in the same setup as you describe above without
> problems. I'm probably not seeing ICMP frag requireds on the wire
> though since I believe the entire path is >= 1492.
> 
> Could you try to find out whether those are responsible?
> 

It's definitely the first large packet or corresponding ICMP that
triggers the crash, slow packets don't harm.
IPSec on pathes with a PMTU of 1500 works fine.

Re: IPSec freeze

[Patrick McHardy]

Beschorner Daniel wrote:
>>I'm running IPsec in the same setup as you describe above without
>>problems. I'm probably not seeing ICMP frag requireds on the wire
>>though since I believe the entire path is >= 1492.
>>
>>Could you try to find out whether those are responsible?
>>
> 
> 
> It's definitely the first large packet or corresponding ICMP that
> triggers the crash, slow packets don't harm.
> IPSec on pathes with a PMTU of 1500 works fine.


I recreated the same setup here, but things work fine even with
different MTUs. Please try to narrow it down further or capture
some more information (serial console/netconsole,
CONFIG_DETECT_SOFTLOCKUP, ..).

Re: IPSec freeze

[Beschorner Daniel]

> I recreated the same setup here, but things work fine even with
> different MTUs. Please try to narrow it down further or capture
> some more information (serial console/netconsole,
> CONFIG_DETECT_SOFTLOCKUP, ..).

Did you turn on IPSec compression? 

Re: IPSec freeze

[Patrick McHardy]

Beschorner Daniel wrote:
>>I recreated the same setup here, but things work fine even with
>>different MTUs. Please try to narrow it down further or capture
>>some more information (serial console/netconsole,
>>CONFIG_DETECT_SOFTLOCKUP, ..).
> 
> 
> Did you turn on IPSec compression? 


No. Please send the policy you're using.

Re: IPSec freeze

[Patrick McHardy]

Patrick McHardy wrote:
> Beschorner Daniel wrote:
> 
>>>I recreated the same setup here, but things work fine even with
>>>different MTUs. Please try to narrow it down further or capture
>>>some more information (serial console/netconsole,
>>>CONFIG_DETECT_SOFTLOCKUP, ..).
>>
>>
>>Did you turn on IPSec compression? 
> 
> 
> 
> No. Please send the policy you're using.


And please try disabling ipcomp yourself to see if it makes a difference.

Re: IPSec freeze

[Patrick McHardy]

Patrick McHardy wrote:
> Beschorner Daniel wrote:
> 
>>>I recreated the same setup here, but things work fine even with
>>>different MTUs. Please try to narrow it down further or capture
>>>some more information (serial console/netconsole,
>>>CONFIG_DETECT_SOFTLOCKUP, ..).
>>
>>
>>Did you turn on IPSec compression? 
> 
> 
> 
> No. Please send the policy you're using.


I managed to reproduce a crash with ipcomp, will try to fix it later.

Re: IPSec freeze

[Beschorner Daniel]

> >>Did you turn on IPSec compression? 
> > 
> > 
> > 
> > No. Please send the policy you're using.
> 
> 
> I managed to reproduce a crash with ipcomp, will try to fix it later.

Yes, I can confirm this.
After disabling IPComp the crashes went away.

Thank you
Daniel

Re: IPSec freeze

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 646 bytes --]

Beschorner Daniel wrote:
>>I managed to reproduce a crash with ipcomp, will try to fix it later.
> 
> 
> Yes, I can confirm this.
> After disabling IPComp the crashes went away.


The crash happens in xfrm_bundle_ok when walking the bundle upwards
following xfrm_dst->u.next. The loop should be stopped when
xfrm_dst->u.next == first (the topmost xfrm_dst), but it points to
NULL instead. I'm pretty sure the attached patch is responsible,
it breaks XFRM's assumption that dst->next and xfrm_dst->u.next are
the same pointer and xfrm_dst now shares the next pointer with
rcu_head.next in struct dst_entry.

Eric, could you look into this please?

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1910 bytes --]

[NET]: Reorder fields of struct dst_entry

This last patch (but not least :) ) finally moves the next pointer at
the end of struct dst_entry. This permits to perform route cache
lookups with a minimal cost of one cache line per entry, instead of
two.

Both 32bits and 64bits platforms benefit from this new layout.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

---
commit 1e19e02ca0c5e33ea73a25127dbe6c3b8fcaac4b
tree 23afba2945a9e09b137b094a868ea176c1e1c800
parent 0c195c3fc4e95a06b0c0017506f074c94af99c35
author Eric Dumazet <dada1@cosmosbay.com> Fri, 09 Feb 2007 16:26:55 -0800
committer David S. Miller <davem@sunset.davemloft.net> Sat, 10 Feb 2007 23:20:45 -0800

 include/net/dst.h |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/include/net/dst.h b/include/net/dst.h
index 5d62342..e12a8ce 100644
--- a/include/net/dst.h
+++ b/include/net/dst.h
@@ -37,14 +37,7 @@ struct sk_buff;
 
 struct dst_entry
 {
-	union {
-		struct dst_entry *next;
-		struct rtable    *rt_next;
-		struct rt6_info   *rt6_next;
-		struct dn_route  *dn_next;
-	};
-	atomic_t		__refcnt;	/* client references	*/
-	int			__use;
+	struct rcu_head		rcu_head;
 	struct dst_entry	*child;
 	struct net_device       *dev;
 	short			error;
@@ -55,7 +48,6 @@ struct dst_entry
 #define DST_NOPOLICY		4
 #define DST_NOHASH		8
 #define DST_BALANCED            0x10
-	unsigned long		lastuse;
 	unsigned long		expires;
 
 	unsigned short		header_len;	/* more space at head required */
@@ -80,8 +72,16 @@ struct dst_entry
 #endif
 
 	struct  dst_ops	        *ops;
-	struct rcu_head		rcu_head;
 		
+	unsigned long		lastuse;
+	atomic_t		__refcnt;	/* client references	*/
+	int			__use;
+	union {
+		struct dst_entry *next;
+		struct rtable    *rt_next;
+		struct rt6_info   *rt6_next;
+		struct dn_route  *dn_next;
+	};
 	char			info[0];
 };
 

Re: IPSec freeze

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Patrick McHardy wrote:
> Beschorner Daniel wrote:
>   
>>> I managed to reproduce a crash with ipcomp, will try to fix it later.
>>>       
>> Yes, I can confirm this.
>> After disabling IPComp the crashes went away.
>>     
>
>
> The crash happens in xfrm_bundle_ok when walking the bundle upwards
> following xfrm_dst->u.next. The loop should be stopped when
> xfrm_dst->u.next == first (the topmost xfrm_dst), but it points to
> NULL instead. I'm pretty sure the attached patch is responsible,
> it breaks XFRM's assumption that dst->next and xfrm_dst->u.next are
> the same pointer and xfrm_dst now shares the next pointer with
> rcu_head.next in struct dst_entry.
>
> Eric, could you look into this please?

I fixed it myself. Daniel, can you please test this patch?




[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1543 bytes --]

[XFRM]: Fix crash introduced by struct dst_entry reordering

XFRM expects xfrm_dst->u.next to be same pointer as dst->next, which
was broken by the dst_entry reordering in commit 1e19e02c~, causing
an oops in xfrm_bundle_ok when walking the bundle upwards.

Kill xfrm_dst->u.next and change the only user to use dst->next instead.

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit 20c2fee8cc562817f11752e1d87350d5994fa098
tree f42318b847e962aa637136e94722a688c231111a
parent 308ac1b6249226730b70fcf7c13a289c27ce2bf3
author Patrick McHardy <kaber@trash.net> Tue, 17 Jul 2007 18:11:29 +0200
committer Patrick McHardy <kaber@trash.net> Tue, 17 Jul 2007 18:11:29 +0200

 include/net/xfrm.h     |    1 -
 net/xfrm/xfrm_policy.c |    2 +-
 2 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/include/net/xfrm.h b/include/net/xfrm.h
index ae959e9..a5f80bf 100644
--- a/include/net/xfrm.h
+++ b/include/net/xfrm.h
@@ -585,7 +585,6 @@ static inline int xfrm_sec_ctx_match(struct xfrm_sec_ctx *s1, struct xfrm_sec_ct
 struct xfrm_dst
 {
 	union {
-		struct xfrm_dst		*next;
 		struct dst_entry	dst;
 		struct rtable		rt;
 		struct rt6_info		rt6;
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 157bfbd..b48f06f 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -2141,7 +2141,7 @@ int xfrm_bundle_ok(struct xfrm_policy *pol, struct xfrm_dst *first,
 		if (last == first)
 			break;
 
-		last = last->u.next;
+		last = (struct xfrm_dst *)last->u.dst.next;
 		last->child_mtu_cached = mtu;
 	}
 

Re: IPSec freeze

[Beschorner Daniel]

> >>> I managed to reproduce a crash with ipcomp, will try to 
> fix it later.
> >>>       
> >> Yes, I can confirm this.
> >> After disabling IPComp the crashes went away.
> >>     
> > The crash happens in xfrm_bundle_ok when walking the bundle upwards
> > following xfrm_dst->u.next. The loop should be stopped when
> > xfrm_dst->u.next == first (the topmost xfrm_dst), but it points to
> > NULL instead. I'm pretty sure the attached patch is responsible,
> > it breaks XFRM's assumption that dst->next and xfrm_dst->u.next are
> > the same pointer and xfrm_dst now shares the next pointer with
> > rcu_head.next in struct dst_entry.
> >
> > Eric, could you look into this please?
> 
> I fixed it myself. Daniel, can you please test this patch?

Many thanks Patrick!!!
I tested it and found it working!

No more crashes with IPComp and smaller PMTUs.
But the "pmtu discovery on SA ESP/..." messages don't disappear.

Daniel

Re: IPSec freeze

[Patrick McHardy]

Beschorner Daniel wrote:
>>I fixed it myself. Daniel, can you please test this patch?
> 
> 
> Many thanks Patrick!!!
> I tested it and found it working!


Thanks for testing.

> No more crashes with IPComp and smaller PMTUs.
> But the "pmtu discovery on SA ESP/..." messages don't disappear.


Thats probably a different issue. Please post the output of
"ip -x xfrm state" (obfuscate keys if you care ..).

pmtu discovery on SA

[Beschorner Daniel]

> > No more crashes with IPComp and smaller PMTUs.
> > But the "pmtu discovery on SA ESP/..." messages don't disappear.
> 
> Thats probably a different issue. Please post the output of
> "ip -x xfrm state" (obfuscate keys if you care ..).

Linux (MTU 1500) <-> ADSL router (1492) <-> ... <-> Linux (1500)

Both boxes show these messages.

src 1.1.1.1 dst 2.2.2.2
        proto comp spi 0x0000e901 reqid 16398 mode tunnel
        replay-window 0
        comp deflate 0x
src 2.2.2.2 dst 1.1.1.1
        proto comp spi 0x000001a3 reqid 16398 mode tunnel
        replay-window 0
        comp deflate 0x
src 1.1.1.1 dst 2.2.2.2
        proto esp spi 0x767eaf0b reqid 16397 mode transport
        replay-window 32
        auth hmac(sha1) 0xxxxxxxxxxxxxxxxxxxxxx
        enc cbc(aes) 0xxxxxxxxxxxxxxxxxxxxxx
src 2.2.2.2 dst 1.1.1.1
        proto esp spi 0x44c5adda reqid 16397 mode transport
        replay-window 32
        auth hmac(sha1) 0xxxxxxxxxxxxxxxxxxxxxx
        enc cbc(aes) 0xxxxxxxxxxxxxxxxxxxxxx
src 2.2.2.2 dst 1.1.1.1
        proto (null) spi 0xd527fed2 reqid 0 mode tunnel
        replay-window 0
src 1.1.1.1 dst 2.2.2.2
        proto (null) spi 0x4cf1a149 reqid 0 mode tunnel
        replay-window 0 

Re: pmtu discovery on SA

[Patrick McHardy]

Beschorner Daniel wrote:
>>>No more crashes with IPComp and smaller PMTUs.
>>>But the "pmtu discovery on SA ESP/..." messages don't disappear.
>>
>>Thats probably a different issue. Please post the output of
>>"ip -x xfrm state" (obfuscate keys if you care ..).
> 
> 
> Linux (MTU 1500) <-> ADSL router (1492) <-> ... <-> Linux (1500)

In a previous mail you mentioned:

> The router on the ADSL line (behind an ADSL Netopia router via Ethernet)
> crashed too with 1500, now it uses 1492.

So its using 1492 now? Otherwise it would be expected since the
initial MTU calculation would be based on 1500.

Please capture a full ICMP packet (with -s0) and send me the
binary dump.

Re: pmtu discovery on SA

[Beschorner Daniel]

> Beschorner Daniel wrote:
> >>>No more crashes with IPComp and smaller PMTUs.
> >>>But the "pmtu discovery on SA ESP/..." messages don't disappear.
> >>
> >>Thats probably a different issue. Please post the output of
> >>"ip -x xfrm state" (obfuscate keys if you care ..).
> > 
> > 
> > Linux (MTU 1500) <-> ADSL router (1492) <-> ... <-> Linux (1500)
> 
> In a previous mail you mentioned:
> 
> > The router on the ADSL line (behind an ADSL Netopia router 
> via Ethernet)
> > crashed too with 1500, now it uses 1492.
> 
> So its using 1492 now? Otherwise it would be expected since the
> initial MTU calculation would be based on 1500.

I'm using 1500 again.

Just to make it clear: Is this message an error, or a warning or just
for information?
Could you explain a bit please.
I have no connections hangs or other kind of problems.

Thank you
Daniel

Re: pmtu discovery on SA

[Patrick McHardy]

Beschorner Daniel wrote:
>>Beschorner Daniel wrote:
>>
>>So its using 1492 now? Otherwise it would be expected since the
>>initial MTU calculation would be based on 1500.
> 
> 
> I'm using 1500 again.
> 
> Just to make it clear: Is this message an error, or a warning or just
> for information?
> Could you explain a bit please.


Its a debugging message nowadays (NETDEBUG). I was mostly interested
in this since I changed the IPsec MTU calculation in 2.6.22 and it
might have been a bug.

> I have no connections hangs or other kind of problems.


Thanks for clarifying this. I guess the reason why you're seeing
them now and not before is that we're now always using the maximum
room available, while previously packets were usually a bit smaller
than possible.

Re: pmtu discovery on SA

[Mika Penttilä]

Patrick McHardy wrote:
> Beschorner Daniel wrote:
>   
>>> Beschorner Daniel wrote:
>>>
>>> So its using 1492 now? Otherwise it would be expected since the
>>> initial MTU calculation would be based on 1500.
>>>       
>> I'm using 1500 again.
>>
>> Just to make it clear: Is this message an error, or a warning or just
>> for information?
>> Could you explain a bit please.
>>     
>
>
> Its a debugging message nowadays (NETDEBUG). I was mostly interested
> in this since I changed the IPsec MTU calculation in 2.6.22 and it
> might have been a bug.
>
>   
And we don't have pmtu discovery for esp yet, right?

--Mika

Re: pmtu discovery on SA

[Patrick McHardy]

Mika Penttilä wrote:
> Patrick McHardy wrote:
> 
>> Its a debugging message nowadays (NETDEBUG). I was mostly interested
>> in this since I changed the IPsec MTU calculation in 2.6.22 and it
>> might have been a bug.
>>
>>   
> 
> And we don't have pmtu discovery for esp yet, right?


We do. The best I have seen to date in any IPsec implementation :)

Re: pmtu discovery on SA

[Mika Penttilä]

Patrick McHardy wrote:
> Mika Penttilä wrote:
>   
>> Patrick McHardy wrote:
>>
>>     
>>> Its a debugging message nowadays (NETDEBUG). I was mostly interested
>>> in this since I changed the IPsec MTU calculation in 2.6.22 and it
>>> might have been a bug.
>>>
>>>   
>>>       
>> And we don't have pmtu discovery for esp yet, right?
>>     
>
>
> We do. The best I have seen to date in any IPsec implementation :)
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   
Hmm. esp4_err() looks like this :

    struct iphdr *iph = (struct iphdr*)skb->data;
    struct ip_esp_hdr *esph = (struct ip_esp_hdr*)(skb->data+(iph->ihl<<2));
    struct xfrm_state *x;

    if (icmp_hdr(skb)->type != ICMP_DEST_UNREACH ||
        icmp_hdr(skb)->code != ICMP_FRAG_NEEDED)
        return;

    x = xfrm_state_lookup((xfrm_address_t *)&iph->daddr, esph->spi, 
IPPROTO_ESP, AF_INET);
    if (!x)
        return;
    NETDEBUG(KERN_DEBUG "pmtu discovery on SA ESP/%08x/%08x\n",
         ntohl(esph->spi), ntohl(iph->daddr));
    xfrm_state_put(x);



where could pmtu discovery be happening?

--Mika

Re: pmtu discovery on SA

[Patrick McHardy]

Mika Penttilä wrote:
> Hmm. esp4_err() looks like this :
>
>    struct iphdr *iph = (struct iphdr*)skb->data;
>    struct ip_esp_hdr *esph = (struct 
> ip_esp_hdr*)(skb->data+(iph->ihl<<2));
>    struct xfrm_state *x;
>
>    if (icmp_hdr(skb)->type != ICMP_DEST_UNREACH ||
>        icmp_hdr(skb)->code != ICMP_FRAG_NEEDED)
>        return;
>
>    x = xfrm_state_lookup((xfrm_address_t *)&iph->daddr, esph->spi, 
> IPPROTO_ESP, AF_INET);
>    if (!x)
>        return;
>    NETDEBUG(KERN_DEBUG "pmtu discovery on SA ESP/%08x/%08x\n",
>         ntohl(esph->spi), ntohl(iph->daddr));
>    xfrm_state_put(x);
>
>
>
> where could pmtu discovery be happening?

xfrm_init_pmtu, xfrm_bundle_ok, xfrm_state_mtu, esp4_get_mtu, ...

Re: pmtu discovery on SA

[Mika Penttilä]

Patrick McHardy wrote:
> Mika Penttilä wrote:
>> Hmm. esp4_err() looks like this :
>>
>>    struct iphdr *iph = (struct iphdr*)skb->data;
>>    struct ip_esp_hdr *esph = (struct 
>> ip_esp_hdr*)(skb->data+(iph->ihl<<2));
>>    struct xfrm_state *x;
>>
>>    if (icmp_hdr(skb)->type != ICMP_DEST_UNREACH ||
>>        icmp_hdr(skb)->code != ICMP_FRAG_NEEDED)
>>        return;
>>
>>    x = xfrm_state_lookup((xfrm_address_t *)&iph->daddr, esph->spi, 
>> IPPROTO_ESP, AF_INET);
>>    if (!x)
>>        return;
>>    NETDEBUG(KERN_DEBUG "pmtu discovery on SA ESP/%08x/%08x\n",
>>         ntohl(esph->spi), ntohl(iph->daddr));
>>    xfrm_state_put(x);
>>
>>
>>
>> where could pmtu discovery be happening?
>
> xfrm_init_pmtu, xfrm_bundle_ok, xfrm_state_mtu, esp4_get_mtu, ...
>
Okay yes but for instance the current tcp session isn't recovering from 
from esp4_err() ? The next connect attempt uses the new pmtu?

--Mika

Re: pmtu discovery on SA

[Beschorner Daniel]

> > Just to make it clear: Is this message an error, or a 
> warning or just
> > for information?
> > Could you explain a bit please.
> 
> 
> Its a debugging message nowadays (NETDEBUG). I was mostly interested
> in this since I changed the IPsec MTU calculation in 2.6.22 and it
> might have been a bug.
> 
> > I have no connections hangs or other kind of problems.
> 
> 
> Thanks for clarifying this. I guess the reason why you're seeing
> them now and not before is that we're now always using the maximum
> room available, while previously packets were usually a bit smaller
> than possible.

Yes, our tunnels gained 2 bytes in 2.6.22. Maybe I would have seen the
messages earlier with older kernels too, but our ADSL line started with
2.6.22.

I tested a little bit around and found the ESP pmtud is doing it's job
well.
The message really just says what it does.

Thank you for clarifying this and most notably for fixing the crash bug
so fast!

Daniel

Re: IPSec freeze

[David Miller]

From: "Beschorner Daniel" <Daniel.Beschorner@facton.com>
Date: Tue, 17 Jul 2007 21:03:20 +0200

> > >>> I managed to reproduce a crash with ipcomp, will try to 
> > fix it later.
> > >>>       
> > >> Yes, I can confirm this.
> > >> After disabling IPComp the crashes went away.
> > >>     
> > > The crash happens in xfrm_bundle_ok when walking the bundle upwards
> > > following xfrm_dst->u.next. The loop should be stopped when
> > > xfrm_dst->u.next == first (the topmost xfrm_dst), but it points to
> > > NULL instead. I'm pretty sure the attached patch is responsible,
> > > it breaks XFRM's assumption that dst->next and xfrm_dst->u.next are
> > > the same pointer and xfrm_dst now shares the next pointer with
> > > rcu_head.next in struct dst_entry.
> > >
> > > Eric, could you look into this please?
> > 
> > I fixed it myself. Daniel, can you please test this patch?
> 
> Many thanks Patrick!!!
> I tested it and found it working!

Thank you for testing.

Re: IPSec freeze

[David Miller]

From: Patrick McHardy <kaber@trash.net>
Date: Tue, 17 Jul 2007 18:10:13 +0200

> [XFRM]: Fix crash introduced by struct dst_entry reordering
> 
> XFRM expects xfrm_dst->u.next to be same pointer as dst->next, which
> was broken by the dst_entry reordering in commit 1e19e02c~, causing
> an oops in xfrm_bundle_ok when walking the bundle upwards.
> 
> Kill xfrm_dst->u.next and change the only user to use dst->next instead.
> 
> Signed-off-by: Patrick McHardy <kaber@trash.net>

Applied, thanks a lot Patrick.

I'll toss this to -stable too.

Re: IPSec freeze

[Beschorner Daniel]

> Today a new site joined our Linux IPSec VPN, now all the 
> other routers (all 2.6.22) freeze hard reproducible.
> No oops, no sysreq, only hard reset rewakes them.

Ok, I did a longer test and nothing crashed in the mean time without
IPComp.
So it really must have been the reason.

BTW now I see the "pmtu discovery on SA" regression in 2.6.22, there is
already another thread about.
With 2.6.15 we had this already, but a patch from Herbert fixed it (TOS
field matching in ICMP IIRC).

Daniel