IPSec broken in 2.6.18-rc4-mm3

IPSec broken in 2.6.18-rc4-mm3

[Gnome42 Gnome42]

Hi Folks,

(please CC me ...)

IPSec got broken in 2.6.18-rc4-mm3+, 2.6.18-rc4-mm2 works and
2.6.18-rc5 also works.

The tunnel looks like its established correctly in the racoon logs and
the traffic is encrypted on the wire. However, the other side does not
decrypt the traffic it just seems to disappear.

I have confirmed this problem exists between two linux boxen and a
Netopia router as well.

The git-net.patch increased in size by about 50% between
2.6.18-rc4-mm2 and 2.6.18-rc4-mm3 (likely suspect?), but i was unable
to simply patch -R it cleanly.

Suggestions?

Shane

Re: IPSec broken in 2.6.18-rc4-mm3

[Patrick McHardy]

Gnome42 Gnome42 wrote:
> IPSec got broken in 2.6.18-rc4-mm3+, 2.6.18-rc4-mm2 works and
> 2.6.18-rc5 also works.
> 
> The tunnel looks like its established correctly in the racoon logs and
> the traffic is encrypted on the wire. However, the other side does not
> decrypt the traffic it just seems to disappear.

Can you see the decrypted packets on the incoming interface on the
other side?

> I have confirmed this problem exists between two linux boxen and a
> Netopia router as well.
> 
> The git-net.patch increased in size by about 50% between
> 2.6.18-rc4-mm2 and 2.6.18-rc4-mm3 (likely suspect?), but i was unable
> to simply patch -R it cleanly.
> 
> Suggestions?

Please post your policies and related SAs from both sides.
Are you using NAT, iptables or anything like that?

Re: IPSec broken in 2.6.18-rc4-mm3

[Gnome42 Gnome42]

On 9/8/06, Patrick McHardy <kaber@trash.net> wrote:
> Gnome42 Gnome42 wrote:

> Can you see the decrypted packets on the incoming interface on the
> other side?

No, not the decrypted ones only the encrypted ones. I never see the
decrypted packets. ( I should see them twice right? Once encrypted and
once decrypted?)

> Please post your policies and related SAs from both sides.
> Are you using NAT, iptables or anything like that?

(Beware, I am not knowledgeable about IPSec :)

I am testing this between my workstation and my linux/firewall/nat box
with adsl. So encrypted on my local lan only.

The firewall box is using iptables and is natting for me but the ipsec
traffic is just local and is not natted. I am testing roadwarrior
mode, with the firewall as the responder. No iptables/NAT on my
workstation. I have allowed protocols 50/51 and udp 500 and it works
well with other kernels including 2.6.18-rc5, so I think the iptables
stuff is OK.

On my workstation(34.34.36.1) I use:
spdadd 34.34.36.1 206.207.0.0/16 any -P out ipsec
           esp/tunnel/34.34.36.1-34.34.36.6/use;
spdadd 206.207.0.0/16 34.34.36.1 any -P in ipsec
           esp/tunnel/34.34.36.6-34.34.36.1/use;

and on the firewall:
remote anonymous {
        exchange_mode aggressive,main;
        passive on;
        my_identifier fqdn "blah1";
        peers_identifier fqdn "blah2";
        verify_identifier on;
        proposal {
                encryption_algorithm aes;
                hash_algorithm md5;
                authentication_method pre_shared_key;
                dh_group modp1024;
        }
        generate_policy on;
}
sainfo anonymous {
        pfs_group modp1024;
        encryption_algorithm aes;
        authentication_algorithm hmac_md5;
        compression_algorithm deflate;
}

... or did you mean dumps from setkey -D[P]?

Re: IPSec broken in 2.6.18-rc4-mm3

[Patrick McHardy]

Gnome42 Gnome42 wrote:
> On 9/8/06, Patrick McHardy <kaber@trash.net> wrote:
>>
>> Can you see the decrypted packets on the incoming interface on the
>> other side?
> 
> 
> No, not the decrypted ones only the encrypted ones. I never see the
> decrypted packets. ( I should see them twice right? Once encrypted and
> once decrypted?)

Yes.

>> Please post your policies and related SAs from both sides.
>> Are you using NAT, iptables or anything like that?
> 
> 
> [...]
> ... or did you mean dumps from setkey -D[P]?

Yes, I meant the SAs. But please use "ip -s xfrm state" and "ip -s xfrm
policy" (on both sides), they include a bit more information than
setkey.

Re: IPSec broken in 2.6.18-rc4-mm3

[Gnome42]

On 9/9/06, Patrick McHardy <kaber@trash.net> wrote:
>
> Yes, I meant the SAs. But please use "ip -s xfrm state" and "ip -s xfrm
> policy" (on both sides), they include a bit more information than
> setkey.

Workstation running 2.6.18-rc5-mm1 is the initiator, and responder is
2.6.17-rc6-mm1. This is the not working scenario.

I have snipped stanzas from the policy ouput that contained 0.0.0.0 as
src and dest addr.

initiator.state:

src 34.34.36.1 dst 34.34.36.6
        proto esp spi 0x0dc3aba4(230927268) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x00000001 flag  (0x00000000)
        auth hmac(md5) 0xfea9e3e8d324265d8b7e17ec42d69b15 (128 bits)
        enc cbc(aes) 0x21ca0a9677ff0225acd0d3f4a9bdcf61 (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          4560(bytes), 30(packets)
          add 2006-09-09 10:21:41 use 2006-09-09 10:21:46
        stats:
          replay-window 0 replay 0 failed 0
src 34.34.36.6 dst 34.34.36.1
        proto esp spi 0x0c882b3c(210250556) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x00000001 flag  (0x00000000)
        auth hmac(md5) 0x93f6d1f6474834e8c82ea4b9865da961 (128 bits)
        enc cbc(aes) 0xad702c25e42826e5f2ad704808dcc381 (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use -
        stats:
          replay-window 0 replay 0 failed 0

initiator.policy:

src 206.207.0.0/16 dst 34.34.36.1/32 uid 0
        dir in action allow index 40 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:14 use 2006-09-09 10:21:22
        tmpl src 34.34.36.6 dst 34.34.36.1
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level use share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 34.34.36.1/32 dst 206.207.0.0/16 uid 0
        dir out action allow index 33 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:14 use 2006-09-09 10:22:15
        tmpl src 34.34.36.1 dst 34.34.36.6
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level use share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 206.207.0.0/16 dst 34.34.36.1/32 uid 0
        dir fwd action allow index 50 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 0(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:14 use -
        tmpl src 34.34.36.6 dst 34.34.36.1
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level use share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff

[.. snip 4 stanza with src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 ..]

responder.state:

src 34.34.36.6 dst 34.34.36.1
        proto esp spi 0x0c882b3c(210250556) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x991250886 flag  (0x00000000)
        auth md5 0x93f6d1f6474834e8c82ea4b9865da961 (128 bits)
        enc aes 0xad702c25e42826e5f2ad704808dcc381 (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use -
        stats:
          replay-window 0 replay 0 failed 0
src 34.34.36.1 dst 34.34.36.6
        proto esp spi 0x0dc3aba4(230927268) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x991250886 flag  (0x00000000)
        auth md5 0xfea9e3e8d324265d8b7e17ec42d69b15 (128 bits)
        enc aes 0x21ca0a9677ff0225acd0d3f4a9bdcf61 (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use 2006-09-09 10:21:46
        stats:
          replay-window 0 replay 0 failed 30

responder.policy:

src 34.34.36.1/32 dst 206.207.0.0/16 uid 0
        dir in action allow index 2728 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use -
        tmpl src 34.34.36.1 dst 34.34.36.6
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 206.207.0.0/16 dst 34.34.36.1/32 uid 0
        dir out action allow index 2745 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use -
        tmpl src 34.34.36.6 dst 34.34.36.1
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff
src 34.34.36.1/32 dst 206.207.0.0/16 uid 0
        dir fwd action allow index 2738 priority 0 share any flag 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 0(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 10:21:41 use -
        tmpl src 34.34.36.1 dst 34.34.36.6
                proto esp spi 0x00000000(0) reqid 0(0x00000000) mode tunnel
                level required share any
                enc-mask ffffffff auth-mask ffffffff comp-mask ffffffff

[.. snip 6 stanza with src 0.0.0.0/0 dst 0.0.0.0/0 uid 0 ..]

Regards,

Shane

Re: IPSec broken in 2.6.18-rc4-mm3

[Patrick McHardy]

Gnome42 wrote:
> src 34.34.36.1 dst 34.34.36.6
>        proto esp spi 0x0dc3aba4(230927268) reqid 0(0x00000000) mode tunnel
>        replay-window 4 seq 0x00000001 flag  (0x00000000)
>        auth hmac(md5) 0xfea9e3e8d324265d8b7e17ec42d69b15 (128 bits)
>        enc cbc(aes) 0x21ca0a9677ff0225acd0d3f4a9bdcf61 (128 bits)
>        lifetime config:
>          limit: soft (INF)(bytes), hard (INF)(bytes)
>          limit: soft (INF)(packets), hard (INF)(packets)
>          expire add: soft 23040(sec), hard 28800(sec)
>          expire use: soft 0(sec), hard 0(sec)
>        lifetime current:
>          4560(bytes), 30(packets)
>          add 2006-09-09 10:21:41 use 2006-09-09 10:21:46
>        stats:
>          replay-window 0 replay 0 failed 0

> src 34.34.36.1 dst 34.34.36.6
>        proto esp spi 0x0dc3aba4(230927268) reqid 0(0x00000000) mode tunnel
>        replay-window 4 seq 0x991250886 flag  (0x00000000)
>        auth md5 0xfea9e3e8d324265d8b7e17ec42d69b15 (128 bits)
>        enc aes 0x21ca0a9677ff0225acd0d3f4a9bdcf61 (128 bits)
>        lifetime config:
>          limit: soft (INF)(bytes), hard (INF)(bytes)
>          limit: soft (INF)(packets), hard (INF)(packets)
>          expire add: soft 23040(sec), hard 28800(sec)
>          expire use: soft 0(sec), hard 0(sec)
>        lifetime current:
>          0(bytes), 0(packets)
>          add 2006-09-09 10:21:41 use 2006-09-09 10:21:46
>        stats:
>          replay-window 0 replay 0 failed 30

                                           ^^
This seems to be the problem, the sequence-numbers are outside the valid
window. I can't find anything that would cause this, please post a
tcpdump of the packets so we can see which values get used.

Re: IPSec broken in 2.6.18-rc4-mm3

[Gnome42]

On 9/9/06, Patrick McHardy <kaber@trash.net> wrote:

> > src 34.34.36.1 dst 34.34.36.6
> >        proto esp spi 0x0dc3aba4(230927268) reqid 0(0x00000000) mode tunnel
> >        replay-window 4 seq 0x991250886 flag  (0x00000000)
> >        auth md5 0xfea9e3e8d324265d8b7e17ec42d69b15 (128 bits)
> >        enc aes 0x21ca0a9677ff0225acd0d3f4a9bdcf61 (128 bits)
> >        lifetime config:
> >          limit: soft (INF)(bytes), hard (INF)(bytes)
> >          limit: soft (INF)(packets), hard (INF)(packets)
> >          expire add: soft 23040(sec), hard 28800(sec)
> >          expire use: soft 0(sec), hard 0(sec)
> >        lifetime current:
> >          0(bytes), 0(packets)
> >          add 2006-09-09 10:21:41 use 2006-09-09 10:21:46
> >        stats:
> >          replay-window 0 replay 0 failed 30
>
>                                            ^^
> This seems to be the problem, the sequence-numbers are outside the valid
> window. I can't find anything that would cause this, please post a
> tcpdump of the packets so we can see which values get used.

On the responder:  tcpdump -n -i eth0 proto 50 or proto 51

13:27:30.416624 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x1), length 132
13:27:31.415752 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x2), length 132
13:27:32.415582 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x3), length 132
13:27:33.415390 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x4), length 132
13:27:34.415228 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x5), length 132
13:27:35.415054 IP 34.34.36.1 > 34.34.36.6:
ESP(spi=0x09a205f0,seq=0x6), length 132

and the  ip -s xfrm state  output:

These first two entries (X's and Y's) are from another working tunnel
on the firewall. I had shutdown this tunnel when I posted the previous
results.

src X.X.X.X dst Y.Y.Y.Y
        proto esp spi 0xb435a4c1(3023414465) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x00000023 flag  (0x00000000)
        auth md5 0x862b4e72af7910ceaa014f758294b965 (128 bits)
        enc des3_ede
0x328a552e490122c0eb74c966446cca0ee9df334ffbafc7c4 (192 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 483840(sec), hard 604800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          18619776(bytes), 50494(packets)
          add 2006-09-09 10:45:20 use 2006-09-09 10:45:21
        stats:
          replay-window 0 replay 0 failed 0
src Y.Y.Y.Y dst X.X.X.X
        proto esp spi 0x0bb67022(196505634) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x00000023 flag  (0x00000000)
        auth md5 0xdb7e6457c94cb90e2e5743afbf85517d (128 bits)
        enc des3_ede
0x263be9438dccf9497966e37b1082dec5d4bec2656b34d377 (192 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 483840(sec), hard 604800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          10811535(bytes), 27716(packets)
          add 2006-09-09 10:45:20 use 2006-09-09 10:45:21
        stats:
          replay-window 0 replay 0 failed 166
src 34.34.36.6 dst 34.34.36.1
        proto esp spi 0x079c9d34(127704372) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x3698599912 flag  (0x00000000)
        auth md5 0x6c995f0e7feda87c4ffae49697bdb773 (128 bits)
        enc aes 0x50a3ad275e4441844ed775357ef74bcd (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 13:27:29 use -
        stats:
          replay-window 0 replay 0 failed 0
src 34.34.36.1 dst 34.34.36.6
        proto esp spi 0x09a205f0(161613296) reqid 0(0x00000000) mode tunnel
        replay-window 4 seq 0x3698599912 flag  (0x00000000)
        auth md5 0x5e607ebf5614c79c7eec3064d25fa2a9 (128 bits)
        enc aes 0x34d12010227216e2b3de254090c4fc40 (128 bits)
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 23040(sec), hard 28800(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2006-09-09 13:27:29 use 2006-09-09 13:27:30
        stats:
          replay-window 0 replay 0 failed 399

Re: IPSec broken in 2.6.18-rc4-mm3

[Gnome42]

Hi Patrick,

It is working in 2.6.18-rc6-mm1. I thought it was the compile option
'optimize for size' that was causing a miscompilation because when I
compiled -rc6-mm1 I turned that option off and it suddenly started
working. But, then I recompiled -rc5-mm1 with that option off and it
still didn't work. So, it must have been fixed in -mm1 itself I guess.

Thanks for your time & help,

Shane

Re: IPSec broken in 2.6.18-rc4-mm3

[Patrick McHardy]

Gnome42 wrote:
> It is working in 2.6.18-rc6-mm1. I thought it was the compile option
> 'optimize for size' that was causing a miscompilation because when I
> compiled -rc6-mm1 I turned that option off and it suddenly started
> working. But, then I recompiled -rc5-mm1 with that option off and it
> still didn't work. So, it must have been fixed in -mm1 itself I guess.

Good, I coulnd't find anything (but discovered that the statistic didn't
count out of replay window packets but failed authenticity checks).

Thanks for the information.