Iptables / KAME IPSec problem: source port information lost

Iptables / KAME IPSec problem: source port information lost

[Carsten Maass]

Dear List,

my network layout looks like this:
----------------------------------

(Localnet A)---(Gateway A)==IPSec==(Gateway B)---(Localnet B)

where "==IPSec==" is a IPSec connection using the backport of the 2.6 
KAME IPSec stack on a static 2.4.24 kernel in tunnel-mode. Both gateways 
are running Debian stable with racoon and the ipsec-tools coming from 
testing:

kernel-source-2.4.24          2.4.24-3
iptables                      1.2.6a-5
ipsec-tools                   0.2.2-8
racoon                        0.2.2-8


My problem is:
--------------

At some point in the filtering process, the information about the source 
ports of the incoming and decrypted packet gets lost, so that the 
clients on the localnet don't accept it as one of the established 
connection.

Here is some example:
---------------------

The tcpdump was taken on Gateway A, listening on all interfaces.

Client 192.168.3.4 on Localnet A tries to establish a http-connection to 
client 192.168.0.1 on Localnet B.

12:05:48.370984 192.168.3.4.35114 > 192.168.0.1.80: S 
3186461765:3186461765(0) win 5840 <mss 1460,sackOK,timestamp 220395665 
0,nop,wscale 0> (DF)

The initial packet...

12:05:48.372462 218.10.35.139 > 146.254.136.108: 
ESP(spi=0x0a6825be,seq=0xaf) (DF)

...gets correctly encrypted and sent out to Gateway B (146.254.136.108).

12:05:48.447361 146.254.136.108 > 218.10.35.139: 
ESP(spi=0x03e34241,seq=0xa5) (DF)

The encrypted response arrives at Gateway A (218.10.35.139).

12:05:48.447361 192.168.0.1.80 > 192.168.3.4.35114: S 
969293027:969293027(0) ack 3186461766 win 5792 <mss 
1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)

The packet gets decrypted, the source port is correct.

12:05:48.448584 192.168.0.1.1 > 192.168.3.4.35114: S 
969293027:969293027(0) ack 3186461766 win 5792 <mss 
1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)

And here the strange thing apperars: The packet is sent out with a 
source port of 1!

12:05:48.448816 192.168.3.4.35114 > 192.168.0.1.1: R 
3186461766:3186461766(0) win 0 (DF)

As it comes from source port 1, the client does not recognize it as a 
packet which belongs to the established connection and rejects it.

12:05:48.449038 218.10.35.139 > 192.168.0.1: 
ESP(spi=0x0a6825be,seq=0xb0) (DF)

The reject gets encrypted, sent out over the tunnel and the game starts 
over again.

Portless connections like ping don't show this behavior and gets 
established correctly.

So my questions are:
--------------------

I this an iptables problem or IPSec related?
Can you think of any iptables rules, which causes this behavior?
Do you know how to explore this problem further?
Do you have any other hints?


I hope i was able to make the problem clear. Please tell me if you need 
more information. As i have no clue how to go on with this problem, any 
help would be highly appreciated.

Thanks for your time and patience,
Carsten.

Re: Iptables / KAME IPSec problem: source port information lost

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 3440 bytes --]

On Tue, Mar 09, 2004 at 01:27:18PM +0100, Carsten Maass wrote:
> Dear List,
> 
> my network layout looks like this:
> ----------------------------------
> 
> (Localnet A)---(Gateway A)==IPSec==(Gateway B)---(Localnet B)
> 
> where "==IPSec==" is a IPSec connection using the backport of the 2.6 
> KAME IPSec stack on a static 2.4.24 kernel in tunnel-mode. Both gateways 
> are running Debian stable with racoon and the ipsec-tools coming from 
> testing:
> 
> kernel-source-2.4.24          2.4.24-3
> iptables                      1.2.6a-5
> ipsec-tools                   0.2.2-8
> racoon                        0.2.2-8
> 
> 
> My problem is:
> --------------
> 
> At some point in the filtering process, the information about the source 
> ports of the incoming and decrypted packet gets lost, so that the 
> clients on the localnet don't accept it as one of the established 
> connection.
> 
> Here is some example:
> ---------------------
> 
> The tcpdump was taken on Gateway A, listening on all interfaces.
> 
> Client 192.168.3.4 on Localnet A tries to establish a http-connection to 
> client 192.168.0.1 on Localnet B.
> 
> 12:05:48.370984 192.168.3.4.35114 > 192.168.0.1.80: S 
> 3186461765:3186461765(0) win 5840 <mss 1460,sackOK,timestamp 220395665 
> 0,nop,wscale 0> (DF)
> 
> The initial packet...
> 
> 12:05:48.372462 218.10.35.139 > 146.254.136.108: 
> ESP(spi=0x0a6825be,seq=0xaf) (DF)
> 
> ...gets correctly encrypted and sent out to Gateway B (146.254.136.108).
> 
> 12:05:48.447361 146.254.136.108 > 218.10.35.139: 
> ESP(spi=0x03e34241,seq=0xa5) (DF)
> 
> The encrypted response arrives at Gateway A (218.10.35.139).
> 
> 12:05:48.447361 192.168.0.1.80 > 192.168.3.4.35114: S 
> 969293027:969293027(0) ack 3186461766 win 5792 <mss 
> 1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)

Sorry but isnt this the Sync/Ack packat in response to the sync above

> 
> The packet gets decrypted, the source port is correct.
> 
> 12:05:48.448584 192.168.0.1.1 > 192.168.3.4.35114: S 
> 969293027:969293027(0) ack 3186461766 win 5792 <mss 
> 1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)

Isn't this the same packet above but looks NAT'd ? Not sure why it is
coming from port 1 though.

> 
> And here the strange thing apperars: The packet is sent out with a 
> source port of 1!
> 
> 12:05:48.448816 192.168.3.4.35114 > 192.168.0.1.1: R 
> 3186461766:3186461766(0) win 0 (DF)
> 
> As it comes from source port 1, the client does not recognize it as a 
> packet which belongs to the established connection and rejects it.
> 
> 12:05:48.449038 218.10.35.139 > 192.168.0.1: 
> ESP(spi=0x0a6825be,seq=0xb0) (DF)
> 
> The reject gets encrypted, sent out over the tunnel and the game starts 
> over again.
> 
> Portless connections like ping don't show this behavior and gets 
> established correctly.
> 
> So my questions are:
> --------------------
> 
> I this an iptables problem or IPSec related?
> Can you think of any iptables rules, which causes this behavior?
> Do you know how to explore this problem further?
> Do you have any other hints?
> 
> 
> I hope i was able to make the problem clear. Please tell me if you need 
> more information. As i have no clue how to go on with this problem, any 
> help would be highly appreciated.
> 
> Thanks for your time and patience,
> Carsten.
> 
> 
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Iptables / KAME IPSec problem: source port information lost

[Carsten Maass]

Alexander Samad wrote:
>>My problem is:
>>--------------
>>
>>At some point in the filtering process, the information about the source 
>>ports of the incoming and decrypted packet gets lost, so that the 
>>clients on the localnet don't accept it as one of the established 
>>connection.
>>
>>Here is some example:
>>---------------------
>>
>>The tcpdump was taken on Gateway A, listening on all interfaces.
>>
>>Client 192.168.3.4 on Localnet A tries to establish a http-connection to 
>>client 192.168.0.1 on Localnet B.
>>
>>12:05:48.370984 192.168.3.4.35114 > 192.168.0.1.80: S 
>>3186461765:3186461765(0) win 5840 <mss 1460,sackOK,timestamp 220395665 
>>0,nop,wscale 0> (DF)
>>
>>The initial packet...
>>
>>12:05:48.372462 218.10.35.139 > 146.254.136.108: 
>>ESP(spi=0x0a6825be,seq=0xaf) (DF)
>>
>>...gets correctly encrypted and sent out to Gateway B (146.254.136.108).
>>
>>12:05:48.447361 146.254.136.108 > 218.10.35.139: 
>>ESP(spi=0x03e34241,seq=0xa5) (DF)
>>
>>The encrypted response arrives at Gateway A (218.10.35.139).
>>
>>12:05:48.447361 192.168.0.1.80 > 192.168.3.4.35114: S 
>>969293027:969293027(0) ack 3186461766 win 5792 <mss 
>>1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)
> 
> 
> Sorry but isnt this the Sync/Ack packat in response to the sync above

Yes, but did i say anything else?. Maybe my formatting was misleading:

First Line = packet
Next line = description of packet

>>The packet gets decrypted, the source port is correct.
>>
>>12:05:48.448584 192.168.0.1.1 > 192.168.3.4.35114: S 
>>969293027:969293027(0) ack 3186461766 win 5792 <mss 
>>1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)
> 
> 
> Isn't this the same packet above but looks NAT'd ? Not sure why it is
> coming from port 1 though.

My guess: As tcpdump was listening on all interfaces, the packet appears 
three times. 1. in encrypted form arriving on inet_iface, 2. in 
decrypted form on inet_iface, 3. as it leaves the box on lan_iface. But 
the question remains: why does the source port change? Is there a way to 
determine at which stage of the filtering chain the change occurs?

>>And here the strange thing apperars: The packet is sent out with a 
>>source port of 1!
>>
>>12:05:48.448816 192.168.3.4.35114 > 192.168.0.1.1: R 
>>3186461766:3186461766(0) win 0 (DF)
>>
>>As it comes from source port 1, the client does not recognize it as a 
>>packet which belongs to the established connection and rejects it.
>>
>>12:05:48.449038 218.10.35.139 > 192.168.0.1: 
>>ESP(spi=0x0a6825be,seq=0xb0) (DF)
>>
>>The reject gets encrypted, sent out over the tunnel and the game starts 
>>over again.
>>
>>Portless connections like ping don't show this behavior and gets 
>>established correctly.

Thank you for your answer,
Carsten.

[solved] Re: Iptables / KAME IPSec problem: source port information lost

[Carsten Maass]

Carsten Maass wrote:
> my network layout looks like this:
> ----------------------------------
> 
> (Localnet A)---(Gateway A)==IPSec==(Gateway B)---(Localnet B)
> 
> where "==IPSec==" is a IPSec connection using the backport of the 2.6 
> KAME IPSec stack on a static 2.4.24 kernel in tunnel-mode. Both gateways 
> are running Debian stable with racoon and the ipsec-tools coming from 
> testing:
> 
> kernel-source-2.4.24          2.4.24-3
> iptables                      1.2.6a-5
> ipsec-tools                   0.2.2-8
> racoon                        0.2.2-8
> 
> 
> My problem is:
> --------------
> 
> At some point in the filtering process, the information about the source 
> ports of the incoming and decrypted packet gets lost, so that the 
> clients on the localnet don't accept it as one of the established 
> connection.
> 
> Here is some example:
> ---------------------
> 
> The tcpdump was taken on Gateway A, listening on all interfaces.
> 
> Client 192.168.3.4 on Localnet A tries to establish a http-connection to 
> client 192.168.0.1 on Localnet B.
> 
> 12:05:48.370984 192.168.3.4.35114 > 192.168.0.1.80: S 
> 3186461765:3186461765(0) win 5840 <mss 1460,sackOK,timestamp 220395665 
> 0,nop,wscale 0> (DF)
> 
> The initial packet...
> 
> 12:05:48.372462 218.10.35.139 > 146.254.136.108: 
> ESP(spi=0x0a6825be,seq=0xaf) (DF)
> 
> ...gets correctly encrypted and sent out to Gateway B (146.254.136.108).
> 
> 12:05:48.447361 146.254.136.108 > 218.10.35.139: 
> ESP(spi=0x03e34241,seq=0xa5) (DF)
> 
> The encrypted response arrives at Gateway A (218.10.35.139).
> 
> 12:05:48.447361 192.168.0.1.80 > 192.168.3.4.35114: S 
> 969293027:969293027(0) ack 3186461766 win 5792 <mss 
> 1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)
> 
> The packet gets decrypted, the source port is correct.
> 
> 12:05:48.448584 192.168.0.1.1 > 192.168.3.4.35114: S 
> 969293027:969293027(0) ack 3186461766 win 5792 <mss 
> 1460,sackOK,timestamp 2220996862 220395665,nop,wscale 0> (DF)
> 
> And here the strange thing apperars: The packet is sent out with a 
> source port of 1!
> 
> 12:05:48.448816 192.168.3.4.35114 > 192.168.0.1.1: R 
> 3186461766:3186461766(0) win 0 (DF)
> 
> As it comes from source port 1, the client does not recognize it as a 
> packet which belongs to the established connection and rejects it.
> 
> 12:05:48.449038 218.10.35.139 > 192.168.0.1: 
> ESP(spi=0x0a6825be,seq=0xb0) (DF)
> 
> The reject gets encrypted, sent out over the tunnel and the game starts 
> over again.
> 
> Portless connections like ping don't show this behavior and gets 
> established correctly.

Even if it's not completely clear to me why: the offending rule was

$IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP

It seems to do some kind of NATing to all traffic leaving the gateway. 
When i narrowed it down to only match traffic which comes out of the LAN 
it works like a charm:

$IPTABLES -t nat -A POSTROUTING -s 192.168.3.0/24 -o $INET_IFACE -j SNAT 
--to-source $INET_IP

Is this a bug?

Greetings,
Carsten.

Re: [solved] Re: Iptables / KAME IPSec problem: source port information lost

[Antony Stone]

On Saturday 13 March 2004 10:16 pm, Carsten Maass wrote:

> Even if it's not completely clear to me why: the offending rule was
>
> $IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP
>
> It seems to do some kind of NATing to all traffic leaving the gateway.

Er, why shouldn't it do this?   The rule says "for any packet leaving 
$INET_IFACE, change the Source Address so that it is $INET_IP".

Therefore all traffic leaving the gateway is going to be SNATted.... no 
surprise there?

> When i narrowed it down to only match traffic which comes out of the LAN
> it works like a charm:
>
> $IPTABLES -t nat -A POSTROUTING -s 192.168.3.0/24 -o $INET_IFACE -j SNAT
> --to-source $INET_IP

So the new rule will not SNAT anything which already has a source address 
other than 192.168.3.0/24 (such as, for example, $INET_IP).   It seems to me 
that this is probably necessary because of the way your IPsec packets are now 
going twice through the same interface (in the old 2.4 FreeS/WAN 
implementation, they would go once through eth0, and once through ipsec0, but 
I believe this is no longer the case?)

> Is this a bug?

No, I'd say it's a literal interpretation of what the rule says :)

Congratulations on working out what the problem was, though.

Regards,

Antony.

-- 
Anything that improbable is effectively impossible.

 - Murray Gell-Mann, Nobel Prizewinner in Physics

                                                     Please reply to the list;
                                                           please don't CC me.

Re: [solved] Re: Iptables / KAME IPSec problem: source port information lost

[Carsten Maass]

Antony Stone wrote:
> On Saturday 13 March 2004 10:16 pm, Carsten Maass wrote:
> 
> 
>>Even if it's not completely clear to me why: the offending rule was
>>
>>$IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source $INET_IP
>>
>>It seems to do some kind of NATing to all traffic leaving the gateway.
> 
> 
> Er, why shouldn't it do this?   The rule says "for any packet leaving 
> $INET_IFACE, change the Source Address so that it is $INET_IP".
> 
> Therefore all traffic leaving the gateway is going to be SNATted.... no 
> surprise there?

Not all traffic should be SNATed, only traffic leaving via $INET_IFACE. 
Traffic leaving via $LAN_IFACE should not be altered, which the above 
rule nevertheless did.

>>When i narrowed it down to only match traffic which comes out of the LAN
>>it works like a charm:
>>
>>$IPTABLES -t nat -A POSTROUTING -s 192.168.3.0/24 -o $INET_IFACE -j SNAT
>>--to-source $INET_IP
> 
> 
> So the new rule will not SNAT anything which already has a source address 
> other than 192.168.3.0/24 (such as, for example, $INET_IP).   It seems to me 
> that this is probably necessary because of the way your IPsec packets are now 
> going twice through the same interface (in the old 2.4 FreeS/WAN 
> implementation, they would go once through eth0, and once through ipsec0, but 
> I believe this is no longer the case?)

Yes, they are going twice through $INET_IFACE, but the alteration occurs 
when leaving through $LAN_IFACE, which should not be matched as 
SNAT-target by the original rule. And in fact occurs no full SNAT but a 
change of the source port.

>>Is this a bug?
> 
> 
> No, I'd say it's a literal interpretation of what the rule says :)

Maybe at least we both agree that the above behavior is not *intended* 
by the applied rule? :)

Greetings,
Carsten.

Re: [solved] Re: Iptables / KAME IPSec problem: source port information lost

[Antony Stone]

On Sunday 14 March 2004 12:55 am, Carsten Maass wrote:

> Antony Stone wrote:
> > On Saturday 13 March 2004 10:16 pm, Carsten Maass wrote:
> > >
> > > $IPTABLES -t nat -A POSTROUTING -o $INET_IFACE -j SNAT --to-source
> > > $INET_IP
> > >
> > > It seems to do some kind of NATing to all traffic leaving the gateway.
> >
> > Er, why shouldn't it do this?   The rule says "for any packet leaving
> > $INET_IFACE, change the Source Address so that it is $INET_IP".
>
> Not all traffic should be SNATed, only traffic leaving via $INET_IFACE.
> Traffic leaving via $LAN_IFACE should not be altered, which the above
> rule nevertheless did.

Oh, sorry - when you said "leaving the gateway", I kind of assumed you meant 
"going to the outside world" - my mistake.

> > So the new rule will not SNAT anything which already has a source address
> > other than 192.168.3.0/24 (such as, for example, $INET_IP).   It seems to
> > me that this is probably necessary because of the way your IPsec packets
> > are now going twice through the same interface (in the old 2.4 FreeS/WAN
> > implementation, they would go once through eth0, and once through ipsec0,
> > but I believe this is no longer the case?)
>
> Yes, they are going twice through $INET_IFACE, but the alteration occurs
> when leaving through $LAN_IFACE, which should not be matched as
> SNAT-target by the original rule. And in fact occurs no full SNAT but a
> change of the source port.

I wonder if this is a problem the reverse-nat code is having, trying to keep 
track of packets which are going through the same interface twice?   When a 
packet gets natted, netfilter keeps a record of the original IP+port and the 
substituted IP+port, so that when a reply packet comes back the reverse 
operation can be automagically applied.   The code does not change the port 
number unless something is already bound to that port on the natting machine, 
in which case the port number will be changed.

I wonder if the first time the packet goes through, the IP is changed but not 
the port number, however the second time, there's already an entry in the nat 
table, so the port number does get changed, and then when the response is 
seen, the code incorrectly applies the reverse nat to the IP+port 
combination?

Just a theory, but maybe it'll point somebody in a helpful direction when 
investigating this further?   (Maybe even me, when I get round to dealing 
with 2.6 and inbuilt IPsec....)

> Maybe at least we both agree that the above behavior is not *intended*
> by the applied rule? :)

Yes :)

Antony.

-- 
Anyone that's normal doesn't really achieve much.

 - Mark Blair, Australian rocket engineer

                                                     Please reply to the list;
                                                           please don't CC me.