DNAT sporadically doesn't replace destination IP address

DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

* Problem:
The replacing of the destination IP address doesn't occur always.  I've got a PC on which I run a program which retrieves web pages of 30 modems which all have the same IP address.  In order to do this I created the configuration as listed below.  For the program the IP addresses will be 10.9.9.1 to 10.9.9.30.  Those will be send out on a specific VLAN and the IP address will be replaced by 192.168.1.1.  A number of times I get a 'no route to host' for those web pages, this happens sporadically.  If I start looking at the ARP messages, I see that when it goes wrong an ARP message is send out for an 10.9.9.x address which shouldn't be the case.
On an older setup I didn't  had this problem, so I downgraded the kernel version from 2.6.22-14-generic to 2.6.17-12-generic on the setup where I had the problem.  
With this downgraded kernel the problem didn't occur anymore. 
The distro is Ubuntu. An upgrade to Ubuntu 8.04 which has kernel 2.6.24 didn't solve the problem.

* Setup:
- On which I have the problem:
PC -> Vlan enabled ethernet switch -> 30 modems with the same IP address.

- Simplified:
PC -> PC
for 1 port use the configuration as mentioned below. For the other port use 192.168.1.1 as IP address for each VLAN.
Then you can just ping 10.9.9.x to test.
I have not tried this.

* Perl script to generate configure script:
!/usr/bin/perl -w

my $vlan  = 100;
my $modem = 1;
my $subip = 2;
for (my $modem = 1;$modem <= 30;$modem++) {
    print "echo \"------------$vlan--------------------\"\n";
    print "vconfig add eth2 $vlan\n";
    print "ifconfig eth2.$vlan 192.168.1.$subip\n";
    print "route del -net 192.168.1.0/24\n";
    print "route add -host 10.9.9.$modem eth2.$vlan\n";
    print "iptables -t nat -A OUTPUT -o eth2.$vlan -j DNAT --to 192.168.1.1\n";

    print "iptables -t mangle -N VLAN$vlan\n";
    print "iptables -t mangle -F VLAN$vlan\n";
    print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
    print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
    print "ip ro add table $vlan default dev eth2.$vlan\n";
    print "ip ru add fwmark $vlan table $vlan\n";
    print "ip ro fl ca\n";
    $vlan++;
    $subip++;
};

* HW related info as far it is relevant:
10:04.0 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
10:04.1 Ethernet controller: Intel Corporation 82546GB Gigabit Ethernet Controller (rev 03)
3f:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5755 Gigabit Ethernet PCI Express (rev 02)

Kind regards,
Kris

Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
> * Problem:
> The replacing of the destination IP address doesn't occur always.  I've got a PC on which I run a program which retrieves web pages of 30 modems which all have the same IP address.  In order to do this I created the configuration as listed below.  For the program the IP addresses will be 10.9.9.1 to 10.9.9.30.  Those will be send out on a specific VLAN and the IP address will be replaced by 192.168.1.1.  A number of times I get a 'no route to host' for those web pages, this happens sporadically.  If I start looking at the ARP messages, I see that when it goes wrong an ARP message is send out for an 10.9.9.x address which shouldn't be the case.
> On an older setup I didn't  had this problem, so I downgraded the kernel version from 2.6.22-14-generic to 2.6.17-12-generic on the setup where I had the problem.  
> With this downgraded kernel the problem didn't occur anymore. 
> The distro is Ubuntu. An upgrade to Ubuntu 8.04 which has kernel 2.6.24 didn't solve the problem.
> 
> * Setup:
> - On which I have the problem:
> PC -> Vlan enabled ethernet switch -> 30 modems with the same IP address.
> 
> - Simplified:
> PC -> PC
> for 1 port use the configuration as mentioned below. For the other port use 192.168.1.1 as IP address for each VLAN.
> Then you can just ping 10.9.9.x to test.
> I have not tried this.
> 
> * Perl script to generate configure script:
> !/usr/bin/perl -w
> 
> my $vlan  = 100;
> my $modem = 1;
> my $subip = 2;
> for (my $modem = 1;$modem <= 30;$modem++) {
>     print "echo \"------------$vlan--------------------\"\n";
>     print "vconfig add eth2 $vlan\n";
>     print "ifconfig eth2.$vlan 192.168.1.$subip\n";
>     print "route del -net 192.168.1.0/24\n";
>     print "route add -host 10.9.9.$modem eth2.$vlan\n";
>     print "iptables -t nat -A OUTPUT -o eth2.$vlan -j DNAT --to 192.168.1.1\n";
> 
>     print "iptables -t mangle -N VLAN$vlan\n";
>     print "iptables -t mangle -F VLAN$vlan\n";
>     print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
>     print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
>     print "ip ro add table $vlan default dev eth2.$vlan\n";
>     print "ip ru add fwmark $vlan table $vlan\n";
>     print "ip ro fl ca\n";
>     $vlan++;
>     $subip++;
> };
> 

Please try loading ipt_LOG and executing

"echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid"

and see if something shows up in the ringbuffer.

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

>>> On 22/05/2008 at 16:57, in message <483589C0.4080006@trash.net>, Patrick
McHardy <kaber@trash.net> wrote:
> Kris Op de Beeck wrote:
>> * Problem:
>> The replacing of the destination IP address doesn't occur always.  I've got 
> a PC on which I run a program which retrieves web pages of 30 modems which 
> all have the same IP address.  In order to do this I created the 
> configuration as listed below.  For the program the IP addresses will be 
> 10.9.9.1 to 10.9.9.30.  Those will be send out on a specific VLAN and the IP 
> address will be replaced by 192.168.1.1.  A number of times I get a 'no route 
> to host' for those web pages, this happens sporadically.  If I start looking 
> at the ARP messages, I see that when it goes wrong an ARP message is send out 
> for an 10.9.9.x address which shouldn't be the case.
>> On an older setup I didn't  had this problem, so I downgraded the kernel 
> version from 2.6.22-14-generic to 2.6.17-12-generic on the setup where I had the 
> problem.  
>> With this downgraded kernel the problem didn't occur anymore. 
>> The distro is Ubuntu. An upgrade to Ubuntu 8.04 which has kernel 2.6.24 
> didn't solve the problem.
>> 
>> * Setup:
>> - On which I have the problem:
>> PC -> Vlan enabled ethernet switch -> 30 modems with the same IP address.
>> 
>> - Simplified:
>> PC -> PC
>> for 1 port use the configuration as mentioned below. For the other port use 
> 192.168.1.1 as IP address for each VLAN.
>> Then you can just ping 10.9.9.x to test.
>> I have not tried this.
>> 
>> * Perl script to generate configure script:
>> !/usr/bin/perl -w
>> 
>> my $vlan  = 100;
>> my $modem = 1;
>> my $subip = 2;
>> for (my $modem = 1;$modem <= 30;$modem++) {
>>     print "echo \"------------$vlan--------------------\"\n";
>>     print "vconfig add eth2 $vlan\n";
>>     print "ifconfig eth2.$vlan 192.168.1.$subip\n";
>>     print "route del -net 192.168.1.0/24\n";
>>     print "route add -host 10.9.9.$modem eth2.$vlan\n";
>>     print "iptables -t nat -A OUTPUT -o eth2.$vlan -j DNAT --to 192.168.1.1\n";
>> 
>>     print "iptables -t mangle -N VLAN$vlan\n";
>>     print "iptables -t mangle -F VLAN$vlan\n";
>>     print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
>>     print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
>>     print "ip ro add table $vlan default dev eth2.$vlan\n";
>>     print "ip ru add fwmark $vlan table $vlan\n";
>>     print "ip ro fl ca\n";
>>     $vlan++;
>>     $subip++;
>> };
>> 
> 
> Please try loading ipt_LOG and executing
> 
> "echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid"
> 
> and see if something shows up in the ringbuffer.

[  917.584000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.23 DST=10.9.9.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3257 DF PROTO=TCP SPT=40018 DPT=80 SEQ=1136088214 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025B1C0000000001030307) UID=1000
[  918.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26411 DF PROTO=TCP SPT=47713 DPT=80 SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025C4C0000000001030307) UID=1000
[  921.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26412 DF PROTO=TCP SPT=47713 DPT=80 SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025F3A0000000001030307) UID=1000
[  924.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25744 DF PROTO=TCP SPT=52775 DPT=80 SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A000261930000000001030307) UID=1000
[  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A000264810000000001030307) UID=1000

For those DST ip addresses I've got failures

Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
> 
>>>> On 22/05/2008 at 16:57, in message <483589C0.4080006@trash.net>, Patrick
> McHardy <kaber@trash.net> wrote:
>> Please try loading ipt_LOG and executing
>>
>> "echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid"
>>
>> and see if something shows up in the ringbuffer.
> 
> [  917.584000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.23 DST=10.9.9.22 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3257 DF PROTO=TCP SPT=40018 DPT=80 SEQ=1136088214 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025B1C0000000001030307) UID=1000
> [  918.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26411 DF PROTO=TCP SPT=47713 DPT=80 SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025C4C0000000001030307) UID=1000
> [  921.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26412 DF PROTO=TCP SPT=47713 DPT=80 SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00025F3A0000000001030307) UID=1000
> [  924.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25744 DF PROTO=TCP SPT=52775 DPT=80 SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A000261930000000001030307) UID=1000
> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A000264810000000001030307) UID=1000
> 
> For those DST ip addresses I've got failures

Which kernel is this test running on? That message
is gone since 2.6.22.

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

>>> On 22/05/2008 at 17:28, in message <4835913A.4020909@trash.net>, Patrick
McHardy <kaber@trash.net> wrote:
> Kris Op de Beeck wrote:
>> 
>>>>> On 22/05/2008 at 16:57, in message <483589C0.4080006@trash.net>, Patrick
>> McHardy <kaber@trash.net> wrote:
>>> Please try loading ipt_LOG and executing
>>>
>>> "echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid"
>>>
>>> and see if something shows up in the ringbuffer.
>> 
>> [  917.584000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.23 DST=10.9.9.22 
> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3257 DF PROTO=TCP SPT=40018 DPT=80 
> SEQ=1136088214 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A00025B1C0000000001030307) UID=1000
>> [  918.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 
> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26411 DF PROTO=TCP SPT=47713 DPT=80 
> SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A00025C4C0000000001030307) UID=1000
>> [  921.800000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.22 DST=10.9.9.21 
> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26412 DF PROTO=TCP SPT=47713 DPT=80 
> SEQ=1041054567 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A00025F3A0000000001030307) UID=1000
>> [  924.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25744 DF PROTO=TCP SPT=52775 DPT=80 
> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A000261930000000001030307) UID=1000
>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
> (020405B40402080A000264810000000001030307) UID=1000
>> 
>> For those DST ip addresses I've got failures
> 
> Which kernel is this test running on? That message
> is gone since 2.6.22.

s3p@burnin:~$ uname -a
Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux

This was the config where I detected the problem. (Ubuntu 7.10)

Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
>>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
>> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
>> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>> (020405B40402080A000264810000000001030307) UID=1000
>>> For those DST ip addresses I've got failures
>> Which kernel is this test running on? That message
>> is gone since 2.6.22.
> 
> s3p@burnin:~$ uname -a
> Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
> 
> This was the config where I detected the problem. (Ubuntu 7.10)

Odd. I'll have a look at the ubuntu kernel.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
> 
>>>> On 22/05/2008 at 17:28, in message <4835913A.4020909@trash.net>, Patrick
> McHardy <kaber@trash.net> wrote:
>>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
>> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
>> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>> (020405B40402080A000264810000000001030307) UID=1000
>>> For those DST ip addresses I've got failures
>> Which kernel is this test running on? That message
>> is gone since 2.6.22.
> 
> s3p@burnin:~$ uname -a
> Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
> 
> This was the config where I detected the problem. (Ubuntu 7.10)


I didn't find the reason why your kernel even has that message
(didn't try to hard though). Could you rerun the test with a
more current kernel, like 2.6.24 or 2.6.25 please?

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

>>> On 22/05/2008 at 18:29, in message <48359F66.50503@trash.net>, Patrick McHardy
<kaber@trash.net> wrote:
> Kris Op de Beeck wrote:
>> 
>>>>> On 22/05/2008 at 17:28, in message <4835913A.4020909@trash.net>, Patrick
>> McHardy <kaber@trash.net> wrote:
>>>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
> 
>>> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
>>> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>>> (020405B40402080A000264810000000001030307) UID=1000
>>>> For those DST ip addresses I've got failures
>>> Which kernel is this test running on? That message
>>> is gone since 2.6.22.
>> 
>> s3p@burnin:~$ uname -a
>> Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 
> GNU/Linux
>> 
>> This was the config where I detected the problem. (Ubuntu 7.10)
> 
> 
> I didn't find the reason why your kernel even has that message
> (didn't try to hard though). Could you rerun the test with a
> more current kernel, like 2.6.24 or 2.6.25 please?

I will try to and let you know.
Thanks so far for looking into it.

Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

>>> On 22/05/2008 at 18:29, in message <48359F66.50503@trash.net>, Patrick McHardy
<kaber@trash.net> wrote:
> Kris Op de Beeck wrote:
>> 
>>>>> On 22/05/2008 at 17:28, in message <4835913A.4020909@trash.net>, Patrick
>> McHardy <kaber@trash.net> wrote:
>>>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
> 
>>> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
>>> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>>> (020405B40402080A000264810000000001030307) UID=1000
>>>> For those DST ip addresses I've got failures
>>> Which kernel is this test running on? That message
>>> is gone since 2.6.22.
>> 
>> s3p@burnin:~$ uname -a
>> Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 
> GNU/Linux
>> 
>> This was the config where I detected the problem. (Ubuntu 7.10)
> 
> 
> I didn't find the reason why your kernel even has that message
> (didn't try to hard though). Could you rerun the test with a
> more current kernel, like 2.6.24 or 2.6.25 please?

Upgraded from Ubuntu 7.10 to 8.04

s3p@burnin:~$ uname -a
Linux burnin 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux

[ 7651.979390] printk: 8 messages suppressed.
[ 7651.979395] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.24 DST=10.9.9.23 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47660 DF PROTO=TCP SPT=42451 DPT=80 SEQ=3352606539 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A001C104E0000000001030307) UID=1000
[ 7656.667006] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.15 DST=10.9.9.14 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59137 DF PROTO=TCP SPT=49734 DPT=80 SEQ=3327322852 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A001C14E50000000001030307) UID=1000

Nothing works now.  Can't even ping the modems...




Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Ulrik De Bie]

> Kris Op de Beeck wrote:
>> 
>>>>> On 22/05/2008 at 17:28, in message <4835913A.4020909@trash.net>, Patrick McHardy <kaber@trash.net> wrote:
>>>> [  927.204000] nf_ct_tcp: invalid SYNIN= OUT= SRC=192.168.1.30 DST=10.9.9.29 
>>> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=25745 DF PROTO=TCP SPT=52775 DPT=80 
>>> SEQ=2154890499 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT 
>>> (020405B40402080A000264810000000001030307) UID=1000
>>>> For those DST ip addresses I've got failures
>>> Which kernel is this test running on? That message
>>> is gone since 2.6.22.
>> 
>> s3p@burnin:~$ uname -a
>> Linux burnin 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
>> 
>> This was the config where I detected the problem. (Ubuntu 7.10)

>I didn't find the reason why your kernel even has that message
>(didn't try to hard though). Could you rerun the test with a
>more current kernel, like 2.6.24 or 2.6.25 please?
Kris already provided you with the results for 2.6.24, behaviour there seems to have become even worse. 

But about your first remark:
The ubuntu 2.6.22-14 kernel has that message still, it's based on kernel version 2.6.22.9 and I downloaded the original from kernel.org, and that still contains the message:
linux-2.6.22.9/net/netfilter/nf_conntrack_proto_tcp.c at line 909 in function tcp_packet.

I also compared with the 2.6.17.1 (which is definitely older than the ubuntu 2.6.17), and the function tcp_packet has not changed between these versions. In 2.6.25 this function did have some changes though. So the relevant change causing the behaviour must be outside function tcp_packet.

Best regards,
Ulrik De Bie
Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
>>>> On 22/05/2008 at 18:29, in message <48359F66.50503@trash.net>, Patrick McHardy
> <kaber@trash.net> wrote:
 >>
>> I didn't find the reason why your kernel even has that message
>> (didn't try to hard though). Could you rerun the test with a
>> more current kernel, like 2.6.24 or 2.6.25 please?
> 
> Upgraded from Ubuntu 7.10 to 8.04
> 
> s3p@burnin:~$ uname -a
> Linux burnin 2.6.24-16-generic #1 SMP Thu Apr 10 13:23:42 UTC 2008 i686 GNU/Linux
> 
> [ 7651.979390] printk: 8 messages suppressed.
> [ 7651.979395] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.24 DST=10.9.9.23 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47660 DF PROTO=TCP SPT=42451 DPT=80 SEQ=3352606539 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A001C104E0000000001030307) UID=1000
> [ 7656.667006] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.15 DST=10.9.9.14 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59137 DF PROTO=TCP SPT=49734 DPT=80 SEQ=3327322852 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A001C14E50000000001030307) UID=1000

What does "grep <srcport from above> /proc/net/nf_conntrack" show
when the problem occurs?

> Nothing works now.  Can't even ping the modems...

I can't say anything about this without further information.

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

> What does "grep <srcport from above> /proc/net/nf_conntrack" show
> when the problem occurs?
> 
[ 1976.495472] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.29 DST=10.9.9.28 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58096 DF PROTO=TCP SPT=41675 DPT=80 SEQ=3967333855 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00065A1E0000000001030305) UID=1000

 sudo grep 41675 /proc/net/nf_conntrack
ipv4     2 tcp      6 43 SYN_RECV src=192.168.1.29 dst=10.9.9.28 sport=41675 dport=80 packets=1 bytes=60 src=192.168.1.1 dst=192.168.1.29 sport=80 dport=41675 packets=3 bytes=180 mark=0 secmark=0 use=1


Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.

Re: DNAT sporadically doesn't replace destination IP address

[Patrick McHardy]

Kris Op de Beeck wrote:
>> What does "grep <srcport from above> /proc/net/nf_conntrack" show
>> when the problem occurs?
>>
> [ 1976.495472] nf_ct_tcp: invalid packet ignored IN= OUT= SRC=192.168.1.29 DST=10.9.9.28 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=58096 DF PROTO=TCP SPT=41675 DPT=80 SEQ=3967333855 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A00065A1E0000000001030305) UID=1000
> 
>  sudo grep 41675 /proc/net/nf_conntrack
> ipv4     2 tcp      6 43 SYN_RECV src=192.168.1.29 dst=10.9.9.28 sport=41675 dport=80 packets=1 bytes=60 src=192.168.1.1 dst=192.168.1.29 sport=80 dport=41675 packets=3 bytes=180 mark=0 secmark=0 use=1

That looks like the client send a SYN, the server sent three
SYN/ACKs that never reached the client and the client retransmits
its SYN. The SYN should still be NATed, but conntrack thinks
its out of sync because its already in SYN_RECV state, while the
client is apparently still in SYN_SENT state.

Looking back at your first mail:

>     print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
>     print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
>     print "ip ro add table $vlan default dev eth2.$vlan\n";
>     print "ip ru add fwmark $vlan table $vlan\n";

This looks like a chicken-and-egg problem. You mark packets based
on the output device, but use the mark to direct them to the output
device.

I guess if you use the source IP for routing table selection it
will work. Not sure why it works at all currently.

Re: DNAT sporadically doesn't replace destination IP address

[Kris Op de Beeck]

>>> On 27/05/2008 at 16:44, in message <483C1E3F.4030802@trash.net>, Patrick
McHardy <kaber@trash.net> wrote:
>>     print "iptables -t mangle -A VLAN$vlan -j MARK --set-mark $vlan\n";
>>     print "iptables -t mangle -A OUTPUT -o eth2.$vlan -j VLAN$vlan\n";
>>     print "ip ro add table $vlan default dev eth2.$vlan\n";
>>     print "ip ru add fwmark $vlan table $vlan\n";
> 
> This looks like a chicken-and-egg problem. You mark packets based
> on the output device, but use the mark to direct them to the output
> device.
> 
> I guess if you use the source IP for routing table selection it
> will work. Not sure why it works at all currently.

We have found the problem why the IP addresses were not pingable anymore with Ubuntu 8.04.  It seems that with that release the rp_filter is default on.  If we put this off the modems are pingable again.  In the mean time I don't have my setup fully operational anymore. I've got 1 modem connected for the others I'm sending wget's which timeout since no modem is connected.  I nolonger see any tracing with dmesg, I will need to try again with the full setup to be sure but that will take some time.

When changing the configuration like you suggested, you get something like this for every modem:
vconfig add eth1 129
ifconfig eth1.129 192.168.1.31
route del -net 192.168.1.0/24
route add -host 10.9.9.30 eth1.129
iptables -t nat -A OUTPUT -o eth1.129 -j DNAT --to 192.168.1.1
ip ro add table 129 default dev eth1.129
ip rule add from 192.168.1.31 table 129
ip ro fl ca

Then it works also, and rp_filter can be on. This also still needs to be tested with the full setup.

I think the problem is solved. Thanks for that in guiding me.
If I should still encounter any problems I will let you know.

** Still wondering why there wasn't any problem with the old kernel ... maybe I just was lucky ...


Newtec will be present at the following exhibitions :

CommunicAsia, 17-20 June, Singapore, Booth 6B2-01

*** e-mail confidentiality footer ***
This message and any attachments thereto are confidential. They may also
be privileged or otherwise protected by work product immunity or other
legal rules. If you have received it by mistake, please let us know by
e-mail reply and delete it from your system; you may not copy this
message or disclose its contents to anyone.
E-mail transmission cannot be guaranteed to be secure or error free as
information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore is in no
way liable for any errors or omissions in the content of this message,
which may arise as a result of e-mail transmission. If verification is
required, please request a hard copy.