ICMP packets associated with NAT connections sent out wrong interface?

ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Hi,

My machine is functioning as a NAT box. It has two NICs:
- eth0, connected to the LAN, IP address 192.168.0.1
- eth1, connected to the Internet, IP address 123.23.23.23

In the OUTPUT chain, I accept packets sent out eth0 with a destination
address of 192.168.0.x. Any packets sent out other interfaces with a
destination address of 192.168.0.x are logged and dropped:

-A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
-A OUTPUT -d 192.168.0.0/24 -j DROP

In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
anything, just as expected.

With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
TYPE=3 CODE=3" packets going out eth1. Example:

[outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.4
DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]

What the logged packets have in common:

1. The second SRC= is always a machine on the LAN (192.168.0.x).
   (Note: The machines aren't doing anything 'weird', just normal web
   browsing.)
2. The second DST= is always the address of the Internet interface
   (123.23.23.23). (This doesn't seem right as no machines on the LAN
   ever connect to that address directly?)
3. DPT is always within the range of ports I have allocated for NAT use
   (I use "-p tcp --to-ports 41800-59999" in my MASQUERADE rule).

Any ideas as to what's causing packets destined for the LAN to be sent
out the Internet interface, and only in 2.6.20+? Does the
"DST=123.23.23.23" indicate a failure to perform address translation?
(Could a conntrack entry be expiring prematurely?)

Thanks.
-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Martijn Lievaart]

Jordan Russell wrote:
> Hi,
>
> My machine is functioning as a NAT box. It has two NICs:
> - eth0, connected to the LAN, IP address 192.168.0.1
> - eth1, connected to the Internet, IP address 123.23.23.23
>
> In the OUTPUT chain, I accept packets sent out eth0 with a destination
> address of 192.168.0.x. Any packets sent out other interfaces with a
> destination address of 192.168.0.x are logged and dropped:
>
> -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> -A OUTPUT -d 192.168.0.0/24 -j DROP
>
> In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> anything, just as expected.
>
> With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> TYPE=3 CODE=3" packets going out eth1. Example:
>
> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.4
> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
>   

That's a port unreachable message, seemingly coming from the outside, in 
response to a RST coming from the inside. That in itself is not very 
usual, but it can happen.

This packet going out the wrong interface seems like a pretty serious 
bug. I advice you to repost on the netfilter-devel list, there are many 
more people there that are really into this. However, this may be a bug 
in the network code, not the netfilter code. Or something else completely.


HTH,
M4

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Ray Leach]

On Wed, 2007-06-27 at 00:22 +0200, Martijn Lievaart wrote:
> Jordan Russell wrote:
> > Hi,
> >
> > My machine is functioning as a NAT box. It has two NICs:
> > - eth0, connected to the LAN, IP address 192.168.0.1
> > - eth1, connected to the Internet, IP address 123.23.23.23
> >
> > In the OUTPUT chain, I accept packets sent out eth0 with a destination
> > address of 192.168.0.x. Any packets sent out other interfaces with a
> > destination address of 192.168.0.x are logged and dropped:
> >
> > -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> > -A OUTPUT -d 192.168.0.0/24 -j DROP
> >
> > In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> > anything, just as expected.
> >
> > With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> > TYPE=3 CODE=3" packets going out eth1. Example:
> >
> > [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> > PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.4
> > DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> > SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
> >   
> 
> That's a port unreachable message, seemingly coming from the outside, in 
> response to a RST coming from the inside. That in itself is not very 
> usual, but it can happen.
> 
> This packet going out the wrong interface seems like a pretty serious 
> bug. I advice you to repost on the netfilter-devel list, there are many 
> more people there that are really into this. However, this may be a bug 
> in the network code, not the netfilter code. Or something else completely.
> 
> 
> HTH,
> M4
> 
Doesn't seem like a bug ...

The port unreachable ICMP replay is coming from your machine in response
to a packet coming in eth1 trying to get to 192.168.0.x ...
You have just denied the echo replies by youe DROP rule.

-- 
--------------------------------------------------
Raymond Leach (spoons@rchq.co.za)
RCHQ Hobbies cc       Web: http://www.rchq.co.za/
Tel: +27 82 575 6975  Fax: +27 86 652 2773
"No matter where you go, there you are ..."
--------------------------------------------------

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Ray Leach wrote:
> The port unreachable ICMP replay is coming from your machine in response
> to a packet coming in eth1 trying to get to 192.168.0.x ...

I don't think that's correct. Take a look at the square-bracketed part
of the log entry, which is (as I understand it) the packet that caused
the ICMP response to be generated. 192.168.0.4 was trying to send a TCP
packet *out*. The kernel then (for reasons unclear) tried to send an
ICMP message back to 192.168.0.4, but sent it out the wrong interface,
eth1, thus tripping my rule.

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Martijn Lievaart]

Ray Leach wrote:
> On Wed, 2007-06-27 at 00:22 +0200, Martijn Lievaart wrote:
>   
>> Jordan Russell wrote:
>>     
>>> Hi,
>>>
>>> My machine is functioning as a NAT box. It has two NICs:
>>> - eth0, connected to the LAN, IP address 192.168.0.1
>>> - eth1, connected to the Internet, IP address 123.23.23.23
>>>
>>>       

(snip)

>>> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
>>> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.4
>>> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
>>> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
>>>   
>>>       
>> That's a port unreachable message, seemingly coming from the outside, in 
>> response to a RST coming from the inside. That in itself is not very 
>> usual, but it can happen.
>>
>> This packet going out the wrong interface seems like a pretty serious 
>> bug. I advice you to repost on the netfilter-devel list, there are many 
>> more people there that are really into this. However, this may be a bug 
>> in the network code, not the netfilter code. Or something else completely.
>>
>>
>> HTH,
>> M4
>>
>>     
> Doesn't seem like a bug ...
>
> The port unreachable ICMP replay is coming from your machine in response
> to a packet coming in eth1 trying to get to 192.168.0.x ...
> You have just denied the echo replies by youe DROP rule.
>   

The source of the packet is 123.23.23.23, it comes from the Internet. 
The destination is local. Why do you think this packet is generated locally?

M4

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Martijn Lievaart wrote:
> The source of the packet is 123.23.23.23, it comes from the Internet. 
> The destination is local. Why do you think this packet is generated locally?

123.23.23.23 is the address of the machine's eth1 (Internet) interface.

Since this only happens when I boot into certain kernel versions (2.6.20
and 2.6.21), it seems unlikely that someone on the Internet is sending
me ICMP packets with a spoofed source address.

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Martijn Lievaart]

Jordan Russell wrote:
> Martijn Lievaart wrote:
>   
>> The source of the packet is 123.23.23.23, it comes from the Internet. 
>> The destination is local. Why do you think this packet is generated locally?
>>     
>
> 123.23.23.23 is the address of the machine's eth1 (Internet) interface.
>
> Since this only happens when I boot into certain kernel versions (2.6.20
> and 2.6.21), it seems unlikely that someone on the Internet is sending
> me ICMP packets with a spoofed source address.
>   

My bad. But still seems like a bug somewhere.

M4

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Jordan Russell <jr-list-2007@quo.to>
Date: Sat, 16 Jun 2007 11:43:06 -0500

> Hi,
> 
> My machine is functioning as a NAT box. It has two NICs:
> - eth0, connected to the LAN, IP address 192.168.0.1
> - eth1, connected to the Internet, IP address 123.23.23.23
> 
> In the OUTPUT chain, I accept packets sent out eth0 with a destination
> address of 192.168.0.x. Any packets sent out other interfaces with a
> destination address of 192.168.0.x are logged and dropped:
> 
> -A OUTPUT -d 192.168.0.0/24 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/24 -j LOG '[outdrop] '
> -A OUTPUT -d 192.168.0.0/24 -j DROP
> 
> In kernel 2.6.19 and earlier, the LOG & DROP rules never matched
> anything, just as expected.
> 
> With 2.6.20.12 and 2.6.21.5, however, they occasionally catch "ICMP
> TYPE=3 CODE=3" packets going out eth1. Example:
> 
> [outdrop] IN= OUT=eth1 SRC=123.23.23.23 DST=192.168.0.4 LEN=68 TOS=0x00
> PREC=0xC0 TTL=64 ID=61136 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.4
> DST=123.23.23.23 LEN=40 TOS=0x00 PREC=0x20 TTL=53 ID=16088 PROTO=TCP
> SPT=1229 DPT=44851 WINDOW=0 RES=0x00 ACK RST FIN URGP=0 ]
> 
> What the logged packets have in common:
> 
> 1. The second SRC= is always a machine on the LAN (192.168.0.x).
>    (Note: The machines aren't doing anything 'weird', just normal web
>    browsing.)
> 2. The second DST= is always the address of the Internet interface
>    (123.23.23.23). (This doesn't seem right as no machines on the LAN
>    ever connect to that address directly?)

What is real TCP packet on wire ? Is it really from 192.168.0.4 to
123.23.23.23 ? If there is bug in kernel, we cannot believe
output of LOG because NAT usually mangles addresses and ports in ICMP body.

> 3. DPT is always within the range of ports I have allocated for NAT use
>    (I use "-p tcp --to-ports 41800-59999" in my MASQUERADE rule).
> 
> Any ideas as to what's causing packets destined for the LAN to be sent
> out the Internet interface, and only in 2.6.20+? Does the
> "DST=123.23.23.23" indicate a failure to perform address translation?
> (Could a conntrack entry be expiring prematurely?)

What is your kernel configration and other nat rules ? And please try to
'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
packet is logged or not.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

[re-sending without the attachment]

Yasuyuki KOZAKAI wrote:
> What is real TCP packet on wire ? Is it really from 192.168.0.4 to
> 123.23.23.23 ? If there is bug in kernel, we cannot believe
> output of LOG because NAT usually mangles addresses and ports in ICMP body.

For this logged packet:

Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]

the real packet on eth1 according to tcpdump seems to be:

14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

> What is your kernel configration

Full kernel config here: http://76.187.96.234/config-2.6.21.5
All settings under /proc/sys/net/netfilter are default.

> and other nat rules ?

I can see the logged packets with just the following rules:

*nat
:PREROUTING ACCEPT [32:2910]
:POSTROUTING ACCEPT [29:2330]
:OUTPUT ACCEPT [2:152]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j DROP
-A FORWARD -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
wrong interface] "
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -j DROP
COMMIT

And I've found that the problem occurs with DNAT connections too.

Steps to reproduce:

1. On the server, iptables-restore the above rules.
2. On 192.168.0.133, install BitTorrent client (e.g. Azureus) with TCP
and UDP ports set to 25000.
3. Download
<http://torrent.fedoraproject.org/torrents/Fedora-7-i386.torrent> using
the BitTorrent client.

Within 5-20 minutes, packets begin matching the LOG rule:

Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
ID=32353 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
Jul  4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39699 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39700 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
ID=21127 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
WINDOW=0 RES=0x00 RST URGP=0 ]

> And please try to
> 'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
> packet is logged or not.

When I do that, some "bad HW ICMP checksum" messages are logged, e.g.:

Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
LEN=71 ]

but the times don't coincide with the "[packet out wrong interface]" LOG
messages.

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

Hi,

Cc: netfilter-devel (He said kernel <= 2.6.19 is OK, but problem on 2.6.20.5)

From: Jordan Russell <jr-list-2007@quo.to>
Date: Wed, 04 Jul 2007 16:49:34 -0500

> Yasuyuki KOZAKAI wrote:
> > What is real TCP packet on wire ? Is it really from 192.168.0.4 to
> > 123.23.23.23 ? If there is bug in kernel, we cannot believe
> > output of LOG because NAT usually mangles addresses and ports in ICMP body.
> 
> For this logged packet:
> 
> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> 
> the real packet on eth1 according to tcpdump seems to be:
> 
> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
marked 'correct' ?

> > What is your kernel configration
> 
> Full kernel config attached.
> All settings under /proc/sys/net/netfilter are default.

Sorry for ambiguous, I asked .config for building kernel.
Anyway I've got that you are using nf_conntrack.


> > and other nat rules ?
> 
> I can see the logged packets with just the following rules:
> 
> *nat
> :PREROUTING ACCEPT [32:2910]
> :POSTROUTING ACCEPT [29:2330]
> :OUTPUT ACCEPT [2:152]
> -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
> -A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
> -A POSTROUTING -o eth1 -j MASQUERADE
> COMMIT
> *filter
> :INPUT DROP [0:0]
> :FORWARD DROP [0:0]
> :OUTPUT DROP [0:0]
> -A INPUT -i lo -j ACCEPT
> -A INPUT -i eth0 -j ACCEPT
> -A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
> -A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
> -A INPUT -j DROP
> -A FORWARD -j ACCEPT
> -A OUTPUT -o lo -j ACCEPT
> -A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
> -A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
> wrong interface] "
> -A OUTPUT -o eth1 -j ACCEPT
> -A OUTPUT -j DROP
> COMMIT
> 
> And I've found that the problem occurs with DNAT connections too.
> 
> Steps to reproduce:
> 
> 1. On the server, iptables-restore the above rules.
> 2. On 192.168.0.133, install BitTorrent client (e.g. Azureus) with TCP
> and UDP ports set to 25000.
> 3. Download
> <http://torrent.fedoraproject.org/torrents/Fedora-7-i386.torrent> using
> the BitTorrent client.
> 
> Within 5-20 minutes, packets begin matching the LOG rule:
> 
> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
> ID=32353 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
> DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
> Jul  4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39699 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39700 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
> ID=21127 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=0 RES=0x00 RST URGP=0 ]
> 
> > And please try to
> > 'echo 1 > /proc/sys/net/nf_conntrack_log_invalid' and see whether any
> > packet is logged or not.
> 
> When I do that, some "bad HW ICMP checksum" messages are logged, e.g.:

OK, then might be kernel bug in netfilter, bug in IP stack, or broken
hardware, of cause, if the checksum of ICMP packet is really correct.

workaround fix is disable hardware checksum offload if you use it.

developpers, anyone has idea ? I have not deeply checked codes yet,
but I suspect following.

    commit 43bc0ca7eadc024e9e5b935fa5e0892df4fec9eb
    Author: Al Viro <viro@zeniv.linux.org.uk>
    Date:   Tue Nov 14 21:43:23 2006 -0800

    [NET]: netfilter checksum annotations

    Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: David S. Miller <davem@davemloft.net>


> Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> LEN=71 ]

This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
logged ?


> but the times don't coincide with the "[packet out wrong interface]" LOG
> messages.

This seems other problem, but indeed strange..

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Thu, 05 Jul 2007 10:11:55 +0900 (JST)

> Cc: netfilter-devel (He said kernel <= 2.6.19 is OK, but problem on
> 2.6.20.5)

Sorry, not 2.6.20.5, but 2.6.20.12 and 2.6.21.5.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Yasuyuki KOZAKAI wrote:
>> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
>> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
>> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
>> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
>> WINDOW=64172 RES=0x00 RST URGP=0 ]
>>
>> the real packet on eth1 according to tcpdump seems to be:
>>
>> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
>> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
>> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172
>
> Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
> marked 'correct' ?

The logged ICMP packet doesn't seem to show up in the tcpdump output.
When I grep for the ID 39698 there are no matches at 14:54:xx.  (??)

BTW: does the LOG output indicate that netfilter translated the source
address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
instead translated the *destination* address of 123.23.23.23 (=eth1) to
192.168.0.133? Could this be why the ICMP packet was generated in the
first place?

> workaround fix is disable hardware checksum offload if you use it.

eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
appear to support hardware checksums. dmesg says:

  0000:01:0c.0: scatter/gather disabled. h/w checksums disabled

>> Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
>> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
>> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
>> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
>> LEN=71 ]
> 
> This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
> logged ?

Yes, there are TYPE=3 CODE=3 too. Here's a log snippet showing the "bad
HW ICMP checksum" messages together with the messages from my LOG rule:

...
Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=102
ID=23775 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
LEN=91 TOS=0x00 PREC=0x00 TTL=115 ID=41422 PROTO=UDP SPT=25000 DPT=21227
LEN=71 ]
Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=98 TOS=0x00 PREC=0x20 TTL=102
ID=23783 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
LEN=70 TOS=0x00 PREC=0x00 TTL=115 ID=41504 PROTO=UDP SPT=25000 DPT=21227
LEN=50 ]
Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
ID=32353 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
LEN=71 ]
Jul  4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39699 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39700 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
ID=21127 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
WINDOW=0 RES=0x00 RST URGP=0 ]
...

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Yasuyuki KOZAKAI wrote:
>> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
>> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
>> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
>> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
>> WINDOW=64172 RES=0x00 RST URGP=0 ]
>>
>> the real packet on eth1 according to tcpdump seems to be:
>>
>> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
>> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
>> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172
>
> Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
> marked 'correct' ?

The logged ICMP packet doesn't seem to show up in the tcpdump output.
When I grep for the ID 39698 there are no matches at 14:54:xx.  (??)

BTW: does the LOG output indicate that netfilter translated the source
address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
instead translated the *destination* address of 123.23.23.23 (=eth1) to
192.168.0.133? Could this be why the ICMP packet was generated in the
first place?

> workaround fix is disable hardware checksum offload if you use it.

eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
appear to support hardware checksums. dmesg says:

  0000:01:0c.0: scatter/gather disabled. h/w checksums disabled

>> Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
>> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
>> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
>> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
>> LEN=71 ]
> 
> This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
> logged ?

Yes, there are TYPE=3 CODE=3 too. Here's a log snippet showing the "bad
HW ICMP checksum" messages together with the messages from my LOG rule:

...
Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=102
ID=23775 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
LEN=91 TOS=0x00 PREC=0x00 TTL=115 ID=41422 PROTO=UDP SPT=25000 DPT=21227
LEN=71 ]
Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.203.45.12 DST=123.23.23.23 LEN=98 TOS=0x00 PREC=0x20 TTL=102
ID=23783 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
LEN=70 TOS=0x00 PREC=0x00 TTL=115 ID=41504 PROTO=UDP SPT=25000 DPT=21227
LEN=50 ]
Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
ID=32353 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
LEN=71 ]
Jul  4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39699 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39700 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]
Jul  4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
ID=21127 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
WINDOW=0 RES=0x00 RST URGP=0 ]
...

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Jordan Russell <jr-list-2007@quo.to>
Date: Thu, 05 Jul 2007 00:51:05 -0500

> Yasuyuki KOZAKAI wrote:
> >> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> >> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> >> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> >> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> >> WINDOW=64172 RES=0x00 RST URGP=0 ]
> >>
> >> the real packet on eth1 according to tcpdump seems to be:
> >>
> >> 14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
> >> proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
> >> cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172
> >
> > Thanks, I want to see dump of real ICMP packet. 'cksum' of ICMP packet is
> > marked 'correct' ?
> 
> The logged ICMP packet doesn't seem to show up in the tcpdump output.
> When I grep for the ID 39698 there are no matches at 14:54:xx.  (??)

Indeed it seems to be generated by local node and dropped.


> BTW: does the LOG output indicate that netfilter translated the source
> address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> instead translated the *destination* address of 123.23.23.23 (=eth1) to
> 192.168.0.133? Could this be why the ICMP packet was generated in the
> first place?

Hmmm, REJECT in your rule might generate it, but I'm not sure.


> > workaround fix is disable hardware checksum offload if you use it.
> 
> eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
> appear to support hardware checksums. dmesg says:
> 
>   0000:01:0c.0: scatter/gather disabled. h/w checksums disabled

Then it's purely kernel bug, I think.


> >> Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> >> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> >> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
> >> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> >> LEN=71 ]
> > 
> > This is ICMP error for UDP pakcet. ICMP packets TYPE=3 and CODE=3 were
> > logged ?
> 
> Yes, there are TYPE=3 CODE=3 too. Here's a log snippet showing the "bad
> HW ICMP checksum" messages together with the messages from my LOG rule:
> 
> ...
> Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.203.45.12 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=102
> ID=23775 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
> LEN=91 TOS=0x00 PREC=0x00 TTL=115 ID=41422 PROTO=UDP SPT=25000 DPT=21227
> LEN=71 ]
> Jul  4 14:53:57 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.203.45.12 DST=123.23.23.23 LEN=98 TOS=0x00 PREC=0x20 TTL=102
> ID=23783 PROTO=ICMP TYPE=3 CODE=3 [SRC=123.23.23.23 DST=80.203.45.12
> LEN=70 TOS=0x00 PREC=0x00 TTL=115 ID=41504 PROTO=UDP SPT=25000 DPT=21227
> LEN=50 ]
> Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 14:58:04 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=92 TOS=0x00 PREC=0xC0 TTL=64
> ID=32353 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=64 TOS=0x00 PREC=0x20 TTL=38 ID=47850 DF PROTO=TCP SPT=25000
> DPT=25000 WINDOW=63 RES=0x00 ACK URGP=0 ]
> Jul  4 14:58:39 webby kernel: nf_ct_icmp: bad HW ICMP checksum IN= OUT=
> SRC=80.133.170.211 DST=123.23.23.23 LEN=119 TOS=0x00 PREC=0x20 TTL=234
> ID=22079 PROTO=ICMP TYPE=3 CODE=1 [SRC=123.23.23.23 DST=80.133.170.211
> LEN=91 TOS=0x00 PREC=0x00 TTL=114 ID=25502 PROTO=UDP SPT=25000 DPT=21519
> LEN=71 ]
> Jul  4 15:01:06 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39699 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39688 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 15:09:18 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> ID=39700 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=40226 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=64172 RES=0x00 RST URGP=0 ]
> Jul  4 15:11:10 webby kernel: [packet out wrong interface] IN= OUT=eth1
> SRC=123.23.23.23 DST=192.168.0.133 LEN=71 TOS=0x00 PREC=0xC0 TTL=64
> ID=21127 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> LEN=43 TOS=0x00 PREC=0x20 TTL=17 ID=0 PROTO=TCP SPT=25000 DPT=25000
> WINDOW=0 RES=0x00 RST URGP=0 ]
> ...

The issue of UDP and TCP seems to be different.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Patrick McHardy]

Yasuyuki KOZAKAI wrote:
> From: Jordan Russell <jr-list-2007@quo.to>
> Date: Thu, 05 Jul 2007 00:51:05 -0500
> 
> 
>>Yasuyuki KOZAKAI wrote:
>>
>>>>Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
>>>>SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
>>>>ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
>>>>LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
>>>>WINDOW=64172 RES=0x00 RST URGP=0 ]
>>>>
>>BTW: does the LOG output indicate that netfilter translated the source
>>address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
>>instead translated the *destination* address of 123.23.23.23 (=eth1) to
>>192.168.0.133? Could this be why the ICMP packet was generated in the
>>first place?

> Hmmm, REJECT in your rule might generate it, but I'm not sure.


Its pretty certain the REJECT target, it defauls to port unreachable
and the network stack doesn't generate port unreachables for TCP.
Jordan, please post your ruleset.

>>>workaround fix is disable hardware checksum offload if you use it.
>>
>>eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
>>appear to support hardware checksums. dmesg says:
>>
>>  0000:01:0c.0: scatter/gather disabled. h/w checksums disabled


I can't find this message in the kernel tree. Which driver are you
using?

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Krzysztof Oledzki]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 560 bytes --]



On Thu, 5 Jul 2007, Patrick McHardy wrote:
<CUT>
>>>> workaround fix is disable hardware checksum offload if you use it.
>>>
>>> eth1 is running off a 10-year-old 3Com 3C905 adapter, which doesn't
>>> appear to support hardware checksums. dmesg says:
>>>
>>>  0000:01:0c.0: scatter/gather disabled. h/w checksums disabled
>
>
> I can't find this message in the kernel tree. Which driver are you
> using?

3c59x.c:                printk(KERN_INFO "%s: scatter/gather %sabled. h/w checksums %sabled\n",

Best reragds,

 				Krzysztof Olędzki

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Patrick McHardy wrote:
> Its pretty certain the REJECT target, it defauls to port unreachable
> and the network stack doesn't generate port unreachables for TCP.
> Jordan, please post your ruleset.

Yes, I have a REJECT rule for non-ESTABLISHED incoming packets on eth1.
Please see this post for rules sufficient to reproducing the issue:

http://lists.netfilter.org/pipermail/netfilter/2007-July/069182.html

>>>  0000:01:0c.0: scatter/gather disabled. h/w checksums disabled
>
>
> I can't find this message in the kernel tree. Which driver are you
> using?

It's 3c59x.

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Thu, 05 Jul 2007 14:21:52 +0200

> Yasuyuki KOZAKAI wrote:
> > From: Jordan Russell <jr-list-2007@quo.to>
> > Date: Thu, 05 Jul 2007 00:51:05 -0500
> > 
> > 
> >>Yasuyuki KOZAKAI wrote:
> >>
> >>>>Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
> >>>>SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
> >>>>ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
> >>>>LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
> >>>>WINDOW=64172 RES=0x00 RST URGP=0 ]
> >>>>
> >>BTW: does the LOG output indicate that netfilter translated the source
> >>address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> >>instead translated the *destination* address of 123.23.23.23 (=eth1) to
> >>192.168.0.133? Could this be why the ICMP packet was generated in the
> >>first place?
> 
> > Hmmm, REJECT in your rule might generate it, but I'm not sure.
> 
> 
> Its pretty certain the REJECT target, it defauls to port unreachable
> and the network stack doesn't generate port unreachables for TCP.
> Jordan, please post your ruleset.

He has already posted it. REJECT is in INPUT chain. This means TCP packet
was not mangled. TCP packet might be handled as error, but there seemed to
be no log "nf_ct_tcp: ...".

Jordan, is there the message "nf_conntrack: table full, dropping packet"
in your log ? I've heard that BitTorrent creates huge connections.

> > *nat
> > :PREROUTING ACCEPT [32:2910]
> > :POSTROUTING ACCEPT [29:2330]
> > :OUTPUT ACCEPT [2:152]
> > -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> > --to-destination 192.168.0.133
> > -A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
> > --to-destination 192.168.0.133
> > -A POSTROUTING -o eth1 -j MASQUERADE
> > COMMIT
> > *filter
> > :INPUT DROP [0:0]
> > :FORWARD DROP [0:0]
> > :OUTPUT DROP [0:0]
> > -A INPUT -i lo -j ACCEPT
> > -A INPUT -i eth0 -j ACCEPT
> > -A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
> > -A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
> > -A INPUT -j DROP
> > -A FORWARD -j ACCEPT
> > -A OUTPUT -o lo -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
> > -A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out
> > wrong interface] "
> > -A OUTPUT -o eth1 -j ACCEPT
> > -A OUTPUT -j DROP
> > COMMIT

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Yasuyuki KOZAKAI wrote:
> Jordan, is there the message "nf_conntrack: table full, dropping packet"
> in your log ? I've heard that BitTorrent creates huge connections.

No; there are no messages from the kernel other than "nf_ct_icmp: bad HW
ICMP checksum" and "[packet out wrong interface]" (my LOG rule).

Note that my LOG rule isn't only triggered when using BitTorrent. It's
hit during normal web browsing from the client machines too (i.e.
outbound port 80 through my MASQUERADE rule), just not nearly as often.

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Jordan Russell wrote:
> BTW: does the LOG output indicate that netfilter translated the source
> address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> instead translated the *destination* address of 123.23.23.23 (=eth1) to
> 192.168.0.133? Could this be why the ICMP packet was generated in the
> first place?

To clarify my question:

If tcpdump on eth1 reports:

  70.243.226.250.1703 > 123.23.23.23.25000

while my LOG rule reports for the same packet:

  ... [SRC=192.168.0.133 DST=123.23.23.23 ... SPT=25000 DPT=25000

isn't this saying that netfilter translated the *source* address of the
packet?

Since port 25000 is covered by a DNAT rule:

-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133

shouldn't it have set the *destination* address of the packet to
192.168.0.133, while leaving the source address unchanged?

So: It appears as though netfilter is (in rare cases) translating the
source address of packets when it should be translating the destination
address. Or am I misinterpreting the log output?

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Jordan Russell wrote:
> BTW: does the LOG output indicate that netfilter translated the source
> address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> instead translated the *destination* address of 123.23.23.23 (=eth1) to
> 192.168.0.133? Could this be why the ICMP packet was generated in the
> first place?

To clarify my question:

If tcpdump on eth1 reports:

  70.243.226.250.1703 > 123.23.23.23.25000

while my LOG rule reports for the same packet:

  ... [SRC=192.168.0.133 DST=123.23.23.23 ... SPT=25000 DPT=25000

isn't this saying that netfilter translated the *source* address of the
packet?

Since port 25000 is covered by a DNAT rule:

-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133

shouldn't it have set the *destination* address of the packet to
192.168.0.133, while leaving the source address unchanged?

So: It appears as though netfilter is (in rare cases) translating the
source address of packets when it should be translating the destination
address. Or am I misinterpreting the log output?

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

Hi, Jordan,

I'll sort out observations from your report and try to explain my idea in
other mail after finishing some tests.


From: Jordan Russell <jr-list-2007@quo.to>
Date: Fri, 06 Jul 2007 12:42:24 -0500

> Jordan Russell wrote:
> > BTW: does the LOG output indicate that netfilter translated the source
> > address of 70.243.226.250 to 192.168.0.133? If so, shouldn't it have
> > instead translated the *destination* address of 123.23.23.23 (=eth1) to
> > 192.168.0.133? Could this be why the ICMP packet was generated in the
> > first place?
> 
> To clarify my question:
> 
> If tcpdump on eth1 reports:
> 
>   70.243.226.250.1703 > 123.23.23.23.25000
> 
> while my LOG rule reports for the same packet:
> 
>   ... [SRC=192.168.0.133 DST=123.23.23.23 ... SPT=25000 DPT=25000
> 
> isn't this saying that netfilter translated the *source* address of the
> packet?

Yes, but from this log, we can say only that source address of TCP packet
was mangled, OR the source address of IP header in ICMP body was mangled.

Inserting LOG rule just before REJECT rule will clarify that.


> Since port 25000 is covered by a DNAT rule:
> 
> -A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
> --to-destination 192.168.0.133
> 
> shouldn't it have set the *destination* address of the packet to
> 192.168.0.133, while leaving the source address unchanged?

If TCP packet is valid, yes. If not, no address is mangled.

> So: It appears as though netfilter is (in rare cases) translating the
> source address of packets when it should be translating the destination
> address. Or am I misinterpreting the log output?

The current NAT code does that, but I suspect that it is due to bug.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

[-- Attachment #1: Type: Text/Plain, Size: 3933 bytes --]


From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Sat, 07 Jul 2007 15:27:06 +0900 (JST)

> Hi, Jordan,
> 
> I'll sort out observations from your report and try to explain my idea in
> other mail after finishing some tests.

OK, I try to explain what the current netfilter does in Jordan's situation.
I also attach the patch to fix this problem. Jordan, can you try it ?

Patrick, please consider to apply it. Even if following my idea is not
correct, clearly it fixes a bug. I'm wondering why I missed '!'
when I copied & pasted ip_conntrack_proto_icmp.c...


- This situation occurred on 2.6.20.12 and 2.6.21.5 at least.
- your rules are following

*nat
:PREROUTING ACCEPT [32:2910]
:POSTROUTING ACCEPT [29:2330]
:OUTPUT ACCEPT [2:152]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j DROP
-A FORWARD -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out wrong interface] "
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -j DROP
COMMIT

- ICMP error is generated by this router but destination address of it is strange.

Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]

- original TCP reset is following

14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

- no error message is printed by nf_ct_tcp, no 'table is full' error.


My idea is following.

- This TCP reset is not initial packet of TCP connection. If it is initial packet,
  no address in ICMP packet should be mangled. Jordan, if you see
  /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.

- TCP packet was marked as error packet. Because '--state ESTABLISHED'
  didn't match the packet. No conntrack entry wasn't assigned to the
  packet. Usually, error log by nf_conntrack_tcp should be generated in
  such case, but no message is generated in some cases. I don't know why
  this TCP reset was handled as error.

- Then ICMP error generated at this router was not assigned to any conntrack entry.

- nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
  the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.

        h = nf_conntrack_find_get(&innertuple, NULL);
        if (!h) {
                /* Locally generated ICMPs will match inverted if they
                   haven't been SNAT'ed yet */
                /* FIXME: NAT code has to handle half-done double NAT --RR  */
                if (hooknum == NF_IP_LOCAL_OUT)
                        h = nf_conntrack_find_get(&origtuple, NULL);

                if (!h) {
                        DEBUGP("icmp_error_message: no match\n");
                        return -NF_ACCEPT;
                }

                /* Reverse direction from that found */
                if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
                        *ctinfo += IP_CT_IS_REPLY;
        } else {
                if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
                        *ctinfo += IP_CT_IS_REPLY;


- As a result, nf_nat_packet mistook the address in ICMP packet which
  should be mangled.

[-- Attachment #2: 0001-NETFILTER-nf_conntrack-Fixes-direction-of-locall.patch --]
[-- Type: Text/Plain, Size: 1532 bytes --]

From e36c67a5f57a0bd45f6666627ad3d60d42ee4497 Mon Sep 17 00:00:00 2001
From: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
Date: Sat, 7 Jul 2007 20:29:39 +0900
Subject: [PATCH 1/1] [NETFILTER]: nf_conntrack: Fixes direction of locally generated ICMP error

The conntrack assigned to locally generated ICMP error is usually the one
assigned to the original packet which has caused the error. But if
the original packet is handled as invalid by nf_conntrack, no conntrack
is assigned to the original packet. Then nf_ct_attach() cannot assign
any conntrack to the ICMP error packet. In that case nf_conntrack_icmp
tries to assign appropriate conntrack to it. But the current code
mistakes the direction of the packet. As a result, NAT code mistakes
the address in the packet to mangle.

Spotted by Jordan Russell.

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
---
 net/ipv4/netfilter/nf_conntrack_proto_icmp.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
index f4fc657..5973b58 100644
--- a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
+++ b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
@@ -201,7 +201,7 @@ icmp_error_message(struct sk_buff *skb,
 		}
 
 		/* Reverse direction from that found */
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
+		if (NF_CT_DIRECTION(h) != IP_CT_DIR_REPLY)
 			*ctinfo += IP_CT_IS_REPLY;
 	} else {
 		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-- 
1.5.2.2

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

[-- Attachment #1: Type: Text/Plain, Size: 3933 bytes --]


From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Sat, 07 Jul 2007 15:27:06 +0900 (JST)

> Hi, Jordan,
> 
> I'll sort out observations from your report and try to explain my idea in
> other mail after finishing some tests.

OK, I try to explain what the current netfilter does in Jordan's situation.
I also attach the patch to fix this problem. Jordan, can you try it ?

Patrick, please consider to apply it. Even if following my idea is not
correct, clearly it fixes a bug. I'm wondering why I missed '!'
when I copied & pasted ip_conntrack_proto_icmp.c...


- This situation occurred on 2.6.20.12 and 2.6.21.5 at least.
- your rules are following

*nat
:PREROUTING ACCEPT [32:2910]
:POSTROUTING ACCEPT [29:2330]
:OUTPUT ACCEPT [2:152]
-A PREROUTING -i eth1 -p tcp -m tcp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A PREROUTING -i eth1 -p udp -m udp --dport 25000 -j DNAT
--to-destination 192.168.0.133
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -i eth1 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -j DROP
-A FORWARD -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -o eth0 -j ACCEPT
-A OUTPUT -d 192.168.0.0/255.255.255.0 -j LOG --log-prefix "[packet out wrong interface] "
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -j DROP
COMMIT

- ICMP error is generated by this router but destination address of it is strange.

Jul  4 14:54:33 webby kernel: [packet out wrong interface] IN= OUT=eth1
SRC=123.23.23.23 DST=192.168.0.133 LEN=68 TOS=0x00 PREC=0xC0 TTL=64
ID=39698 PROTO=ICMP TYPE=3 CODE=3 [SRC=192.168.0.133 DST=123.23.23.23
LEN=40 TOS=0x00 PREC=0x20 TTL=239 ID=39262 PROTO=TCP SPT=25000 DPT=25000
WINDOW=64172 RES=0x00 RST URGP=0 ]

- original TCP reset is following

14:54:33.931831 IP (tos 0x20, ttl 239, id 39262, offset 0, flags [none],
proto: TCP (6), length: 40) 70.243.226.250.1703 > 123.23.23.23.25000: R,
cksum 0xacb6 (correct), 4070626809:4070626809(0) win 64172

- no error message is printed by nf_ct_tcp, no 'table is full' error.


My idea is following.

- This TCP reset is not initial packet of TCP connection. If it is initial packet,
  no address in ICMP packet should be mangled. Jordan, if you see
  /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.

- TCP packet was marked as error packet. Because '--state ESTABLISHED'
  didn't match the packet. No conntrack entry wasn't assigned to the
  packet. Usually, error log by nf_conntrack_tcp should be generated in
  such case, but no message is generated in some cases. I don't know why
  this TCP reset was handled as error.

- Then ICMP error generated at this router was not assigned to any conntrack entry.

- nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
  the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.

        h = nf_conntrack_find_get(&innertuple, NULL);
        if (!h) {
                /* Locally generated ICMPs will match inverted if they
                   haven't been SNAT'ed yet */
                /* FIXME: NAT code has to handle half-done double NAT --RR  */
                if (hooknum == NF_IP_LOCAL_OUT)
                        h = nf_conntrack_find_get(&origtuple, NULL);

                if (!h) {
                        DEBUGP("icmp_error_message: no match\n");
                        return -NF_ACCEPT;
                }

                /* Reverse direction from that found */
                if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
                        *ctinfo += IP_CT_IS_REPLY;
        } else {
                if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
                        *ctinfo += IP_CT_IS_REPLY;


- As a result, nf_nat_packet mistook the address in ICMP packet which
  should be mangled.

[-- Attachment #2: 0001-NETFILTER-nf_conntrack-Fixes-direction-of-locall.patch --]
[-- Type: Text/Plain, Size: 1533 bytes --]

>From e36c67a5f57a0bd45f6666627ad3d60d42ee4497 Mon Sep 17 00:00:00 2001
From: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
Date: Sat, 7 Jul 2007 20:29:39 +0900
Subject: [PATCH 1/1] [NETFILTER]: nf_conntrack: Fixes direction of locally generated ICMP error

The conntrack assigned to locally generated ICMP error is usually the one
assigned to the original packet which has caused the error. But if
the original packet is handled as invalid by nf_conntrack, no conntrack
is assigned to the original packet. Then nf_ct_attach() cannot assign
any conntrack to the ICMP error packet. In that case nf_conntrack_icmp
tries to assign appropriate conntrack to it. But the current code
mistakes the direction of the packet. As a result, NAT code mistakes
the address in the packet to mangle.

Spotted by Jordan Russell.

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
---
 net/ipv4/netfilter/nf_conntrack_proto_icmp.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
index f4fc657..5973b58 100644
--- a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
+++ b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
@@ -201,7 +201,7 @@ icmp_error_message(struct sk_buff *skb,
 		}
 
 		/* Reverse direction from that found */
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
+		if (NF_CT_DIRECTION(h) != IP_CT_DIR_REPLY)
 			*ctinfo += IP_CT_IS_REPLY;
 	} else {
 		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-- 
1.5.2.2

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Patrick McHardy]

On Sat, 7 Jul 2007, Yasuyuki KOZAKAI wrote:

> OK, I try to explain what the current netfilter does in Jordan's situation.
> I also attach the patch to fix this problem. Jordan, can you try it ?
>
> Patrick, please consider to apply it. Even if following my idea is not
> correct, clearly it fixes a bug. I'm wondering why I missed '!'
> when I copied & pasted ip_conntrack_proto_icmp.c...


I'm currently travelling, I'll look into it tomorrow. Just one
question below ..

> My idea is following.
>
> - This TCP reset is not initial packet of TCP connection. If it is initial packet,
>  no address in ICMP packet should be mangled. Jordan, if you see
>  /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.
>
> - TCP packet was marked as error packet. Because '--state ESTABLISHED'
>  didn't match the packet. No conntrack entry wasn't assigned to the
>  packet. Usually, error log by nf_conntrack_tcp should be generated in
>  such case, but no message is generated in some cases. I don't know why
>  this TCP reset was handled as error.
>
> - Then ICMP error generated at this router was not assigned to any conntrack entry.
>
> - nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
>  the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.
>
>        h = nf_conntrack_find_get(&innertuple, NULL);
>        if (!h) {
>                /* Locally generated ICMPs will match inverted if they
>                   haven't been SNAT'ed yet */
>                /* FIXME: NAT code has to handle half-done double NAT --RR  */
>                if (hooknum == NF_IP_LOCAL_OUT)
>                        h = nf_conntrack_find_get(&origtuple, NULL);


The local ICMP tracking is basically useless nowadays since we always
manually attach the conntrack reference from the original packet
(exactly because of the half-done double NAT FIXME quoted above).
But this is an interesting case, the connection tracking code itself
thought the RST was invalid, but ICMP tracking will still associate
the ICMP containing the RST with the original connection. I'm wondering
whether it should really do that. In case it shouldn't, just removing
all locally generated ICMP special-casing should also fix the bug,
right?

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)

> > My idea is following.
> >
> > - This TCP reset is not initial packet of TCP connection. If it is initial packet,
> >  no address in ICMP packet should be mangled. Jordan, if you see
> >  /proc/net/netfilter/nf_conntrack, you will find the entry matched the TCP packet.
> >
> > - TCP packet was marked as error packet. Because '--state ESTABLISHED'
> >  didn't match the packet. No conntrack entry wasn't assigned to the
> >  packet. Usually, error log by nf_conntrack_tcp should be generated in
> >  such case, but no message is generated in some cases. I don't know why
> >  this TCP reset was handled as error.
> >
> > - Then ICMP error generated at this router was not assigned to any conntrack entry.
> >
> > - nf_conntarck_icmp.c assigns the ICMP error to the conntrack which matches
> >  the TCP reset. But IP_CT_IS_REPLY didn't set to *ctinfo. This is bug.
> >
> >        h = nf_conntrack_find_get(&innertuple, NULL);
> >        if (!h) {
> >                /* Locally generated ICMPs will match inverted if they
> >                   haven't been SNAT'ed yet */
> >                /* FIXME: NAT code has to handle half-done double NAT --RR  */
> >                if (hooknum == NF_IP_LOCAL_OUT)
> >                        h = nf_conntrack_find_get(&origtuple, NULL);
> 
> 
> The local ICMP tracking is basically useless nowadays since we always
> manually attach the conntrack reference from the original packet
> (exactly because of the half-done double NAT FIXME quoted above).
> But this is an interesting case, the connection tracking code itself
> thought the RST was invalid, but ICMP tracking will still associate
> the ICMP containing the RST with the original connection. I'm wondering
> whether it should really do that. In case it shouldn't, just removing
> all locally generated ICMP special-casing should also fix the bug,
> right?

At first I thought so. But I didn't come up with any bad situation caused
by returning ICMP error to such invalid packets.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Sun, 08 Jul 2007 02:28:54 +0900 (JST)

> From: Patrick McHardy <kaber@trash.net>
> Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)
> > 
> > The local ICMP tracking is basically useless nowadays since we always
> > manually attach the conntrack reference from the original packet
> > (exactly because of the half-done double NAT FIXME quoted above).
> > But this is an interesting case, the connection tracking code itself
> > thought the RST was invalid, but ICMP tracking will still associate
> > the ICMP containing the RST with the original connection. I'm wondering
> > whether it should really do that. In case it shouldn't, just removing
> > all locally generated ICMP special-casing should also fix the bug,
> > right?
> 
> At first I thought so. But I didn't come up with any bad situation caused
> by returning ICMP error to such invalid packets.

Can kernel correctly return ICMP error without conntrack in this case ?
If so, I agree. (maybe yes, I'll check it after waking up).

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>

> From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
>
> > From: Patrick McHardy <kaber@trash.net>
> > Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)
> > > 
> > > The local ICMP tracking is basically useless nowadays since we always
> > > manually attach the conntrack reference from the original packet
> > > (exactly because of the half-done double NAT FIXME quoted above).
> > > But this is an interesting case, the connection tracking code itself
> > > thought the RST was invalid, but ICMP tracking will still associate
> > > the ICMP containing the RST with the original connection. I'm wondering
> > > whether it should really do that. In case it shouldn't, just removing
> > > all locally generated ICMP special-casing should also fix the bug,
> > > right?
> > 
> > At first I thought so. But I didn't come up with any bad situation caused
> > by returning ICMP error to such invalid packets.
> 
> Can kernel correctly return ICMP error without conntrack in this case ?
> If so, I agree. (maybe yes, I'll check it after waking up).

I've tested following patch in some situations. Unless I used state or
conntrack match in OUTPUT (e.g. --state INVALID -j DROP), there is
no change of behavior. But can we accept this change ? ICMP error itself
is not invalid.


[NETFILTER]: nf_conntrack: Don't track locally generated special ICMP error

The conntrack assigned to locally generated ICMP error is usually the one
assigned to the original packet which has caused the error. But if
the original packet is handled as invalid by nf_conntrack, no conntrack
is assigned to the original packet. Then nf_ct_attach() cannot assign
any conntrack to the ICMP error packet. In that case the current
nf_conntrack_icmp assigns appropriate conntrack to it. But the current
code mistakes the direction of the packet. As a result, NAT code mistakes
the address to be mangled.

To fix the bug, this changes nf_conntrack_icmp not to assign conntrack
to such ICMP error. Actually no address is necessary to be mangled
in this case.

Spotted by Jordan Russell.

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
---
 net/ipv4/netfilter/nf_conntrack_proto_icmp.c |   22 +++++-----------------
 1 files changed, 5 insertions(+), 17 deletions(-)

diff --git a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
index f4fc657..474b4ce 100644
--- a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
+++ b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
@@ -189,25 +189,13 @@ icmp_error_message(struct sk_buff *skb,
 
 	h = nf_conntrack_find_get(&innertuple, NULL);
 	if (!h) {
-		/* Locally generated ICMPs will match inverted if they
-		   haven't been SNAT'ed yet */
-		/* FIXME: NAT code has to handle half-done double NAT --RR */
-		if (hooknum == NF_IP_LOCAL_OUT)
-			h = nf_conntrack_find_get(&origtuple, NULL);
-
-		if (!h) {
-			DEBUGP("icmp_error_message: no match\n");
-			return -NF_ACCEPT;
-		}
-
-		/* Reverse direction from that found */
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
-	} else {
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
+		DEBUGP("icmp_error_message: no match\n");
+		return -NF_ACCEPT;
 	}
 
+	if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
+		*ctinfo += IP_CT_IS_REPLY;
+
 	/* Update skb to refer to this connection */
 	skb->nfct = &nf_ct_tuplehash_to_ctrack(h)->ct_general;
 	skb->nfctinfo = *ctinfo;
-- 
1.5.2.2

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Patrick McHardy]

Yasuyuki KOZAKAI wrote:
> From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
> Date: Sun, 08 Jul 2007 02:28:54 +0900 (JST)
> 
> 
>>From: Patrick McHardy <kaber@trash.net>
>>Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)
>>
>>>The local ICMP tracking is basically useless nowadays since we always
>>>manually attach the conntrack reference from the original packet
>>>(exactly because of the half-done double NAT FIXME quoted above).
>>>But this is an interesting case, the connection tracking code itself
>>>thought the RST was invalid, but ICMP tracking will still associate
>>>the ICMP containing the RST with the original connection. I'm wondering
>>>whether it should really do that. In case it shouldn't, just removing
>>>all locally generated ICMP special-casing should also fix the bug,
>>>right?
>>
>>At first I thought so. But I didn't come up with any bad situation caused
>>by returning ICMP error to such invalid packets.
> 
> 
> Can kernel correctly return ICMP error without conntrack in this case ?
> If so, I agree. (maybe yes, I'll check it after waking up).


All locally generated ICMP errors are already associated with the
original conntrack and packets skip connection tracking. The
only locally generated ICMP packets that are handled by conntrack
are errors for INVALID packets.

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Mon, 09 Jul 2007 15:34:46 +0200

> Yasuyuki KOZAKAI wrote:
> > From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
> > Date: Sun, 08 Jul 2007 02:28:54 +0900 (JST)
> > 
> > 
> >>From: Patrick McHardy <kaber@trash.net>
> >>Date: Sat, 7 Jul 2007 17:34:49 +0200 (CEST)
> >>
> >>>The local ICMP tracking is basically useless nowadays since we always
> >>>manually attach the conntrack reference from the original packet
> >>>(exactly because of the half-done double NAT FIXME quoted above).
> >>>But this is an interesting case, the connection tracking code itself
> >>>thought the RST was invalid, but ICMP tracking will still associate
> >>>the ICMP containing the RST with the original connection. I'm wondering
> >>>whether it should really do that. In case it shouldn't, just removing
> >>>all locally generated ICMP special-casing should also fix the bug,
> >>>right?
> >>
> >>At first I thought so. But I didn't come up with any bad situation caused
> >>by returning ICMP error to such invalid packets.
> > 
> > 
> > Can kernel correctly return ICMP error without conntrack in this case ?
> > If so, I agree. (maybe yes, I'll check it after waking up).
> 
> 
> All locally generated ICMP errors are already associated with the
> original conntrack and packets skip connection tracking. The
> only locally generated ICMP packets that are handled by conntrack
> are errors for INVALID packets.

Well, what I wanted to say was wether kernel can use correct address to
locally generated ICMP where NAT is used. 
As I said in previous mail, I tested following patch in some
situations. Unless I used state or conntrack match in OUTPUT
(e.g. --state INVALID -j DROP), there is no change of behavior. But can we
accept this change ? ICMP error itself is not invalid.


[NETFILTER]: nf_conntrack: Don't track locally generated special ICMP error

The conntrack assigned to locally generated ICMP error is usually the one
assigned to the original packet which has caused the error. But if
the original packet is handled as invalid by nf_conntrack, no conntrack
is assigned to the original packet. Then nf_ct_attach() cannot assign
any conntrack to the ICMP error packet. In that case the current
nf_conntrack_icmp assigns appropriate conntrack to it. But the current
code mistakes the direction of the packet. As a result, NAT code mistakes
the address to be mangled.

To fix the bug, this changes nf_conntrack_icmp not to assign conntrack
to such ICMP error. Actually no address is necessary to be mangled
in this case.

Spotted by Jordan Russell.

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
---
 net/ipv4/netfilter/nf_conntrack_proto_icmp.c |   22 +++++-----------------
 1 files changed, 5 insertions(+), 17 deletions(-)

diff --git a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
index f4fc657..474b4ce 100644
--- a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
+++ b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
@@ -189,25 +189,13 @@ icmp_error_message(struct sk_buff *skb,
 
 	h = nf_conntrack_find_get(&innertuple, NULL);
 	if (!h) {
-		/* Locally generated ICMPs will match inverted if they
-		   haven't been SNAT'ed yet */
-		/* FIXME: NAT code has to handle half-done double NAT --RR */
-		if (hooknum == NF_IP_LOCAL_OUT)
-			h = nf_conntrack_find_get(&origtuple, NULL);
-
-		if (!h) {
-			DEBUGP("icmp_error_message: no match\n");
-			return -NF_ACCEPT;
-		}
-
-		/* Reverse direction from that found */
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
-	} else {
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
+		DEBUGP("icmp_error_message: no match\n");
+		return -NF_ACCEPT;
 	}
 
+	if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
+		*ctinfo += IP_CT_IS_REPLY;
+
 	/* Update skb to refer to this connection */
 	skb->nfct = &nf_ct_tuplehash_to_ctrack(h)->ct_general;
 	skb->nfctinfo = *ctinfo;
-- 
1.5.2.2

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

Yasuyuki KOZAKAI wrote:
> [NETFILTER]: nf_conntrack: Don't track locally generated special ICMP error


Thanks. I had to manually apply it because it clashed with the
pr_debug changes, please verify that I made no mistake. I'll
also push your original patch to -stable.

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 2508 bytes --]

[NETFILTER]: nf_conntrack: Don't track locally generated special ICMP error

The conntrack assigned to locally generated ICMP error is usually the one
assigned to the original packet which has caused the error. But if
the original packet is handled as invalid by nf_conntrack, no conntrack
is assigned to the original packet. Then nf_ct_attach() cannot assign
any conntrack to the ICMP error packet. In that case the current
nf_conntrack_icmp assigns appropriate conntrack to it. But the current
code mistakes the direction of the packet. As a result, NAT code mistakes
the address to be mangled.

To fix the bug, this changes nf_conntrack_icmp not to assign conntrack
to such ICMP error. Actually no address is necessary to be mangled
in this case.

Spotted by Jordan Russell.

Signed-off-by: Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp>
Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit b94585f7a2d237fa09af67063b0dc499e5461da6
tree ff17e171f22901060080b61f0e2383603a0407e9
parent 0c066bfa040c11160cbcfbad27e0a513801f6768
author Yasuyuki Kozakai <yasuyuki.kozakai@toshiba.co.jp> Fri, 13 Jul 2007 16:48:17 +0200
committer Patrick McHardy <kaber@trash.net> Fri, 13 Jul 2007 16:48:17 +0200

 net/ipv4/netfilter/nf_conntrack_proto_icmp.c |   22 +++++-----------------
 1 files changed, 5 insertions(+), 17 deletions(-)

diff --git a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
index b8b7999..f965733 100644
--- a/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
+++ b/net/ipv4/netfilter/nf_conntrack_proto_icmp.c
@@ -165,25 +165,13 @@ icmp_error_message(struct sk_buff *skb,
 
 	h = nf_conntrack_find_get(&innertuple);
 	if (!h) {
-		/* Locally generated ICMPs will match inverted if they
-		   haven't been SNAT'ed yet */
-		/* FIXME: NAT code has to handle half-done double NAT --RR */
-		if (hooknum == NF_IP_LOCAL_OUT)
-			h = nf_conntrack_find_get(&origtuple);
-
-		if (!h) {
-			pr_debug("icmp_error_message: no match\n");
-			return -NF_ACCEPT;
-		}
-
-		/* Reverse direction from that found */
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
-	} else {
-		if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
-			*ctinfo += IP_CT_IS_REPLY;
+		pr_debug("icmp_error_message: no match\n");
+		return -NF_ACCEPT;
 	}
 
+	if (NF_CT_DIRECTION(h) == IP_CT_DIR_REPLY)
+		*ctinfo += IP_CT_IS_REPLY;
+
 	/* Update skb to refer to this connection */
 	skb->nfct = &nf_ct_tuplehash_to_ctrack(h)->ct_general;
 	skb->nfctinfo = *ctinfo;

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Fri, 13 Jul 2007 16:50:02 +0200

> Yasuyuki KOZAKAI wrote:
> > [NETFILTER]: nf_conntrack: Don't track locally generated special ICMP error
> 
> 
> Thanks. I had to manually apply it because it clashed with the
> pr_debug changes, please verify that I made no mistake. I'll
> also push your original patch to -stable.

Sorry for inconvenience. I've checked it. Looks fine except of a Hunk.
Thanks.

-- Yasuyuki Kozakai

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Jordan Russell]

Yasuyuki KOZAKAI wrote:
> OK, I try to explain what the current netfilter does in Jordan's situation.
> I also attach the patch to fix this problem. Jordan, can you try it ?

Excellent. The patch does appear to fix the problem. I repeated the
BitTorrent test, and no packets matched my LOG rule.

I'll keep watching the logs and report back if it happens again.

Thanks for looking into this!

-- 
Jordan Russell

Re: ICMP packets associated with NAT connections sent out wrong interface?

[Yasuyuki KOZAKAI]

From: Jordan Russell <jr-list-2007@quo.to>

> Yasuyuki KOZAKAI wrote:
> > OK, I try to explain what the current netfilter does in Jordan's situation.
> > I also attach the patch to fix this problem. Jordan, can you try it ?
> 
> Excellent. The patch does appear to fix the problem. I repeated the
> BitTorrent test, and no packets matched my LOG rule.
> 
> I'll keep watching the logs and report back if it happens again.
> 
> Thanks for looking into this!

Thanks for testing.

-- Yasuyuki Kozakai